In plain language
Tokenization turns a sequence into discrete symbols. An maps each symbol to a . A language model learns to assign probabilities using context.
04 / Language models
The meaning between tokens.
A language model learns statistical patterns in sequences. Compare next- prediction with ’s masked- objective.
Two ways to learn from a sequence
Change context, then compare probabilitiesInput sequence
Token IDs
Left + right context
Logits / scores
Softmax
BERT’s encoder can attend in both directions.
Run the experiment to reveal the distribution.
Self-supervision: the sequence provides the targetWord tokens · tiny corpus
See the training corpus and what BERT actually computes
The count demo learns frequencies from these sentences. Its displayed scores are log(count + 0.1) / temperature before softmax.
the protein is solublethe protein is stablethe protein is solublethe enzyme binds a substratethe enzyme is stablea protein has a sequencea sequence contains amino acidsthe model learns a sequencethe model predicts a tokenthe protein binds a moleculethe enzyme is solublea model learns from dataA real BERT pass
WordPiece tokens → token, position and segment embeddings → stacked bidirectional Transformer encoder blocks → masked-token prediction head → vocabulary softmax. Cross-entropy at selected targets supplies gradients during training. Original BERT also trained a next-sentence objective.
BERT is an encoder reference, not an autoregressive chat model. ProtBERT applies a related masked-encoder idea to protein sequences.
The technical idea
Autoregressive models predict the next from earlier . is an that learns bidirectional representations using masked language modeling; original also used next-sentence prediction.
From data to proteins
makes a useful bridge to masked protein modeling: hide part of a sequence and predict the missing from both sides. The general objective transfers, although tokenization and data differ.
Look deeper: mathematics, methods & limitations
Lnext = −Σ log p(xₜ | x<t)
-base has 12 layers, 768 hidden dimensions and 12 , with about 110 million . It is a useful introductory reference architecture, but not a primitive or a chat model.
In original , 15% of positions are selected for the masked objective. Of those selected positions, 80% become [MASK], 10% become a random , and 10% are left unchanged. is computed on the selected targets.
The interactive probability bars use a tiny count-based corpus. Next- mode uses the preceding word; masked mode uses adjacent left/right words when a context matches, otherwise unigram counts. Additive smoothing and turn counts into probabilities. This is not .
Text tokenization here splits words; actual uses WordPiece subwords. During autoregressive generation, changes the output distribution and sampling chooses a next . That is , not a training- update.