In plain language
A protein sequence is a chain of , often written with a 20-letter alphabet. A learns statistical regularities across many such sequences.
06 / Protein language models
New alphabet. Shared ideas.
Replace words with . Explore how learning from protein sequences can produce representations useful for biology.
A sequence becomes a representation
Select a residue, then mask itPrimary sequence
Residue tokens
Protein encoder
Residue embeddings
Masked language modeling → contextual embeddingsResidue identities · real descriptors
Meet the models
Related ideas. Different roles.ESM-2: a masked protein encoder
Learns contextual residue representations using masked protein sequence modeling. Its embeddings can feed task-specific heads. The largest ESM-2 model in the 2023 work has 15 billion parameters.
The technical idea
Masked such as learn contextual . A representation can be pooled into one per sequence, or passed to a -level prediction head. is part of the ProtTrans family.
From data to proteins
Possible downstream tasks include localization, secondary structure, stability and function prediction. Sequence representations are useful inputs, but a task still needs suitable , evaluation and biological interpretation.
Look deeper: mathematics, methods & limitations
Lmask = −Σᵢ∈M log p(aᵢ | a_visible)
Select a to mask it. The inspector reports its identity and simple chemical grouping; it does not predict which belongs at the masked position. Hydrophobic fraction is a descriptor, not a solubility probability.
is a . ESMFold combines representations with a folding architecture to predict structure. The language model alone is not the complete structure predictor. ProtTrans includes several , including and .
A is a learned , not a named list of physical properties. averages over valid , excluding padding and special , and loses some positional information. For -level tasks, retain per- representations.
learns from large unlabeled sequence collections via self-supervised targets. Evolutionary correlations may help representations capture structure and function, but sequence redundancy, and out-of-distribution sequences limit generalization.
| Model | Training / role | Useful distinction |
|---|---|---|
| ESM-2 | Masked protein language modeling | Contextual residue embeddings |
| ESMFold | Structure prediction using ESM-2 | Includes a folding architecture |
| ProtBERT | BERT-style masked protein modeling | Protein encoder within ProtTrans |
| ProtT5 | T5-based protein pretraining | Its encoder is commonly used for embeddings |