Standing
The scaling study that made masked protein language models a standard component. Later work initialises other models from ESM-2 weights and uses its embeddings as features.
Protein language model · Meta AI · 2023
ESM-2
A bidirectional transformer trained to reconstruct masked amino acids from unaligned sequences. It produces contextual per-residue representations and amino-acid logits. Converting those representations into coordinates requires ESMFold, which is a separate model.
In plain language
A model that was trained by hiding amino acids and guessing them back. Doing that well forces it to learn what each position in a protein is for, and those internal numbers turn out to be useful for almost any protein task.
One way to picture it
Fill in the blanks, ten billion times, until you cannot help but understand the grammar. Nobody taught it chemistry. It learned which residues belong together by being asked to guess them.
Commonly misread as
It does not predict structure. It produces vectors. ESMFold is the model that turns those vectors into coordinates, and it is a separate download.
How it is put together
Select a component to read itToken embeddings
Contextual representations
Per-residue vectors
Try it
Get an embedding for one sequenceimport torch
from transformers import AutoTokenizer, AutoModel
name = "facebook/esm2_t33_650M_UR50D"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name).eval()
seq = "MALWMRLLPLLALLALWGPDPAAA"
batch = tok(seq, return_tensors="pt")
with torch.no_grad():
out = model(**batch).last_hidden_state
# drop the start and end tokens, then average over residues
per_residue = out[0, 1:-1]
sequence_vector = per_residue.mean(0)
print(per_residue.shape, sequence_vector.shape)These snippets have not been executed here. Versions move; check the model card before trusting a line of it.
Distinctions
- No alignment at inference. Evolutionary information lives in the weights.
- Checkpoints from 8M to 15B parameters make the scaling behaviour visible.
- Attention maps correlate with residue contacts although no structural term appears in the loss.
Limits
It is an encoder. It does not generate structures, and its per-residue logits are a poor proxy for fitness in cases where function depends on more than sequence likelihood.
Weights and code
Checked against the registry, not from memory| Repository | Size | Licence | Note |
|---|---|---|---|
| facebook/esm2_t33_650M_UR50D | 650M | MIT | The default choice for embeddings |
| facebook/esm2_t12_35M_UR50D | 35M | MIT | Small enough for a laptop |
| facebook/esm2_t36_3B_UR50D | 3B | MIT | — |
| facebook/esm2_t48_15B_UR50D | 15B | MIT | Rarely worth the memory over 3B |
Sources
Each number above comes from one of theseLin et al. 2023, Science. Evolutionary-scale prediction of atomic-level protein structure with a language modelPeer reviewedESM-2 650M weightsModel card
Same task, other answers
Protein language modelESM3
Sequence, structure and function as one masked prediction.
A generative masked model over three parallel tracks.
Explore the architectureESM C
The representation line, scaled further.
A transformer encoder with pre-layer-norm, rotary embeddings and SwiGLU activations, trained on sequences from UniRef, MGnify and the Joint Genome Institute clustered at 70 percent identity.
Read the entryProtT5 and ProtBERT
The encoders that were there first.
A family of transformers trained on UniRef and BFD with the objectives of their text counterparts.
Read the entry