Protein language model · Rostlab · 2022

ProtT5 and ProtBERT

A family of transformers trained on UniRef and BFD with the objectives of their text counterparts. The T5 encoder is the part normally used, and its per-residue embeddings feed small supervised heads.

In plain language

The first generation of protein language models that worked. Still a fair baseline, and still fast enough to be worth trying before reaching for something larger.

One way to picture it

The reliable old lens. Sharper glass exists now, and this one still takes the picture.

Commonly misread as

Load the encoder half of ProtT5 and ignore the decoder. Almost nobody uses the full sequence-to-sequence model.

01 / Why it is here

Standing

Established that language models trained on protein sequences produce features carrying structural information, and that the embeddings transfer to secondary structure and localisation prediction without alignments.

02 / What sets it apart

Distinctions

ProtT5 embeddings remain a strong baseline for small labelled datasets.
The encoder is used alone; the decoder half is normally discarded.
Medium-sized encoders often match much larger ones once the downstream dataset is small.

03 / Where it stops

Limits

Older tokenisation and training data than the ESM line. For new work these are baselines rather than defaults.

Weights and code

Checked against the registry, not from memory

Repository	Size	Licence	Note
Rostlab/prot_t5_xl_uniref50	3B, encoder used alone	See model card	—
Rostlab/prot_bert	420M	See model card	—

Sources

Each number above comes from one of these

Elnaggar et al. 2022, IEEE TPAMI. ProtTrans: toward understanding the language of life through self-supervised learningPeer reviewed ProtT5-XL weightsModel card

Same task, other answers

Protein language model

Protein language model2023

ESM-2

Read enough proteins and structure falls out of the reading.

A bidirectional transformer trained to reconstruct masked amino acids from unaligned sequences.

Explore the architecture Protein language model2025

ESM3

Sequence, structure and function as one masked prediction.

A generative masked model over three parallel tracks.

Explore the architecture Protein language model2026

ESM C

The representation line, scaled further.

A transformer encoder with pre-layer-norm, rotary embeddings and SwiGLU activations, trained on sequences from UniRef, MGnify and the Joint Genome Institute clustered at 70 percent identity.

Read the entry

Back to the catalogueCompare it with its neighbours