In plain language
For each , asks which other are useful now. It computes a weighted mixture of their information, producing a context-aware representation.
05 / Attention & Transformers
Context changes everything.
A does not have to work alone. Inspect how , keys and values let it exchange information with the rest of a sequence.
Inside self-attention
Hover a cell to follow the calculationKeys / columns →
Q↓MAKLVGMAKLVG
Read one row: how a query distributes attention across all keys.
Output[3] = Σⱼ attention[3, j] × V[j]
= [-0.26, -0.42, 0.55, -0.54]
= [-0.26, -0.42, 0.55, -0.54]
Exact math · fixed, untrained toy projectionsdₖ = 4 · n = 6 · 3 independent heads
Build a Transformer block
Select a layer to see what it doesREPEAT × N TRANSFORMER BLOCKS
Interactive exhibitExplore with mouse, touch or keyboard
The technical idea
Q = XWQ, K = XWK, V = XWV. -key dot products are scaled by √dk, optionally masked, then normalized row-wise with . Each row of sums to one.
From data to proteins
can encode sequence dependencies, including long-range ones. An value is not automatically a physical contact, a causal explanation or proof of biochemical interaction.
Look deeper: mathematics, methods & limitations
Attention(Q,K,V) = softmax(QKᵀ/√dₖ + M)V
The heatmap uses deterministic, untrained 4-dimensional with sinusoidal position information. Each selectable head uses different fixed 4×4 projections. These are exact calculations on toy , not extracted ESM or .
A causal mask sets future-position scores to negative infinity before , making their probabilities zero. An uses bidirectional context; a -only language model uses a causal mask.
concatenates each head’s result and projects it back to the model dimension. A block also includes a position-wise feed-forward network, and . Stacking alone does not describe the whole architecture.
The original is an - model. uses the design; GPT-style models use causal blocks. placement and positional mechanisms vary across model families. Standard dense creates an n×n interaction per head, with quadratic sequence-length cost.