Every idea starts somewhere. Look up the words, see a concrete example, and jump into the experiment where it comes to life.
A rule that transforms a neuron’s weighted sum into its output.
A nonlinear activation allows composed layers to represent nonlinear functions.
Sigmoid, tanh and ReLU are common activations.
A building block of a protein; within a chain it is called a residue.
Standard sequence notation uses 20 common one-letter amino-acid codes; real biology also has modified and nonstandard residues.
K denotes lysine; G denotes glycine.
The field of making computers perform tasks associated with intelligence.
Includes learning-based systems and symbolic approaches such as rules, search and planning.
A chess search program can be AI without being a neural network.
A way for each token to combine information from other tokens using learned relevance weights.
Scaled dot-product attention forms a normalized QKᵀ matrix and multiplies it by V.
A masked residue can use information from distant sequence positions.
One set of attention projections inside a multi-head layer.
Different heads have separate Q, K and V projections; outputs are concatenated and projected.
Select Head 1 and Head 2 to see different toy attention patterns.
Working backward through a network to find how each parameter affected the loss.
The chain rule computes gradients efficiently through the computation graph.
Output error is propagated through the hidden layer to the input weights.
A group of examples processed together in one training step.
Mini-batch gradients estimate the gradient over the full dataset.
A batch of 32 proteins can produce one optimizer update.
A model that learns text representations by using context on both sides.
Bidirectional Encoder Representations from Transformers; original training combines masked language modeling and next-sentence prediction.
Predict the hidden token in “the protein is [MASK]”.
An adjustable offset that shifts a neuron’s response.
The additive b in z = w·x + b. This meaning differs from dataset or social bias.
Even if all inputs are zero, a bias of 0.3 makes z = 0.3.
A loss that penalizes assigning low probability to the correct answer.
For binary labels: −[y log p + (1−y) log(1−p)].
A confident incorrect prediction costs more than an uncertain one.
Information reaches training or model selection that should only be available for final evaluation.
Duplicates, homologous sequence overlap or preprocessing fit on all data can inflate reported performance.
Closely related proteins on both sides of a random split can make evaluation too easy.
A collection of examples used to train or evaluate a model.
Splits have distinct roles: training fits parameters; validation guides choices; test estimates final performance.
A table of protein sequences and measured properties.
A component that produces or models outputs from representations or earlier tokens.
A causal Transformer decoder blocks future context. Encoder-decoder systems also use cross-attention to encoder outputs.
A GPT-style decoder predicts the next token.
Machine learning that uses neural networks with multiple processing layers.
Nonlinear layers compose learned representations; depth is not the same as a large parameter count.
A Transformer stacks attention and feed-forward blocks.
A vector of numbers representing a token or sequence.
Learned coordinates encode useful statistical information; dimensions usually have no simple individual physical meaning.
A protein encoder produces a contextual vector for each residue.
A component that converts an input into a useful representation.
A Transformer encoder typically uses bidirectional self-attention to contextualize input tokens.
ESM-2 produces vectors describing residues in sequence context.
One complete pass through the training dataset.
An epoch may contain many optimizer steps if examples are divided into batches.
This lab uses full-batch training: one update per epoch.
A family of protein language models that learns from masked amino-acid sequences.
Provides contextual representations; the ESMFold system adds a folding architecture for structure prediction.
Use an ESM-2 embedding as input to a downstream classifier.
A measurable piece of information given to a model.
An input variable may be handcrafted or learned by an earlier representation.
Protein length or the fraction of hydrophobic amino acids.
Continuing training to adapt an existing model to a new task or domain.
Updates some or all model parameters using an adaptation objective.
Adapt a pretrained protein encoder and classifier to labeled localization data.
Kept fixed while other parts of the model learn.
Frozen parameters receive no optimizer updates.
In a linear probe, the encoder stays fixed and the task head learns.
A collection of slopes showing how a loss changes when parameters change.
The vector of partial derivatives ∇θL points toward local steepest increase under Euclidean geometry.
Gradient descent moves a small distance in the opposite direction.
Using an already trained model to produce an output.
An ordinary forward pass at inference does not update training parameters.
Predicting a masked token is inference; learning from its error is training.
A vector that a query compares against when computing attention scores.
K = XWK; matching is learned and does not necessarily correspond to a human-interpretable property.
The attention column identifies the token supplying a key and value.
The target answer attached to a training example.
A supervised target can be a class, a number or a structured output.
An experimentally measured soluble / insoluble annotation.
A language model trained at large scale to model token patterns.
There is no universally agreed parameter threshold; the training objective and architecture still matter.
A large causal decoder can generate text token by token.
A group of computations at the same stage in a network.
A hidden layer transforms an internal representation between input and output.
The tiny network has six tanh units in one hidden layer.
The size of each parameter-update step during training.
A hyperparameter η in θ ← θ − η∇θL.
A very small rate can learn slowly; a large one may overshoot.
A raw score before it becomes a probability.
Softmax transforms vocabulary logits into normalized probabilities.
A larger token logit gives it a larger probability relative to other tokens.
Adapting a model by learning small added matrices while keeping its original weights fixed.
Low-Rank Adaptation represents ΔW = BA, often scaled by α/r, in selected weight matrices.
For a 6×2 weight matrix, rank-1 factors have 6 + 2 trainable entries.
A number measuring how badly a prediction misses its target.
Optimization seeks parameters that reduce an objective averaged over training examples.
Predicting 0.01 for a true positive incurs a large cross-entropy loss.
Teaching a computer from examples rather than writing every decision rule.
Algorithms fit parameters or structures to data to improve an objective.
Train a classifier on labeled observations.
A way to hide token identities or block access to positions.
A masked-language target replaces or modifies selected inputs; a causal attention mask sets future scores to −∞. These are different uses of masking.
[MASK] hides a word; a causal mask blocks future attention.
A rectangular table of numbers used in calculations.
Can represent a linear transformation or a collection of vectors.
An attention matrix has one row per query and one column per key.
A mathematical system that turns inputs into outputs.
Its architecture defines the computation; learned parameters determine its specific behavior.
A logistic-regression model maps two features to a class probability.
A feed-forward network of connected layers of artificial neurons.
Typically alternates affine transformations with nonlinear activations.
The 2 → 6 → 1 network can learn a nonlinear boundary.
One small processing unit in a neural-network diagram.
A neuron usually computes a weighted sum plus a bias, followed by an activation. A graph node can have a broader meaning outside neural networks.
Each circle in the neuron exhibit represents a computational unit, not a biological brain cell.
Rescaling a set of values to keep their numerical behavior manageable.
Layer normalization standardizes features within an example and applies learned scale and offset.
A Transformer normalizes hidden activations around sublayers.
Learning details of the training examples that do not generalize well.
Low training error with poorer held-out performance can signal overfitting or distribution shift.
Memorizing near-duplicate sequences can inflate random-split scores.
A stored number that helps define a model’s behavior.
Weights and biases are parameters. Hyperparameters, such as learning rate, govern the training process.
A neuron with two weights and one bias has three parameters.
An early artificial neuron that makes a yes-or-no decision.
The historical perceptron uses a hard threshold of w·x + b and a mistake-driven update rule.
Two features are separated into classes by a straight-line boundary.
Combining several vectors into one summary vector.
Mean pooling averages valid residue vectors while excluding padding and special tokens.
Turn per-residue embeddings into one input for a sequence classifier.
Learning general patterns before adapting to a specific task.
Often uses self-supervised objectives on large unlabeled datasets.
Predicting hidden amino acids in many protein sequences.
A protein model based on BERT-style sequence learning.
Part of ProtTrans; applies masked modeling ideas to amino-acid sequences.
A bridge between BERT’s text encoder and biological sequence models.
A sequence model that learns patterns in amino-acid sequences.
Often uses masked or autoregressive self-supervision to learn contextual residue representations.
ESM-2, ProtBERT and ProtT5 are protein language models.
A T5-based protein language model from the ProtTrans project.
Its encoder can generate residue-level embeddings used in downstream prediction.
Pool ProtT5 embeddings to represent a complete protein sequence.
A token’s vector used to score which other tokens to attend to.
Produced by a learned projection Q = XWQ in self-attention.
A query is compared with every allowed key by dot product.
The number of independent directions a matrix can represent.
A product BA with an inner dimension r has rank at most r.
A rank-1 update changes a weight matrix in one outer-product direction.
An activation that keeps positive values and turns negative ones into zero.
Rectified Linear Unit: ReLU(z) = max(0,z). Its output is not bounded above.
ReLU(−2) = 0; ReLU(3) = 3.
A shortcut that adds an earlier representation to a transformed version.
Computes x + f(x), aiding information and gradient flow through stacked blocks.
A Transformer adds attention output back to its input.
A smooth S-shaped function that squashes a number between 0 and 1.
σ(z) = 1 / (1 + exp(−z)); commonly paired with binary cross-entropy.
At z = 0, sigmoid returns 0.5.
Turns a list of scores into positive weights that add up to one.
softmax(z)i = exp(zi) / Σj exp(zj), usually computed after subtracting the largest score for stability.
Each attention row is a probability distribution over allowed keys.
The final component that converts a representation into the output you need.
A classifier, regressor or token predictor attached to an encoder. Distinct from an attention head.
One sigmoid output can predict a binary protein label.
A setting that makes a probability distribution sharper or flatter.
Logits are divided by T before softmax. Lower T concentrates probability more strongly.
Increasing temperature spreads next-token probability across more candidates.
One discrete piece of a sequence that a model processes.
Tokenization maps raw text or sequences to vocabulary IDs; tokens need not be whole words.
A word piece in BERT, or an amino-acid letter in many pLMs.
A neural architecture that combines attention with feed-forward processing.
Includes residual paths, normalization and position information; variants include encoder, decoder and encoder-decoder systems.
BERT is an encoder-style Transformer.
Examples used to judge training choices without fitting weights directly on them.
Repeatedly choosing settings on validation data makes it part of model selection; an untouched test set is still needed.
Choose an early-stopping epoch from validation loss.
The information a token contributes after attention weights have been computed.
V = XWV; output is a weighted sum of value vectors.
A token with weight 0.2 contributes 0.2 times its value vector.
An ordered list of numbers.
Represents a point, direction or feature collection in a coordinate space.
[0.2, −0.5, 1.1] is a three-dimensional vector.
A number controlling the influence of one input on a computation.
A trainable coefficient in an affine transformation; it may be positive or negative.
With weight −0.5, an input of 0.4 contributes −0.2 to the sum.
A two-input logical rule that is true when exactly one input is true.
The positive and negative cases occupy opposite corners, so a single linear boundary cannot separate them.
The lab labels a point class 1 when its two coordinates have opposite signs.