In plain language
learns reusable representations. To solve task X, add an output head and train it with task-specific . You can also update some or all of the pretrained model.
07 / The fine-tuning lab
A new task. A head start.
Reuse what a model has learned, then adapt it to a specific task. Compare a frozen , full and .
The adaptation experiment
Run, pause and inspect each epoch0epoch
—train loss
—validation accuracy
Inspect a point. Color = predicted probability; shape = observed class.
The initial surface is untrained. Run the experiment to compute the full trajectory.
○ Class 0◇ Class 1 / train lossValidation loss
Ready to train01
Protein sequence
Protein sequence
02
Pretrained pLM
Pretrained pLM
03
Pooled embedding
Pooled embedding
04
Task-specific head
Task-specific head
05
Held-out evaluation
Held-out evaluation
The technical idea
A linear probe freezes the and trains a head. Full updates both. freezes original and learns low-rank changes ΔW = BA in selected .
From data to proteins
For protein solubility prediction, a possible pipeline is sequence → pretrained → → binary head. Use experimentally grounded , family-aware splits and a held-out test set before interpreting performance.
Look deeper: mathematics, methods & limitations
Wadapted = Wfrozen + (α/r)BA
The live experiment runs a tiny two-input, six-hidden-unit network. It first learns a synthetic source task (x₁ + 0.6x₂ > 0), resets the output head, then learns a synthetic target task (x₁x₂ < 0). Both tasks use synthetic coordinates, not real proteins or solubility .
Frozen mode updates seven output-head . Full mode updates all 25 . Low-rank mode updates rank-1 factors A and B around the frozen 6×2 input , plus the seven head : 15 trainable . The low-rank scale is one in this demo.
Each run reuses the same source and target split so strategy comparisons are controlled. Its validation score measures only the synthetic target task. Larger models, real data and different optimizers can change the relative performance.
For a real workflow: deduplicate and cluster sequences; reserve an untouched test set; fit preprocessing only on training data; tune and stopping on validation; compare against a descriptor baseline and frozen ; report uncertainty, class-sensitive metrics and external validation when possible.
is not prompting, retrieval-augmented generation or ordinary . It changes trainable . Large training requires an external Python/GPU environment; this museum intentionally runs only the small educational experiments.