Source-linked AI summary
Deciphering antibody affinity maturation with language models and weakly supervised learning
Jeffrey A. Ruffolo, Jeffrey J. Gray, Jeremias Sulam
TL;DR
The paper asks whether antibody-sequence models can reveal affinity maturation and antigen-binding information from immune repertoires. It trains AntiBERTy and combines its embeddings with multiple instance learning, finding maturation-like sequence trajectories and attention localized to binding residues for VRC01 antibodies.
Problem
Immune repertoires provide sequence snapshots, but general protein models may not capture antibody-specific evolution and individual sequence binding labels are unavailable.
Method
The paper trains AntiBERTy on 558M natural antibody sequences and uses its embeddings in a multiple instance learning model trained to identify highly redundant sequence bags.
Results
AntiBERTy embeddings cluster repertoire sequences into trajectories resembling affinity maturation, while MIL attention localizes to binding residues in VRC01 antibodies.
Takeaways & Limitations
The methods may provide biological insights into affinity maturation and enable identification of paratope residues from repertoire sequences alone with further development.
Takeaways & Limitations
The MIL approach assumes that the most frequently observed antibodies are binders, making redundancy a noisy binding label.
Abstract
from arXiv · showhide
In response to pathogens, the adaptive immune system generates specific antibodies that bind and neutralize foreign antigens. Understanding the composition of an individual's immune repertoire can provide insights into this process and reveal potential therapeutic antibodies. In this work, we explore the application of antibody-specific language models to aid understanding of immune repertoires. We introduce AntiBERTy, a language model trained on 558M natural antibody sequences. We find that within repertoires, our model clusters antibodies into trajectories resembling affinity maturation. Importantly, we show that models trained to predict highly redundant sequences under a multiple instance learning framework identify key binding residues in the process. With further development, the methods presented here will provide new insights into antigen binding from repertoire sequences alone.
1 Introduction
Antibody diversity enables robust pathogen responses, while affinity maturation selects and mutates high-affinity antibodies. Immune repertoires offer sequence snapshots, motivating antibody-specific language models because general protein models may miss antigen-focused evolution.
- 1 Introduction: V(D)J recombination initially generates antibody-sequence diversity, followed by antigen-driven affinity maturation.Affinity maturation selectively expands B-cells producing high-affinity antibodies and mutates them into successive generations.
- 1 Introduction: Immune repertoire samples provide snapshots of an individual's antibody sequence population.Samples typically come from B-cells in blood or lymph and are characterized by next-generation sequencing.
- 1 Introduction: Among binding antibodies, the most frequent sequences tend to be effective binders, but redundancy is generally a poor binding indicator beyond highly expanded sequences.During an immune response, as many as half of repertoire antibodies may exhibit antigen affinity.
- 1 Introduction: General protein language models may poorly capture antibody evolution because antibodies are selected for binding particular antigens.The paper therefore explores an antibody-specific language model for affinity maturation within immune repertoires.
2 Methods
The methods combine AntiBERTy sequence embeddings with repertoire-specific multiple instance learning. Attention pools residue and sequence information to predict bags containing highly redundant sequences.
- 2.1 Antibody encoder model: AntiBERTy is trained with masked language modeling on 558M non-redundant natural antibody sequences.The model is a transformer encoder based on the BERT architecture.
- 2.2 Evolutionary analysis: AntiBERTy embeddings are used to construct k-nearest-neighbor graphs for repertoires from four donors producing VRC01-class antibodies.The analysis examines global trends in antibody sequence evolution within individual repertoires.
- 2.3 Multiple instance learning on sequence embeddings: Because individual binding labels are absent, the MIL model uses high sequence redundancy as a noisy proxy for antigen binding.The approach assumes that the most frequently observed antibodies are binders.
- 2.3 Multiple instance learning on sequence embeddings: Sequences above the 85th redundancy percentile form likely-binder examples, while 64-sequence bags sampled from lower-redundancy partitions form negative examples.Positive and negative bags are sampled uniformly from their respective partitions.
- 2.3.2 Model architecture and training: The instance module transforms AntiBERTy residue representations and applies four-head attention to obtain fixed-size sequence embeddings.Each sequence is processed individually before its embedding enters bag-level pooling.
- 2.3.2 Model architecture and training: Gated attention pools sequence embeddings into a fixed-size bag representation, which a two-layer network and logistic function classify.The model is trained with cross-entropy loss for 20 epochs using equally frequent positive and negative bags.
3 Results
AntiBERTy embeddings revealed repertoire structures resembling affinity-maturation trajectories, while MIL attention localized likely binding residues in most tested VRC01 antibodies.
- Language model reveals trajectories within repertoire: Four donors showed continuous trajectories from germline sequences to highly mutated derivatives in AntiBERTy-based UMAP embeddings.The combined graph also showed substantial overlap among RU3, IAVI57, and IAVI74 sequences.
- Figures: Figure 2 compares germline-distance UMAPs for individual and combined repertoires with sequence redundancy annotations.Figure 3 maps residue-level MIL attention onto ten VRC01–gp120 antibody structures, with warmer colors indicating higher attention.
- Language model reveals trajectories within repertoire: The combined repertoire displayed relatively uniform sequence redundancy throughout the embedded space.The authors interpret this pattern as consistent with iterative clonal expansion and diversification during affinity maturation.
- MIL model identifies VRC01 paratope: Seven of ten VRC01 antibodies showed attention localized to paratope residues in complexes with gp120.The second attention head localized particularly to the H2 loop and following beta strand, consistent with the VRC01 paratope.
4 Conclusion
The study applies antibody-specific language modeling and weakly supervised multiple instance learning to immune repertoires. AntiBERTy embeddings resemble affinity-maturation trajectories, while MIL attention localizes binding residues in VRC01 antibodies.
- Conclusion: AntiBERTy was trained on natural antibody sequences and produced repertoire embeddings that cluster into trajectories resembling affinity maturation.The study also trained MIL models to detect highly redundant sequences.
- Conclusion: MIL model attention localized to binding residues for an extensively studied class of antibodies.The authors identify this as a basis for potentially inferring paratope residues from repertoire sequences alone with further development.
A.1 Antibody sequence dataset
The antibody sequence dataset was constructed from the Observed Antibody Space and reduced to a non-redundant collection spanning multiple species and repertoire studies.
- Antibody sequence dataset: 588M non-redundant antibody sequences were extracted from over 1B variable-domain sequences in the Observed Antibody Space.The source database contains sequences from 80 immune repertoire sequencing studies.
- Antibody sequence dataset: The dataset included heavy and light chains from six species: human, mouse, rat, camel, rabbit, and rhesus.Sequences were clustered at 95% sequence identity with LinClust.
- Antibody sequence dataset: Five percent of the extracted sequences were held out, while 558M were used for training and 1M for evaluation and hyperparameter tuning.The split was applied after extracting the 588M-sequence non-redundant set.
A.2 AntiBERTy model
AntiBERTy is a BERT-based antibody language model trained with masked language modeling. Its training and evaluation losses are reported in Figure 4.
- AntiBERTy model: AntiBERTy contains approximately 26M trainable parameters across eight transformer layers with eight attention heads per layer.The model uses a 512-dimensional hidden representation and a 2048-dimensional feedforward dimension.
- AntiBERTy model: The model was trained for eight epochs over the full dataset, requiring approximately 10 days on four NVIDIA A100 GPUs.Training and evaluation loss are shown in Figure 4.
- AntiBERTy model: Figure 4 reports masked language modeling training and evaluation loss for AntiBERTy.
A.3 HIV-1 donor repertoires
The study uses four previously published repertoires from donors who developed neutralizing VRC01-class antibodies against HIV-1 gp120.
- Four donor repertoires—RU3, IAVI57, IAVI74, and C38—were collected from peripheral blood mononuclear cells.
- Each repertoire contains antibody sequences from the IGHG isotype.
- The repertoire sample sizes for the four donors are reported in Table 1.
A.4 Evo-velocity analysis
The authors compute evo-velocity scores on KNN edges using AntiBERTy representations and visualize them over repertoire embeddings. The resulting arrows consistently point toward germline rather than affinity maturation.
- Evo-velocity scores are computed for each edge between sequences xa and xb in a KNN graph.
- The analysis uses the set of residues differing between xa and xb and AntiBERTy representations obtained with each differing residue masked.
- The evo-velocity arrows consistently align toward germline sequences rather than toward affinity maturation.
- Figure 5 plots evo-velocity scores over Levenshtein-distance-annotated UMAP embeddings for each repertoire.
A.5 MIL model parameters
The study trains MIL models for individual and combined repertoires and reports their parameterization and attention patterns. Each model uses four attention heads, with 31.3K trainable parameters overall.
- Each MIL model contains 31.3K trainable parameters.
- Individual-repertoire models use batch size 32, while the combined model uses batch size 128.
- Models are trained with Adam, starting at learning rate 2e-5 followed by cosine annealing.
- Separate models are trained for each donor repertoire and for the combined dataset, with 20% of sequences held out for hyperparameter evaluation.
- The MIL model uses four attention heads; attention in the remaining heads is scattered throughout the antibody.The paper connects this pattern with prior observations that substantial framework mutations are necessary for functional VRC01 antibodies.