Modern Hopfield Networks and Attention for Immune Repertoire Classification

• Immune repertoire classification involves hundreds of thousands to millions of unlabeled sequences per bag, with witness rates as low as 0.01%.Only a small fraction of receptors may determine disease status.Source: Introduction · p. 2
• Low repertoire overlap, multiple sequences binding the same pathogen, and binding-determining subsequences complicate classification and motivate interpretable motif extraction.Source: Introduction · p. 3
• Prior immune-repertoire MIL approaches include DeepTCR’s basic attention averaging and Ostmeyer et al. (2019)’s fixed 4-mer instances with max-pooled prediction scores.Source: Introduction · p. 3
• DeepRC treats sequences as instances and uses transformer-like attention-pooling over learned representations, avoiding max-pooling over k-mers or prediction scores.Its sequence feature extractors can use 1D convolutions or LSTMs; the illustrated implementation uses stacked 1D convolutions.Source: Figure 1 · p. 2, Introduction · p. 3
• Modern Hopfield networks generalize binary networks to continuous states, use the transformer attention update rule, and can store exponentially many patterns.Source: Introduction · pp. 2, 4

Exponential storage capacity of continuous state modern Hopfield networks with transformer attention as update rule

• The update ξ_new = X softmax(βX^Tξ) converges globally to stationary points and is the transformer attention mechanism.The section describes stationary points as local minima or saddle points; simultaneous query updates with linear mappings yield the transformer attention formula.Source: Exponential storage capacity of… · p. 4
• Well-separated patterns yield fixed points near the pattern most similar to the initial state, while non-separated patterns yield a fixed point near their arithmetic mean.Similar patterns separated from other vectors can support metastable states; updates typically converge after one step and have exponentially small retrieval error.Source: Exponential storage capacity of… · p. 4
• Modern Hopfield networks and transformer attention can store and retrieve exponentially many randomly chosen patterns as dimension grows.The section identifies this capacity as critical for retrieving a few instances in massive MIL problems.Source: Exponential storage capacity of… · pp. 4–5

Problem setting and notation

• The model must handle unordered bags of more than 10,000 complex sequences with variable sizes and learn from bag labels despite hidden instance labels.Because only the bag label is observed, identifying which sequences are responsible for that label creates a credit-assignment problem.Source: Deep Repertoire Classification · p. 5

Modern Hopfield networks viewed as transformer-like attention mechanisms

• Modern Hopfield networks motivate DeepRC because their exponential storage capacity and one-update convergence suit retrieving repertoire patterns; their update rule is key-value attention used in transformers and BERT.The paper presents this as a theoretical rationale for storing and retrieving the many sequence patterns in immune repertoire classification.Source: Deep Repertoire Classification · p. 6

The DeepRC method

• DeepRC applies transformer-like attention pooling to immune-receptor sequence bags, producing a weighted-mean repertoire representation for classification.The attention mechanism can be extended to multiple queries, and the paper states that it can retrieve a single pattern from several hundred thousand.Source: Deep Repertoire Classification · p. 6

Attention mechanism in DeepRC

• DeepRC encodes sequence values with a 1D convolutional network, derives keys through a partly shared network, and learns fixed query vectors to weight sequence representations.The key network shares its first layers with the value network and then uses two self-normalizing layers with 32 units each; each attention head has its own fixed learned query.Source: Deep Repertoire Classification · p. 6

Attention-pooling and interpretability

• Attention weights indicate sequence importance, while Integrated Gradients and Layer-Wise Relevance Propagation provide additional contribution-analysis options.The attention-pooling mechanism reduces each repertoire to a fixed-size feature vector.Source: Deep Repertoire Classification · pp. 6–7

Classification layer and network parameters

• A fully connected output network predicts immune status and can jointly predict multiple classification or regression targets, including auxiliary metadata such as age or sex.The authors report that a single-layer output network was sufficient.Source: Deep Repertoire Classification · p. 7

Datasets

• The study comprised four dataset categories, with 785–5,000 repertoires per dataset and about 30 billion sequences overall.Witness rates ranged from 0.01% to 10%; repertoires averaged about 300,000 sequences except for the low-coverage implanted-signal category, which used 10,000.Source: Experimental Results · p. 8

Results

• DeepRC outperformed all competing methods in average AUC across all four dataset categories; runner-up methods were the MinMax-kernel SVM or burden test.The comparison included real-world data, real-world data with implanted signals, LSTM-generated data, and simulated immunosequencing data.Source: Table 1 · p. 8, Experimental Results · p. 8

Results on simulated immunosequencing data

• 0.846 ± 0.223 AUC for DeepRC exceeded the MinMax-kernel SVM’s 0.827 ± 0.210 AUC on simulated immunosequencing data.When only 0.01% of sequences carried the motif, all methods had AUC below 0.550.Source: Experimental Results · p. 8

Results on LSTM-generated data

• 1.000 ± 0.001 average AUC for DeepRC was nearly perfect on LSTM-generated data, while the MinMax-kernel SVM performed similarly across datasets.The data included noisy motifs implanted at frequencies from 10% to 0.05%; other competing methods performed worse at the 0.05% frequency.Source: Experimental Results · pp. 8–9

Results on real-world data with implanted motifs

• 0.980 ± 0.029 average AUC for DeepRC exceeded the burden test’s 0.883 ± 0.170 AUC on real-world data with implanted motifs.All methods except DeepRC struggled with noisy motifs at a 0.1% frequency.Source: Experimental Results · p. 9

Results on real-world data

• 0.831 ± 0.002 AUC made DeepRC the best-performing method on the CMV dataset, followed by the MinMax-kernel SVM at 0.825 ± 0.022 AUC.DeepRC’s top-ranked sequences significantly corresponded to sequences detected by Emerson et al. (2017) (Mann-Whitney U-test p-value 1.3 · 10^-93).Source: Experimental Results · p. 9

Impact on machine learning and related scientific fields

• With larger immunosequencing and benchmark datasets, confounder-aware analysis, and more computing power, DeepRC could identify disease-related immune receptor motifs and aid immunotherapies, immunodiagnostics, and vaccine design.The authors note potential relevance to SARS-CoV-2 research.Source: Broader Impact · p. 9
• Because most repertoire sequences lack known antigens, experimental verification and interpretation of predicted motifs remain important, while high-throughput antigen-labeled sequencing is beginning to address this limitation.Such data can link individual immune-receptor sequences to antigens.Source: Broader Impact · p. 9
• DeepRC’s application to repertoires with many instances per bag may encourage use of modern Hopfield networks and attention on previously unconsidered datasets and problems.Source: Broader Impact · p. 9

Impact on society

• If successful, blood-based immune-status testing could support faster disease diagnostics and targeted immunization, but compulsory screening could threaten privacy and personal self-determination.The authors also suggest screening resistant individuals could enable faster targeted immunization and reduce pandemic harms.Source: Broader Impact · pp. 9–10

Consequences of failures of the method

• Method failures could produce unsuccessful wet-lab validation, but the proposed pipeline includes follow-up testing and does not directly prescribe potentially harmful treatments.The authors expect wet-lab and in-vitro tests to indicate incorrect system decisions.Source: Broader Impact · p. 10

Leveraging of biases in the data and potential discrimination

• Age, sex, and uneven data coverage could bias predictions or produce unequal subgroup performance and treatment access, so target groups must be considered in subsequent in-vitro testing.Wet-lab failures may reveal such problems, while analysis of confounders could inform countermeasures.Source: Broader Impact · p. 10

• The appendix covers DeepRC and compared methods, experimental setup and results, LSTM-generated repertoire data, motif interpretation and extraction, and DeepRC variant ablations.Source: Appendix · p. 11

1 Immune Repertoire Classification

• The appendix documents DeepRC implementation details and the datasets used, including simulated, LSTM-generated, real-world implanted-signal, and CMV data.It also compares these datasets with other MIL datasets.Source: §1 · p. 12
• It describes the compared methods: known-motif, SVM, KNN, logistic regression, burden test, and logistic MIL.The contents also list hyperparameter selection and results.Source: §1 · p. 12
• The appendix includes results and a procedure for generating repertoires via an LSTM.Source: §1 · p. 12
• It includes an analysis of how DeepRC is interpreted.Source: §1 · p. 12
• It reports attention values for previously associated CMV sequences.Source: §1 · p. 12
• It also presents DeepRC variations and an ablation study.Source: §1 · p. 12

A1 Introduction

• Sections A2–A5 cover DeepRC’s architecture, datasets, comparison methods, and hyperparameter selection.Source: A1 Introduction · p. 13
• Section A6 presents detailed tabular results for each dataset category.Source: A1 Introduction · p. 13
• Sections A7–A10 cover antibody-sequence generation, interpretation, TCR attention correspondence, and DeepRC variations with an ablation study.Source: A1 Introduction · p. 13

Input layer

• Each amino acid is represented by 20 one-hot features and three positional features; CMV sequence features are scaled by log abundance before normalization.The positional features indicate proximity to the sequence start, center, and end, and sum to 1 at each position. Additional sequence- or repertoire-level metadata could be concatenated with CNN-derived features or used as auxiliary training targets.Source: A2 DeepRC implementation details · p. 13, Figure A1 · p. 14

1D CNN for motif recognition

• DeepRC uses a lightweight 1D CNN to recognize sequence motifs, then global max-pooling produces a fixed-length vector for each sequence.Kernel size is adjusted during hyperparameter search because it influences the length of motifs that can be extracted; the CNN uses SELU activations and 16-bit computations.Source: A2 DeepRC implementation details · pp. 13–14

Regularization

• During training, repertoires are randomly subsampled to 10,000 sequences, then reduced to the 10% with highest attention weights for updates and prediction.The authors describe random sequence subsampling as interpretable as dropout on input sequences or attention weights.Source: A2 DeepRC implementation details · p. 14

Hyperparameters

• DeepRC hyperparameters were selected by dataset-specific grid search within nested 5-fold cross-validation, with full searches limited to harder simulated datasets.For simulated data, the full search covered datasets with motif implantation probabilities below 1%; the average score across the five test folds was the final method score.Source: A2 DeepRC implementation details · p. 14

Computation time and optimization

• A batch size of 4 and 16-bit CNN computation reduced implementation demands; an RTX 2080 Ti update took about 0.0129–0.0135 seconds and used 8–11 GB GPU memory.The rest of the network used 32-bit values, and Adam’s numerical-stability parameter was increased from 10^-8 to 10^-4.Source: A2 DeepRC implementation details · pp. 14–15

Limitations

• DeepRC is limited by computationally demanding hyperparameter and model selection, as well as overfitting when real-world datasets contain relatively few repertoire samples.Many thousands of sequences per repertoire must be propagated and retained in GPU memory for attention and weight updates; the authors identify greater GPU memory and computational power as potential enablers of more advanced architectures.Source: A2 DeepRC implementation details · p. 15

A3 Datasets

• The benchmark comprises four dataset categories: simulated data, LSTM-generated data, and real-world data with implanted signals, plus real-world data without implanted signals.The first three categories include datasets with varying signal types and witness rates.Source: A3 Datasets · p. 15
• Implanted short motifs model immune signals because antigen interactions occur through short sequence stretches.Each sequence carries at most one implanted signal, and the fraction of signal-bearing sequences is the witness rate.Source: A3 Datasets · p. 15
• Binary classification simulates healthy and diseased status by implanting motifs in positive repertoires, whose sequences are generated by different procedures.Repertoires vary in sequence count; sequences use 20 amino-acid characters and average 14.5 amino acids in length.Source: A3 Datasets · p. 15

A3.1 Simulated immunosequencing data

• Each dataset has 2,500 repertoires per class, with sequence counts and lengths sampled to match distributions observed in the real-world CMV dataset.Repertoires below 5,000 sequences were resampled; the sequence-count distribution used N(µ = 316k, σ = 132k), and sequence lengths used N(µ = 14.5, σ = 1.8).Source: A3.1 Simulated immunosequencing data · p. 16
• The positive class received motifs averaging 4 amino acids, with datasets varying motif implantation probability (witness rate), central wildcard positions, and central deletion positions.Each deletion position had a 0.5 probability of removal, while wildcard amino acids were sampled randomly for each sequence.Source: A3.1 Simulated immunosequencing data · pp. 15–16
• The 18 simulated datasets contain roughly 28.7 billion sequences and vary in difficulty according to implanted-motif properties.The paper refers to Table A1 for an overview of those properties.Source: A3.1 Simulated immunosequencing data · p. 16

A3.2 LSTM-generated data

• The dataset category tested how signal frequency and complexity interact with more plausible immune-receptor sequences.The LSTM-generation approach incorporated positional amino-acid distributions and other sequence properties to make the sequences more experimentally plausible.Source: A3.2 LSTM-generated data · p. 16
• The LSTM generated immune-receptor-like amino-acid sequences by learning positional distributions and amino-acid relationships and order from CMV immunosequences.It was trained for next-character prediction (Graves, 2013), then used autoregressively to generate 1,000 repertoires whose 4-mer frequencies were visually judged sufficiently similar to real sequences.Source: A3.2 LSTM-generated data · p. 16
• The five datasets contained 500 positive and 500 negative repertoires each, with length-4 motifs implanted centrally in positive sequences.Each positive-class sequence carried the motif with probability ρ, varied across datasets; each motif position was implanted with probability 0.9, leaving the original amino acid with probability 0.1.Source: A3.2 LSTM-generated data · p. 16

A3.3 Real-world data with implanted signals

• Implanted-signal datasets used 750 repertoires per class with 10k sequences each, varying single or multiple motifs and signal frequencies of 1% or 0.1%.Motifs were implanted with positional bias; the four variations used experimentally obtained sequences sampled from CMV-negative individuals.Source: A3.3 Real-world data with implanted… · pp. 16–17

A3.4 Real-world data: CMV dataset

• The CMV dataset analysis included 686 repertoires: 312 CMV-positive and 374 CMV-negative.It was drawn from Emerson et al. (2017), excluding repertoires with unknown CMV status or unknown sequence-abundance information.Source: A3.4 Real-world data: CMV dataset · p. 17

A3.5 Comparison to other MIL datasets

• The compared immunosequencing datasets were described as posing the most challenging MIL problems to date in instances per bag.This comparison concerns the number of instances per bag, not predictive performance.Source: A3.5 Comparison to other MIL datasets · p. 17, Table A2 · p. 18

A4 Compared methods

• DeepRC was compared with known-motif scoring, k-mer SVM and KNN, logistic regression, a burden test, and logistic MIL baselines.Known-motif scoring estimates performance when the implanted motif is known but cannot be applied to the CMV dataset; the burden test selects associated sequences or k-mers on validation data, while logistic MIL max-pools per-k-mer scores.Source: A4 Compared methods · p. 19, A4.1 Known motif · p. 19, A4.2 Support Vector Machine (SVM) · p. 19

A7 Repertoire generation via LSTM

• LSTM-generated repertoires had amino-acid and 4-mer distributions similar to those of the real-world CMV dataset.The study generated 1,000 repertoires and assigned 500 each to positive and negative classes before implanting motifs in the positive class.Source: A7 Repertoire generation via LSTM · p. 26, Figure A2 · p. 27

A8 Interpreting DeepRC

• Integrated Gradients recovered implanted motifs in simulated datasets and revealed complex, variable patterns in the center regions of CMV sequences.DeepRC supports sequence attention weights and contribution analysis; the reported motif analysis averages contributions across validation- and test-set repertoires to reduce overfitting artifacts.Source: A8 Interpreting DeepRC · p. 28, Table A13 · p. 28, Figure A3 · p. 29

A9 Attention values for previously associated CMV sequences

• Previously CMV-associated TCRβ sequences received significantly higher DeepRC attention values than other sequences (p-value 1.3e-93).The table reports attention-value quantiles across all sequences in the dataset.Source: Table A14 · p. 31, A9 Attention values for previously… · p. 31

A10 DeepRC variations and ablation study

• LSTM-based sequence embeddings generalized slightly better than CNN-based embeddings, while DeepRC remained robust across tested hyperparameter settings.The variation study estimated AUC on only 3 of 5 folds and used fewer CNN kernel counts, slightly reducing performance relative to the full procedure.Source: A10 DeepRC variations and ablation study · p. 32
• DeepRC with LSTM sequence encoding was evaluated every 5 · 10^3 updates, with early stopping based on the best validation-fold loss after 2 · 10^5 updates.The stopping hyperparameter was selected using validation-fold loss.Source: Table A16 · p. 33, A10 DeepRC variations and ablation study · p. 33
• For LSTM encoding, Table A17 summarizes mean and standard deviation of ROC AUC over the first three folds of five-fold nested cross-validation across hyperparameter subsets.The subsets included a full search and searches fixing attention beta, head count, or LSTM-block count.Source: Table A17 · p. 33
• For 1D CNN encoding, Table A18 summarizes mean and standard deviation of ROC AUC over the first three folds of five-fold nested cross-validation across hyperparameter subsets.The subsets included a full search and searches fixing attention beta, head count, kernel size, or CNN kernel count.Source: Table A18 · p. 34