Hopfield Networks is All You Need

Motivation

• Modern Hopfield networks extend associative-memory models used for storing and retrieving patterns, addressing the need for alternatives to recurrent-network memories.Prior approaches include external memories, memory networks, fast-weight methods, and attention-based architectures such as transformers.Source: §1 · pp. 1–2

Contributions

• The work introduces deep-learning layers equipped with modern Hopfield memory and a differentiable update rule that typically retrieves patterns after one update.Differentiability supports gradient-descent parameter updates, while one-step retrieval is compatible with activating layers in deep networks.Source: §1 · p. 2

Continuous modern Hopfield networks

• The authors generalize exponential-interaction Hopfield energies to continuous patterns and states and propose an update rule with global convergence to stationary points.The stationary points may be local minima or saddle points, and the method typically retrieves patterns within epsilon of a fixed point with exponentially low error.Source: §1 · p. 2

Connection to transformers

• The proposed update rule minimizes the energy while preserving fast convergence and storage-capacity properties, and it is equivalent to transformer attention.This equivalence is illustrated as the connection between the new Hopfield update and the transformer’s attention mechanism.Source: Figure 1 · p. 2, §1 · p. 2

Continuous-state energy and convergence

• The proposed continuous-state energy remains bounded, and its update rule converges to stationary points, including local minima or saddle points.The quadratic state term keeps the state norm finite, while the convergence results show that iterations either converge or have connected, compact limit sets of stationary points.Source: §2 · pp. 3–4

Storage capacity

• For randomly chosen fixed-norm patterns on a sphere, the model stores exponentially many patterns in dimension d, with storage guaranteed with probability 1 − p under stated conditions.A stated parameter setting proves c ≥1.3718 for β = 1, K = 1, d = 75, and p = 0.001, with a + ln(b) < −0.94.Source: §2 · p. 4

Pattern retrieval

• After one update, well-separated patterns are retrieved near their fixed points, with distance and retrieval error decreasing exponentially with separation ∆i.The one-update guarantee is important for integrating the network into layers activated only once.Source: §2 · p. 4

Metastable states and global fixed point

• When patterns are not individually separated, the dynamics yield one global fixed point near their arithmetic mean or metastable states near similar, mutually separated subsets.The corresponding softmax weights are nearly uniform for the global fixed point, while initializations near similar patterns converge to the associated metastable state.Source: §2 · p. 5

Attention equivalence

• With suitable key, query, value, and scaling projections, the Hopfield update rule is equivalent to transformer and BERT attention.The correspondence uses β = 1/√dk and maps Hopfield patterns and states to the transformer matrices K, Q, and V.Source: §2 · p. 5

• Equation (10)’s left-hand expression is the transformer attention mechanism.Source: Y WKWV . (10) · p. 5
• For transformer self-attention, R = Y and WKWV is replaced by WV.Source: Y WKWV . (10) · p. 5
• The right-hand expression explains Hopfield-specific layers that add functionalities beyond attention in deep network architectures.Source: Y WKWV . (10) · p. 5

Layer integration

• Hopfield layers integrate continuous, differentiable modern Hopfield networks into deep architectures, where one-update retrieval equips layers with associative memories.The framework introduces Hopfield, HopfieldPooling, and HopfieldLayer.Source: §3 · p. 5
• Hopfield layers can replace pooling, permutation-equivariant, GRU, LSTM, and attention layers in deep networks.They support applications including multiple instance learning, point sets, permutation-invariant learning, attention, associative memories, language processing, sequence analysis, and reference-data retrieval.Source: §3 · p. 5

HopfieldLayer

• HopfieldLayer associates two vector sets and can query training, reference, prototype, or learned sets to realize SVM, k-nearest-neighbor, and LVQ models.Its output preserves the number of input vectors while supporting multiple queries of the stored set.Source: Figure 3 · p. 6, Figure 5 · p. 7

HopfieldPooling

• HopfieldPooling searches learned static queries against stored key patterns, averaging similar patterns to summarize sets and support multiple instance learning.Position encoding can enable pooling operations analogous to convolutional filters, and the layer was used for immune repertoire classification experiments.Source: §3 · pp. 6–7, Figure 4 · p. 7

Additional functionality

• Modern Hopfield layers provide multiple updates, β-controlled averaging over patterns, dimension-controlled storage capacity, and pattern normalization controls.β influences metastable-state size and learning dynamics, while associative-space dimension matters for tasks with very large instance sets.Source: §3 · pp. 7–8

4 EXPERIMENTS

• Transformer and BERT attention heads predominantly occupy metastable states, with very-large-pattern averaging common in lower layers, large or small metastable states in middle layers, and medium states predominant in final layers.The metastable-state size is estimated by the minimum number k of softmax values summing to 0.90, with four classes defined by thresholds relative to the sequence length N.Source: §4 · p. 8
• Hopfield layers outperformed competing methods on Tiger, Elephant, and UCSB Breast Cancer, and DeepRC achieved the highest average AUC across immune-repertoire datasets.On immune repertoire classification, DeepRC achieved AUC 0.832 ± 0.022 versus 0.825 ± 0.022 for an SVM with MinMax kernel and 0.699 ± 0.041 for the burden test.Source: §4 · p. 9, Table 1 · p. 9
• HopfieldPooling compresses variable-sized multiple-instance bags by averaging class-indicative embedded instances selected through a trainable fixed query pattern.This mechanism supports tasks where relevant patterns must be extracted from large instance sets, including immune repertoires containing approximately 300,000 instances per repertoire.Source: §4 · pp. 8–9
• Hopfield networks outperformed all compared methods on 75 small UCI datasets, setting a new state-of-the-art for 10 datasets with significant differences except against the first three runner-ups.The comparison used method groups and representatives spanning neural networks, random forests, support vector machines, boosting, bagging, and other methods.Source: §4 · pp. 9–10, Table 2 · p. 9
• HopfieldLayer reached state-of-the-art on SIDER side-effect prediction with 0.672 ± 0.019 and BACE β-secretase prediction with 0.902 ± 0.023.The architecture stores training data as patterns, uses the input as the state pattern, and projects the Hopfield-layer output to labels.Source: §4 · p. 10

A APPENDIX

• The appendix contains the continuous-state Hopfield theory, mathematical background, binary-network review, transformer-attention equivalence, experiment details, and PyTorch implementation guidance.Its six sections cover the new energy and update rules, convergence and fixed points, associations and forgetting, attention, four experiments, and layer functionality and usage.Source: A APPENDIX · p. 11

CONTENTS OF THE APPENDIX

• The appendix contents organize the material into continuous-state theory, binary Hopfield networks, transformer attention, four experiments, and PyTorch implementation subsections.The listed experiments cover attention in transformers, multiple instance learning, small UCI benchmark classification, and drug-design benchmarks; implementation sections cover introduction, functionality, and usage.Source: CONTENTS OF THE APPENDIX · p. 11

LIST OF THEOREMS

• The theorem list covers global convergence, stationary-point characterization, storage capacity for placed and random patterns, one-update retrieval, and exponentially small retrieval error.It also includes a binary modern Hopfield storage-capacity theorem.Source: LIST OF THEOREMS · pp. 11–12

LIST OF DEFINITIONS

• The mathematical background defines softmax, log-sum-exp, convex conjugates, the Legendre transform, epi-sum, and the Lambert function.These definitions support the proofs developed for the continuous-state network.Source: LIST OF DEFINITIONS · p. 12

A.1.1 INTRODUCTION

• The continuous-state network stores and retrieves exponentially many random patterns in the Hopfield-space dimension, typically with one update and exponentially small retrieval error.The introduction also covers learning associations, transformer and BERT architectures, and temporal forgetting that supports infinite memory capacity and controlled forgetting.Source: §A.1.1 · p. 13
• The appendix reviews prior modern Hopfield networks as binary-state systems and contrasts them with the proposed continuous-state formulation.It also provides background on associative-memory networks, external memory, attention mechanisms, and binary modern Hopfield energy and update rules.Source: §A.1.1 · p. 13

A.1.2 NEW ENERGY FUNCTION

• The new energy generalizes Demircigil et al. (2017) from binary to continuous states while retaining log-sum-exp properties associated with high storage capacity and fast convergence.The related binary models include Krotov and Hopfield (2016), whose storage capacity is proportional to d^(n−1), and Demircigil et al. (2017), whose exponential interaction yields N = 2^(d/2) for binary patterns.Source: §A.1.2 · p. 13

A.1.3 NEW UPDATE RULE

• The update rule maps any previous state into the simplex defined by the patterns and replaces the classical interaction vector with softmax(βX^Tξ).The update rule is presented as minimizing the new energy function.Source: §A.1.3 · p. 15
• The update rule ensures global convergence: every limit point of an generated sequence is a stationary point, namely a local minimum or saddle point, of the energy.The subsequent local-convergence analysis shows that stored patterns are retrieved with one update.Source: §A.1.3 · p. 15

• As a CCCP step for minimizing E, the update rule monotonically decreases the energy over time.CCCP minimizes an energy decomposed into convex and concave parts, and the cited convergence results guarantee monotonic decrease.Source: §A.1.4 · p. 16
• The update rule converges globally: E(ξ_t) → E(ξ*) for t → ∞, where ξ* is a fixed point.Source: §A.1.4 · p. 16
• Global convergence does not guarantee convergence to a local minimum, although local maxima are almost surely avoided because the energy strictly decreases away from stationary points.Source: §A.1.4 · pp. 16, 18
• The auxiliary CCCP optimization is strictly convex with a unique global minimum, supported by a strictly positive-definite Hessian and a quadratic-form representation.Source: §A.1.4 · p. 18
• Every sequence generated by the iteration has energy converging to a stationary point, while its limit points form a connected compact subset of the corresponding stationary level set when needed.Source: §A.1.4 · p. 18

Jacobian and contraction analysis

• A symmetric Jacobian bound establishes contraction-based local convergence, with Banach’s theorem supplying uniqueness and convergence when the bound is below one.The analysis uses the softmax Jacobian, spectral-norm estimates, and contraction mappings to control the fixed-point iteration and convergence rate.Source: §A.1.5 · pp. 19, 21, 22

General convergence behavior

• Pattern separation determines whether iterations converge to a global mean, an individual stored pattern, or a metastable state averaging similar patterns.Poorly separated patterns yield a fixed point near the arithmetic mean; well-separated patterns retrieve the pattern nearest the initialization; partially clustered patterns produce metastable averages.Source: §A.1.5 · p. 19

One Stable State: Fixed Point Near the Mean of the Patterns

• When no pattern is well separated, the iteration has a unique global fixed point near the arithmetic mean under a contraction condition.The sufficient condition is expressed through a Jacobian norm bound based on pattern variance and spread; variations outside the largest-variance direction weaken the bound.Source: §A.1.5 · pp. 21–23
• If the data are centered, the arithmetic mean is itself a fixed point, and normalization methods such as batch normalization promote this fixed point.After one update, the state lies in the simplex of the patterns; when mx = 0, substituting ξ = mx yields the mean as a fixed point.Source: §A.1.5 · p. 23

Many Stable States: Fixed Points Near Stored Patterns

• For well-separated patterns, initialization near a stored pattern converges to that pattern, with the corresponding softmax vector approaching its one-hot representation.This many-stable-state regime is analyzed using Banach’s theorem and Jacobian-norm estimates.Source: §A.1.5 · p. 26

Single-pattern fixed points

• Under separation assumption (A2), the update maps the sphere Si into itself, while the contraction condition ensures a fixed point exists in Si.The proof applies Banach’s Fixed Point Theorem after establishing invariance and contraction.Source: §A.1.5 · pp. 28, 30
• Every point in a suitable environment converges under the iteration to the fixed point x*_i, provided the iteration remains in that environment.The map is a contraction there, and each iteration moves the mapped point closer to x*_i.Source: §A.1.5 · p. 32
• For large separation ∆i, the iteration is already close to the fixed point after one update.This behavior is reported as confirmed in several experiments.Source: §A.1.5 · p. 32

Metastable states

• For metastable states, the Jacobian is a covariance matrix, so convergence depends on keeping its largest singular value, equivalently the covariance’s largest eigenvalue, small.This variance interpretation concerns the probability-weighted selection of patterns.Source: §A.1.5 · pp. 32–33
• The covariance becomes small when one pattern dominates, all patterns are selected equally, or a few sufficiently similar patterns receive high probability.For sufficiently similar patterns, the covariance spectral norm is smaller than one.Source: §A.1.5 · p. 33

Invariant sphere

• The iteration maps the compact sphere S_m into itself when the mean m_x is sufficiently dissimilar from patterns outside the dominant subset.This establishes the invariant environment required before applying Banach’s theorem.Source: §A.1.5 · pp. 37, 40

Distance bound

• For queries near m_x, the distance to the mapped point is bounded by ∥m_x − f(ξ)∥ ≤ m_max + 2γM ≤ m_max + 2ϵM.Here M bounds pattern norms, γ is the probability of selecting outside the first l patterns, and ϵ upper-bounds γ.Source: §A.1.5 · p. 37

Contraction

• The update is a contraction on S_m under the stated separation and radius conditions, enabling convergence to the fixed point.The contraction claim is established through a bound on the Jacobian and the associated assumptions.Source: §A.1.5 · pp. 40, 43

Fixed point existence

• Banach’s Fixed Point Theorem guarantees a fixed point in the sphere S_m when the dominant-pattern center is sufficiently separated from other data and the within-group radius is sufficiently small.The proof combines self-mapping of S_m with contraction of f on S_m.Source: §A.1.5 · pp. 40, 43

A.1.6 PROPERTIES OF FIXED POINTS NEAR STORED PATTERN

• Small pattern separation increases storage capacity but slows convergence and raises retrieval error, whereas large separation enables one-step retrieval with exponentially low error.The fixed-point analysis assumes disjoint attraction spheres around patterns and uses separation or minimum angular distance to establish separate fixed points.Source: §A.1.6 · pp. 44–46

Storage capacity for patterns placed on the sphere.

• Placed patterns on a sphere can be stored as fixed points with attraction basins near the patterns, including guarantees for M = 2 when d ≥50 and M = 5 when d ≥13.Source: §A.1.6 · pp. 46–47

Storage capacity for random patterns on the sphere.

• For random sphere patterns, the analysis bounds the minimum pairwise angle and derives storage guarantees by combining tail bounds, geometric estimates, and a Lambert W solution.The random-pattern argument uses results on minimum angles from Cai et al. (2013) and Brauchart et al. (2018), together with bounds on κ_d and cos.Source: §A.1.6 · pp. 48–50

Exponential storage capacity: the base c as a function of the parameter β, the radius of the sphere M, the probability p, and the dimension d of the space.

• With probability 1 −p, randomly chosen sphere patterns can be stored in exponentially many patterns, with base c constrained by β, radius K, dimension d, and failure probability p.The main random-pattern theorem uses the Lambert W function to derive the admissible base c; examples prove c ≥3.1546 for β = 1, K = 3, d = 20, p = 0.001, and c ≥1.3718 for β = 1, K = 1, d = 75, p = 0.001.Source: §A.1.6 · p. 49

A.1.6 PROPERTIES OF FIXED POINTS NEAR STORED PATTERN

• Theorem A6 gives exponential storage capacity for randomly chosen spherical patterns, with dimension constrained by β, radius M, failure probability p, and exponential base c.The bound is expressed using the Lambert W function, with branches W0 or W−1 selected according to the sign of a.Source: §A.1.6 · p. 52
• A separate storage-capacity theorem treats expected minimal separation rather than the probability that all patterns are stored, yielding a lower bound under a stated condition; for β = 1, K = 3, c = 2, it holds when d ≥17.This formulation uses randomly chosen patterns on a sphere and concerns the expected minimal separation.Source: §A.1.6 · pp. 54–55
• For well-separated patterns, one update retrieves a stored pattern by bringing the updated state exponentially close to its fixed point.The contraction bound uses the Jacobian and decreases exponentially with pattern separation and, for random vectors, with dimension.Source: §A.1.6 · p. 56
• Theorem A9 bounds the retrieval error of pattern xi by a quantity that decreases exponentially with the separation ∆i.The result applies when the query and fixed point satisfy the stated β- and M-dependent distance conditions.Source: §A.1.6 · p. 57

A.1.7 LEARNING ASSOCIATIONS

• Learning associations covers raw-pattern association, mapping one pattern set into the other or learning an association matrix, and mapping both sets into a common associative space.The corresponding query and stored patterns are formed using identity mappings, W or W^T, an association matrix, or separate WQ and WK mappings.Source: §A.1.7 · p. 57
• Learnable associations map raw state and stored patterns into an associative space, yielding equivalent update rules and derivatives for integrating Hopfield layers into deep networks.The framework covers one-sided mappings through W and two-sided query/key mappings through WQ and WK, with simplified and full update rules.Source: §A.1.7 · pp. 57–59

A.1.8 INFINITE MANY PATTERNS AND FORGETTING PATTERNS

• Causal masking enables new Hopfield networks to process infinitely many time-indexed patterns for autoregressive tasks, while forgetting adds increasingly negative softmax values to older patterns.Future entries are masked by a large negative value, and the forgetting energy uses a parameter γ that increases penalties with temporal distance.Source: §A.1.8 · pp. 60–61

A.1.9 NUMBER OF SPURIOUS STATES

• The energy analysis shows that spurious states correspond to modes of the associated mixture, which can exceed the number N of stored patterns.Because the negative exponential is monotonic, energy minima correspond to maxima of the relevant mixture; Carreira-Perpiñán & Williams (2003) established that more than N modes can occur.Source: §A.1.9 · p. 62

A.2 PROPERTIES OF SOFTMAX, LOG-SUM-EXPONENTIAL, LEGENDRE TRANSFORM, LAMBERT W FUNCTION

• Softmax is the gradient of log-sum-exp, is monotone for β > 0, and has a symmetric positive-semidefinite Jacobian with a zero eigenvalue.The Jacobian satisfies ∥Js∥2 ⩽ 2ϵβ − 2ϵ^2β < 2ϵβ when pmax ≥ 1 − ϵ ≥ 0.5, while softmax is (β/2)-Lipschitz and becomes contractive when β < 2m.Source: §A.2 · pp. 63–65
• Softmax p = softmax(βx) is 1/(2mβ)-cocoercive on U, with m = maxx∈U maxi pi(1 −pi).The bound follows from the stated inequality relating softmax differences and input differences.Source: §A.2 · p. 66
• Everywhere, softmax is (2/β)-cocoercive; on U with pmax = minx∈U maxi pi = 1 −ϵ, it is 1/(2βϵ)-cocoercive.These specialize the general bound using m ≤ 1/4 globally and m ≤ ϵ on U.Source: §A.2 · p. 66
• The section defines convex conjugates, Legendre transforms, and epi-sums, then states convex-conjugate identities for squared norms, scalar scaling, additive constants, and affine argument transformations.The Legendre transform is presented as a more restrictive version of the convex conjugate.Source: §A.2 · p. 66

5. Convex Conjugate of epi-sums

• The Legendre transform of log-sum-exponential is negative entropy restricted to the probability simplex, and the converse transform recovers log-sum-exponential.These duality relations are stated as Lemma A28 and justified using convex conjugacy and Fenchel–Moreau results.Source: §5 · pp. 67–68
• For nonsingular XXT, the conjugate analysis characterizes when the log-sum-exponential supremum is solvable and shows it is unbounded outside the relevant simplex domain.The derivation uses the Legendre-transform definition and the boundedness conditions for the maximization problem.Source: §5 · p. 68
• The appendix establishes convexity and continuity of lse(β, XTξ), using positive semidefiniteness of diag(p) − ppT and the biconjugacy of lower-semicontinuous convex functions.The supporting arguments invoke Fenchel–Moreau and related convex-analysis propositions.Source: §5 · p. 69
• The appendix introduces Lambert W and its branches and identities, along with a vector-valued mean value theorem, as tools for subsequent capacity bounds and proofs.Lambert W is defined as the inverse of f(y) = yey, while the mean value theorem expresses changes through an integrated Jacobian.Source: §5 · pp. 69–70

A.3.1 MODERN HOPFIELD NETWORKS: INTRODUCTION

• Modern Hopfield networks extend neural-network memory through external storage, differentiable retrieval, associative memory, and attention mechanisms developed in neural Turing machines, memory networks, and transformers.The introduction traces this progression from writable and readable external memory to softmax-based retrieval and transformer attention.Source: §A.3.1 · pp. 70–71
• Classical binary Hopfield networks have limited storage capacity, despite exponentially many energy minima, motivating modern constructions with improved energy functions.The standard model stores at most order-d patterns without errors under the cited conditions, while prior nonlinear interactions increase capacity.Source: §A.3.1 · p. 71
• Exponential interactions yield exponential storage capacity with all stored patterns as fixed points, although the attraction radius vanishes; convergence can occur with high probability after one update.These properties are attributed to prior binary modern Hopfield-network results.Source: §A.3.1 · p. 71

A.3.2 ENERGY AND UPDATE RULE FOR BINARY MODERN HOPFIELD NETWORKS

• The binary model stores patterns x_i with components x_ij ∈ {−1, +1}, and its energy generalizes the classical Hopfield network through an interaction function F(a) = a^n.For n = 2, the energy reduces to that of the classical Hopfield network and supports storage of αnd^(n−1) patterns.Source: §A.3.2 · pp. 71–72
• With F(a) = exp(a), the binary modern Hopfield network stores N = exp(αd) + 1 patterns, exponential in dimension d, and recovers perturbed patterns with high probability after one update.The retrieval guarantee applies for random binary patterns and perturbations within a Hamming sphere, with α constrained by the attraction radius ϱ.Source: §A.3.2 · p. 72
• The exponential interaction analysis extends earlier polynomial-energy models by showing exponential storage capacity and high-probability single-update retrieval for binary modern Hopfield networks.The polynomial formulation includes the classical Hopfield case at n = 2 and was associated with Krotov and Hopfield.Source: §A.3.2 · p. 72
• The storage-capacity condition trades off the number of stored patterns against the radius of attraction ϱd, while the retrieval error probability converges to zero.The theorem states that all stored patterns become fixed points with probability converging to one as d →∞ under the stated asymptotic condition.Source: §A.3.2 · p. 72

A.4 HOPFIELD UPDATE RULE IS ATTENTION OF THE TRANSFORMER

• The Hopfield update rule is the transformer and BERT attention mechanism when stored patterns serve as keys, state patterns as queries, and learned projections define K, Q, and V.Setting β = 1/√d_k and using row-vector softmax yields the transformer attention form after multiplying the update by W_V.Source: §A.4 · p. 73

Y WKWV . (548)

• The left side of Eq. (548) is transformer attention, while the right side provides the starting point for specialized Hopfield layers.These layers extend deep architectures with additional Hopfield-network functionalities beyond attention.Source: Y WKWV . (548) · p. 73
• The continuous-state generalization preserves fast convergence and storage-capacity properties while introducing an energy-minimizing update rule equivalent to transformer attention.Figure A.2 summarizes the correspondence between the generalized Hopfield energy, its update rule, and transformer softmax notation.Source: Figure A.2 · p. 73

A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS

• Pre-trained BERT heads predominantly perform global averaging in lower layers, large or small metastable-state averaging in middle layers, and medium metastable-state averaging in final layers.Class I is abundant in lower layers; Classes II and IV often occur in middle layers, while Class III predominates in the last layers.Source: §A.5.1 · p. 73, Figure A.3 · p. 74

A.5 EXPERIMENTS

• The experiment used a computationally reduced BERT-small model with 12 layers, 4 heads, sequence length 128, hidden dimension 256, and embedding dimension 128.Training for 1.45 million update steps took roughly four days on one NVIDIA V100 GPU.Source: §A.5.1 · p. 73
• Pre-training aimed to reproduce Devlin et al. (2019) using English Wikipedia and an uncased snapshot of BookCorpus because the original dataset was no longer publicly available.The implementation used the Hugging Face transformers repository.Source: §A.5.1 · p. 75

A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS

• Class-IV attention heads commit early to small metastable states, often changing operation mode around 9,000–10,000 training steps alongside the second major loss drop.After this transition, Class-IV heads remain almost unchanged during the rest of training.Source: §A.5.1 · p. 75, Figure A.5 · p. 77
• Gaussian averaging heads replace input-conditioned softmax attention with learned discrete Gaussian kernels using location and scale parameters.The proposal targets lower layers where many heads average over very large numbers of patterns, making fine-grained input-dependent position selection unnecessary.Source: §A.5.1 · p. 75
• The Gaussian-head initialization samples locations uniformly from [−1, 1] and scales from [0.75, 1.25], with supporting-point initialization showing no practical difference from random initialization.The supporting points are equidistant positions over [−1, 1].Source: §A.5.1 · p. 75, Figure A.6 · p. 78
• Gaussian averaging uses 2·N parameters per head, versus 2·dk·dy for standard self-attention; for BERT-base, this means 95.5 times more parameters for the original attention mechanism.For BERT-base, dy = 768, dk = 64, and N = 512; Gaussian heads are less expressive because they are input-independent.Source: §A.5.1 · pp. 75, 78

A.5.2.1 Immune Repertoire Classification

• The immune repertoire experiment covered four dataset categories, with approximately 300,000 instances per repertoire in three categories and over 30 billion sequences overall.The datasets combined experimentally observed receptors, simulated motifs, LSTM-generated sequences, and real-world immune-status data.Source: §A.5.2 · p. 78
• 0.832 ± 0.022 AUC was achieved by DeepRC on the real-world dataset, versus 0.825 ± 0.022 for the SVM with MinMax kernel and 0.699 ± 0.041 for the burden test.DeepRC also uses a modern Hopfield network to extract informative patterns from repertoires containing large numbers of immune receptor sequences.Source: §A.5.2 · pp. 78–79
• DeepRC outperformed all competing methods in average AUC across all four immune repertoire dataset categories.Table A.1 summarizes AUC results across simulated, LSTM-generated, real-world implanted-signal, and real-world immune-status datasets.Source: Table A.1 · p. 79, §A.5.2 · p. 79

A.5.2.2 Multiple Instance Learning Benchmark Datasets

• HopfieldPooling averages class-indicative embedded instances into fixed-sized bag representations for classification on the Elephant, Fox, Tiger, and UCSB breast cancer MIL datasets.The layer uses trainable fixed query patterns and was evaluated with nested cross-validation, while bag dropout addressed overfitting.Source: §A.5.2 · pp. 79–80

A.5.3 EXPERIMENT 3: CLASSIFICATION ON SMALL UCI BENCHMARK DATASETS

• Small UCI datasets are difficult for neural networks to generalize on, motivating Hopfield-based retrieval of indicative samples or learned class prototypes.The Hopfield layer receives representations produced by a self-normalizing network, with the number of stored fixed patterns treated as a hyperparameter.Source: §A.5.3 · p. 81

11. Other Ensembles

• Modern Hopfield networks are compared with 17 groups of methods, including neural networks, random forests, gradient boosting, and support vector machines.The comparison follows method groupings reported by Fernández-Delgado et al. (2014) and Klambauer et al. (2017a).Source: §A.5.3 · p. 81, §11 · p. 81

17. Stacking (Wolpert)

• The Hopfield architecture optionally uses 0, 1, or 7 fully connected embedding layers with SELU activations and 32, 128, or 1024 hidden units.These embeddings feed a Hopfield layer with 1 or 32 heads before class prediction and softmax normalization.Source: §17 · p. 81
• Each dataset uses grid search over model and hyperparameter settings, with 100 training epochs, minibatches of four, cross-entropy loss, and early stopping based on validation accuracy.The selected model is evaluated on a separated test set, with results reported in Table 2 and the supplemental detailed-results file.Source: Table A.3 · p. 82
• The search space makes the Hopfield association-space dimension equal to the embedding hidden-unit count, while the stored-pattern count depends on each task’s number of target classes.The hyperparameter search-space and training protocol are summarized in Table A.3.Source: §17 · p. 81, Table A.3 · p. 82

A.5.3 EXPERIMENT 3: CLASSIFICATION ON SMALL UCI BENCHMARK DATASETS

• The study compares modern Hopfield networks with 25 methods across 75 small UCI datasets using average rank derived from per-dataset test accuracy.The datasets contain fewer than 1,000 samples, and baseline scores come from Klambauer et al. (2017a).Source: §17 · pp. 81–82

A.5.4.1 Experimental design and implementation details

• The benchmark comprised HIV, BACE, BBBP, and SIDER datasets from MoleculeNet, covering four drug-discovery prediction tasks.These tasks target new antivirals, protein inhibitors, metabolic effects, and chemical-compound side effects; SIDER contains 1427 approved drugs and BBBP concerns blood-brain barrier permeability.Source: §A.5.4 · p. 82
• The HopfieldLayer stored training inputs as keys, training labels as values, and used each input as the query state.By concatenating input z_i and target t_i, W_K and W_V can route the input inside the softmax and the target outside it.Source: §A.5.4 · p. 82
• Models were selected by highest validation AUC across five random splits using separate validation sets.The accompanying grid search trained applicable models for four epochs with Adam and batch size one.Source: §A.5.4 · p. 82, Table A.4 · p. 82

A.5.4.2 Results

• 0.672 ± 0.019 on SIDER and 0.902 ± 0.023 on BACE established state-of-the-art performance for side-effect and β-secretase prediction, respectively.Comparisons included SVMs (Cortes & Vapnik, 1995; Schölkopf & Smola, 2002), XGBoost (Chen & Guestrin, 2016), Random Forest (Breiman, 2001), DNNs (LeCun et al., 2015; Schmidhuber, 2015), GCNs (Kipf & Welling, 2016), GATs (Veli˘ckovi´c et al., 2018), MPNNs, and Attentive FP.Source: §A.5.4 · p. 83, Table A.5 · p. 83

• The implementation targets PyTorch and supports applications including multiple instance learning, point-set processing, permutation-invariant learning, attention, natural language processing, sequence analysis, and time-series prediction.It can also store reference or experienced data, such as training data or reinforcement-learning experiences, for later retrieval.Source: §A.6.1 · p. 83
• Hopfield layers implement memory with associative retrieval, conditional pooling and averaging, association-based data combination, associative credit assignment, and attention mechanisms.Source: §A.6.1 · p. 83
• A Hopfield layer can substitute attention layers in transformer and BERT architectures and replace pooling, permutation-equivariant, GRU, LSTM, and attention layers.Source: §A.6.1 · pp. 83–84
• Unlike classical Hopfield networks, the layer uses modern continuous-state networks with increased storage capacity and energy-minimizing dynamics for individual heads.Source: §A.6.1 · p. 84

A.6.2 FUNCTIONALITY

• Hopfield layers add association of two sets, multiple updates, variable β, configurable associative-space dimension, static patterns, and pattern normalization.These controls govern fixed-point behavior, storage capacity, pattern search, and convergence dynamics.Source: §A.6.2 · p. 84
• Associating separate query and stored-pattern sets supports self-attention, encoder-decoder attention, sequence analysis, multiple-instance learning, point-set learning, memory, and pooling.Raw state and stored vectors are projected into a shared associative space through WQ and WK.Source: §A.6.2 · p. 84
• Multiple updates can be applied per head without changing parameter count, with individually specified update counts or convergence thresholds based on state-change norms.After the final update, states are projected through WV to produce result patterns.Source: §A.6.2 · p. 85, Figure A.7 · p. 87
• β, pattern norms, and pattern centers control fixed-point dynamics, gradient flow, metastable-state size, and convergence, with normalization offering three configuration choices.The choices are input normalization, normalization after associative-space projection, or no normalization.Source: §A.6.2 · pp. 85–86
• The associative-space dimension trades exponential storage capacity against metastable-state size, so it should match the number of stored patterns and desired averaging scope.Projecting low-dimensional inputs into a higher-dimensional feature space can support proper fixed-point dynamics.Source: §A.6.2 · p. 85
• Static query or stored patterns enable fixed-pattern search, including identifying particular patterns in data such as immune repertoires.Static patterns are determined by bias weights and remain constant across network inputs.Source: §A.6.2 · pp. 85–86

A.6.3 USAGE

• With core functionalities disabled, HopfieldEncoderLayer and HopfieldDecoderLayer provide one-to-one plug-in replacements for PyTorch TransformerEncoderLayer and TransformerDecoderLayer.The interface exposes association, β, maximum updates, associative-space dimension, static patterns, and pattern normalization.Source: §A.6.3 · p. 86
• Hopfield layers can implement pooling, permutation-equivariant, GRU/LSTM replacement, attention, self-attention, and cross-attention layers for transformer-based or general architectures.A single static query performs sequence pooling, while distinct query and key sets provide attention.Source: §A.6.3 · p. 86