PaperlayerView exact revision on arXiv

Source-linked AI summary

Hopfield Networks is All You Need

Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter

arXiv:2008.02217v3cs.NEcs.CLcs.LGstat.ML

TL;DR

Deep learning lacks flexible memory mechanisms for storing information, especially in continuous architectures. This paper introduces differentiable modern Hopfield layers with globally convergent updates, one-step retrieval, and broad benchmark gains across multiple domains.

  • Problem

    Deep learning needs alternatives to recurrent networks and improved memory mechanisms for storing and accessing information, including on small datasets.

  • Method

    The paper develops continuous modern Hopfield networks with a new differentiable energy function and convergent update rule, integrating them as deep-learning layers.

  • Results

    The update converges globally, while the network typically retrieves well-separated patterns in one step and retains exponential storage capacity.

  • Takeaways & Limitations

    Modern Hopfield layers provide memory, association, pooling, and attention mechanisms for deep-learning architectures across varied application domains.

Abstract

from arXiv · show

We introduce a modern Hopfield network with continuous states and a corresponding update rule. The new Hopfield network can store exponentially (with the dimension of the associative space) many patterns, retrieves the pattern with one update, and has exponentially small retrieval errors. It has three types of energy minima (fixed points of the update): (1) global fixed point averaging over all patterns, (2) metastable states averaging over a subset of patterns, and (3) fixed points which store a single pattern. The new update rule is equivalent to the attention mechanism used in transformers. This equivalence enables a characterization of the heads of transformer models. These heads perform in the first layers preferably global averaging and in higher layers partial averaging via metastable states. The new modern Hopfield network can be integrated into deep learning architectures as layers to allow the storage of and access to raw input data, intermediate results, or learned prototypes. These Hopfield layers enable new ways of deep learning, beyond fully-connected, convolutional, or recurrent networks, and provide pooling, memory, association, and attention mechanisms. We demonstrate the broad applicability of the Hopfield layers across various domains. Hopfield layers improved state-of-the-art on three out of four considered multiple instance learning problems as well as on immune repertoire classification with several hundreds of thousands of instances. On the UCI benchmark collections of small classification tasks, where deep learning methods typically struggle, Hopfield layers yielded a new state-of-the-art when compared to different machine learning methods. Finally, Hopfield layers achieved state-of-the-art on two drug design datasets. The implementation is available at: https://github.com/ml-jku/hopfield-layers

1 INTRODUCTION

The paper introduces differentiable modern Hopfield networks with a new energy function and update rule that typically retrieve patterns in one step. It proposes integrating these memory-equipped layers into neural networks and identifies the update rule with transformer attention.

  • Contribution of this work: The work introduces deep learning layers equipped with memory through modern Hopfield networks.These layers are intended to store information or learned prototypes at different neural-network layers.
  • Contribution of this work: A novel energy function and differentiable update rule typically retrieve patterns after one update.One-step retrieval is compatible with activating layers in deep networks and differentiability supports gradient-descent parameter updates.
  • Background: Modern Hopfield networks serve as associative memories that can retrieve the most similar pattern or an average over similar patterns.Earlier energy-function improvements are presented as improving the properties of Hopfield networks.
  • Contribution of this work: The continuous modern Hopfield network generalizes exponential interaction energies to continuous patterns and states.The proposed update rule ensures global convergence to stationary points, including local minima or saddle points.
  • Contribution of this work: The new update rule is equivalent to the transformer’s attention mechanism while retaining fast convergence and storage-capacity properties.The paper frames this equivalence as a generalization of binary modern Hopfield networks to continuous states.

2 MODERN HOPFIELD NETS WITH CONTINUOUS STATES

This section develops a continuous-state modern Hopfield network with a bounded energy and globally convergent update rule, while proving exponential storage capacity, one-step retrieval, and exponentially small retrieval errors. The update is also shown to be equivalent to transformer attention.

  • Continuous-state formulation: The proposed continuous-state energy adds a quadratic state term, keeping the state norm finite and the energy bounded while retaining exponential storage capacity and fast convergence.The network is designed for integration into deep-learning architectures.
  • Convergence: The update rule converges globally in energy to a stationary point, and all limit points are stationary points that are either local minima or saddle points.If the iteration does not converge, its limit points form a connected and compact set.
  • Storage capacity: Theorem 3 proves that randomly chosen fixed-norm patterns can be stored in a number exponential in the pattern dimension d, with probability 1 − p.The theorem assumes patterns on a sphere and a specified failure probability p.
  • Retrieval: Well-separated patterns are retrieved after one update, with the distance to the fixed point exponentially small in their separation ∆i.One-update retrieval is important for deep-learning layers that activate only once.
  • Retrieval: The retrieval error ∥f(ξ) − xi∥ decreases exponentially with the pattern separation ∆i.The result is stated under a query sufficiently close to the stored pattern and its fixed point.
  • Transformer attention: The Hopfield update rule is equivalent to the attention mechanism used in transformer and BERT models.The equivalence follows by mapping keys, queries, and values into the Hopfield space and setting β = 1/√dk.

Y WKWV . (10)

Equation (10) identifies transformer attention with the left-hand side, while its right-hand side motivates the specific Hopfield layers introduced next.

  • Y WKWV . (10): Equation (10)’s left-hand side is transformer attention; in self-attention, R = Y and WKWV is replaced by WV.The equation’s right-hand side explains the specific Hopfield layers introduced for additional deep-network functionality.

3 NEW HOPFIELD LAYERS FOR DEEP LEARNING

Modern Hopfield networks serve as differentiable deep-learning layers that add associative memory and typically retrieve patterns in one update. The section presents Hopfield layers for pooling, set processing, attention, sequence modeling, and reference-data retrieval, including specialized HopfieldPooling and HopfieldLayer variants.

  • Layer integration: Continuous, differentiable modern Hopfield networks can be integrated as specialized deep-learning layers equipped with memory.Their typical one-update pattern retrieval matches layers activated only once.
  • Layer integration: Hopfield network layers can substitute pooling, permutation equivariant, GRU, LSTM, and attention layers.This extends their use across pooling, set-based, recurrent, and attention-based architectures.
  • HopfieldPooling: HopfieldPooling pools or summarizes sets using queries to search stored patterns from the input or previous layers.Learned static query patterns produce an output set whose size equals the number of queries, and position encoding can support pooling and convolution-like operations.
  • HopfieldLayer: HopfieldLayer supports multiple queries of training, reference, prototype, or learned sets while preserving the number of input and output vectors.It can realize SVM, k-nearest-neighbor, and LVQ models.
  • Additional functionality: Additional functionality includes multiple updates, variable β for controlling fixed-point type and metastable-state size, and adjustable storage capacity.β controls how many patterns are averaged and can be learned alongside learning dynamics.

4 EXPERIMENTS · A APPENDIX · CONTENTS OF THE APPENDIX

The experiments demonstrate that Hopfield layers apply across diverse domains, improving results on multiple instance learning, small-data classification, and drug design tasks. The appendix provides theoretical analyses, transformer-attention connections, experiments, and a PyTorch implementation.

  • 4 EXPERIMENTS: Hopfield layers were applied across natural language processing, multiple instance learning, small classification, and drug design tasks.These experiments assess the layers across a wide range of applications.
  • 4 EXPERIMENTS: DeepRC achieved an AUC of 0.832 ± 0.022 on immune repertoire classification, outperforming competing methods in average AUC across datasets.Each immune repertoire contains approximately 300,000 instances, requiring extraction of a few indicative sequence patterns.
  • 4 EXPERIMENTS: HopfieldPooling was evaluated on image-annotation MIL datasets containing segmented color-image regions described by color, texture, and shape.The benchmark includes Elephant, Fox, and Tiger datasets with 100 positive and 100 negative example images.
  • 4 EXPERIMENTS: Hopfield networks outperformed all other methods on 75 small UCI datasets, establishing a new state-of-the-art.The comparison included neural networks, Random Forests, Support Vector Machines, boosting, bagging, and other machine-learning methods.
  • 4 EXPERIMENTS: The drug-design experiments tested HopfieldLayer on four datasets covering anti-viral discovery, protein inhibitors, metabolic effects, and related modeling tasks.The datasets include HIV, BACE, and BBBP applications.
  • A APPENDIX: The appendix develops continuous-state modern Hopfield networks, including their energy, update rule, convergence, fixed points, learning associations, forgetting, and spurious states.It also supplies mathematical background and reviews binary modern Hopfield networks.
  • CONTENTS OF THE APPENDIX: The appendix establishes that the Hopfield update rule is transformer attention and documents experiments on attention, MIL, UCI classification, and drug-design benchmarks.It concludes with PyTorch implementation sections covering introduction, functionality, and usage.

LIST OF THEOREMS … A.1.3 NEW UPDATE RULE

The paper’s appendix organizes the continuous-state modern Hopfield network around its energy, update rule, convergence, storage capacity, and retrieval results. The new update maps states into the pattern simplex through softmax-weighted associations and yields globally convergent dynamics with one-step pattern retrieval.

  • LIST OF THEOREMS: The theorem list covers global convergence of energy and stationary points, storage-capacity results for placed and random patterns, one-update retrieval, and exponentially small retrieval error.It also lists a theorem for binary modern Hopfield networks and definitions of softmax, log-sum-exp, convex conjugate, Legendre transform, epi-sum, and the Lambert function.
  • LIST OF DEFINITIONS: The definitions list supplies mathematical background on softmax, log-sum-exp, convex conjugacy, Legendre transforms, epi-sums, and the Lambert function.These definitions are listed as prerequisites for the paper’s proofs.
  • A.1 CONTINUOUS STATE MODERN HOPFIELD NETWORKS (A NEW CONCEPT): The appendix introduces a continuous-state modern Hopfield network, then develops its energy, update rule, global convergence, local convergence, learning, transformer associations, and temporal forgetting.The surrounding sections also review binary modern Hopfield networks and provide mathematical background for the proofs.
  • A.1.1 INTRODUCTION: Patterns on a sphere can be stored and retrieved exponentially in the Hopfield-space dimension, with typical one-update retrieval and exponentially small retrieval error.These are identified as the main results of the introduction’s storage-capacity and retrieval analyses.
  • A.1.2 NEW ENERGY FUNCTION: The new energy generalizes Demircigil et al.’s exponential energy from binary to continuous states while preserving log-sum-exp properties for high storage capacity and fast convergence.The paper contrasts this with earlier binary networks, including exponential storage capacity N = 2^d/2 for binary patterns.
  • A.1.3 NEW UPDATE RULE: The new update uses softmax(βX^Tξ) instead of the classical Hopfield vector X^Tξ and always places the new state in the simplex defined by the patterns.The update is introduced as an energy-minimization rule.
  • A.1.3 NEW UPDATE RULE: The update rule ensures global convergence: every limit point generated by the rule is a stationary point, specifically a local minimum or saddle point, of the energy.The subsequent local-convergence analysis reports pattern retrieval with one update.

A.1.4 GLOBAL CONVERGENCE OF THE UPDATE RULE

The update rule is a globally convergent CCCP procedure that monotonically decreases the energy and converges in energy to a fixed point. Its limit points are stationary points of the energy, although convergence need not be to a local minimum.

  • Global convergence: The update rule in Eq. (31) is the Concave-Convex Procedure for minimizing the energy E and is proven to converge globally.Global convergence means the result holds from every initial point.
  • Energy descent: The iteration monotonically decreases the energy E over time.The decrease is strict whenever the current iterate is not a stationary point.
  • Limitations: Although the objective converges in all cases, the iteration does not necessarily converge to a local minimum.Local maxima can only be reached if an iteration lands exactly on one; strict energy decrease prevents convergence to them otherwise.
  • Optimization subproblem: The majorized optimization subproblem is strictly convex with a unique global minimum.Its Hessian is strictly positive definite everywhere, and the auxiliary objective can be written as a quadratic form.
  • Limit points: All limit points generated by the update rule are stationary points of E, including local minima or saddle points.The convergence argument uses Zangwill’s global convergence theory and compactness of the iterates’ domain.

A.1.5 LOCAL CONVERGENCE OF THE UPDATE RULE: FIXED POINT ITERATION

The fixed-point iteration locally converges according to pattern separation: it approaches the global mean, a metastable average, or an individual stored pattern. Under a contraction condition, fixed points are unique and convergence can be rapid, with large separation yielding near-fixed-point behavior after one update.

  • Three convergence regimes: Poorly separated patterns lead to a fixed point near their arithmetic mean, with softmax weights close to p_i = 1/N.This is the global-fixed-point regime.
  • Three convergence regimes: Well-separated patterns make each iteration converge to the pattern most similar to the initial query, with p approaching the corresponding one-hot vector e_i.The limiting fixed point lies close to the selected stored pattern.
  • Three convergence regimes: Clusters of mutually similar but otherwise separated patterns produce metastable states, and nearby initializations converge to the corresponding cluster average.These metastable states lie close to the mean of the similar patterns.
  • Contraction and convergence: If the Jacobian norm satisfies the stated bound below 1, the iteration is a contraction with a unique fixed point in each compact set.Banach’s fixed point theorem then gives convergence to that unique fixed point.
  • Contraction and convergence: For large pattern separation ∆_i, the iteration is close to its fixed point even after one update.The passage states that this behavior was confirmed in several experiments.

A.1.6 PROPERTIES OF FIXED POINTS NEAR STORED PATTERN … A.2 PROPERTIES OF SOFTMAX, LOG-SUM-EXPONENTIAL, LEGENDRE TRANSFORM, LAMBERT W FUNCTION

The appendix establishes exponential storage capacity and exponentially accurate one-update retrieval for well-separated patterns, then formulates association learning through raw or mapped query and stored-pattern spaces. It also specifies gradient-based treatment of these association mappings within deep neural networks.

  • A.1.6 PROPERTIES OF FIXED POINTS NEAR STORED PATTERN: Well-separated patterns trade lower retrieval error and faster convergence for reduced storage capacity, whereas small separation increases capacity but worsens both.Large separation permits one-update retrieval with exponentially low error.
  • A.1.6 PROPERTIES OF FIXED POINTS NEAR STORED PATTERN: Theorem A5 gives random-pattern storage capacity exponential in dimension, with base c constrained by β, sphere radius M, failure probability p, and dimension d.The result holds with probability 1 −p and uses a Lambert W function constraint on c.
  • A.1.6 PROPERTIES OF FIXED POINTS NEAR STORED PATTERN: For β = 1, K = 3, d = 20, and p = 0.001, the random-pattern bound proves c ≥3.1546.A second example proves c ≥1.3718 for β = 1, K = 1, d = 75, and p = 0.001.
  • A.1.6 PROPERTIES OF FIXED POINTS NEAR STORED PATTERN: Theorem A7 bounds storage for the expected minimal separation of random patterns, with β = 1, K = 3, c = 2, and d ≥17 satisfying its condition.This analyzes expected separation rather than the probability that all patterns are stored.
  • A.1.6 PROPERTIES OF FIXED POINTS NEAR STORED PATTERN: Theorem A8 shows that, for large separation ∆i, one update places the query exponentially close to the fixed point associated with pattern xi.The Jacobian norm decreases exponentially with separation, and each iteration moves the point closer to the fixed point.
  • A.1.6 PROPERTIES OF FIXED POINTS NEAR STORED PATTERN: Theorem A9 bounds retrieval error under query and fixed-point proximity conditions, while the appendix states that retrieval error decreases exponentially with separation ∆i.For random vectors, the Lipschitz constant decreases exponentially with dimension, yielding exponentially small one-update error.
  • A.1.7 LEARNING ASSOCIATIONS: Association learning covers raw patterns without mapping, one-set mapping through W, and two-set mapping through WQ and WK into an associative space.These cases lead mathematically to the same update-rule form, including softmax-weighted stored patterns.
  • A.1.7 LEARNING ASSOCIATIONS: The association update can be simplified to ξnew = Y softmax(β Y^T W^T ξ), and its derivatives support chain-rule integration into deep neural networks.For two mappings, derivatives are derived with respect to both WQ and WK, using the Jacobian of the update rule.

5. Convex Conjugate of epi-sums · A.3 MODERN HOPFIELD NETWORKS: BINARY STATES (KROTOV AND HOPFIELD) · A.3.1 MODERN HOPFIELD NETWORKS: INTRODUCTION

The appendix characterizes key convex-conjugacy results underlying modern Hopfield networks, including the duality between log-sum-exp and negative entropy. It also reviews memory mechanisms and the capacity advances from classical binary Hopfield networks to exponential-storage models.

  • 5. Convex Conjugate of epi-sums: The Legendre transform of log-sum-exp is negative entropy restricted to the probability simplex, and the converse transform returns log-sum-exp.These identities are stated in Lemma A28 and justified using Fenchel–Moreau duality for continuous convex functions.
  • 5. Convex Conjugate of epi-sums: For non-singular XXT, the relevant conjugate is finite only on the transformed probability simplex; outside that domain, the defining supremum is unbounded.The derivation uses the Legendre-transform definition and the solvability conditions for the log-sum-exp conjugate.
  • 5. Convex Conjugate of epi-sums: The appendix establishes convexity and continuity of lse(β, XT ξ) from the positive semidefiniteness of diag(p) − ppT.This supports applying biconjugation results to the log-sum-exp expression.
  • 5. Convex Conjugate of epi-sums: The appendix introduces the Lambert W function, including its upper branch W0, lower branch W−1, identities, and special values for later storage-capacity bounds.The function is defined as the inverse of f(y) = yey.
  • A.3.1 MODERN HOPFIELD NETWORKS: INTRODUCTION: Modern Hopfield networks provide neural networks with additional associative memory, complementing recurrent connections and attention-based external-memory systems.Related approaches include neural Turing machines, memory networks, differentiable softmax retrieval, and associative memories for RNN hidden states.
  • A.3 MODERN HOPFIELD NETWORKS: BINARY STATES (KROTOV AND HOPFIELD): In d dimensions, classical binary Hopfield networks store d uncorrelated patterns without errors but only Cd/ ln(d) random patterns under fixed-stability constraints.The coefficient satisfies C < 1/2 for a fixed stable pattern and C < 1/4 when all patterns are stable.
  • A.3 MODERN HOPFIELD NETWORKS: BINARY STATES (KROTOV AND HOPFIELD): Higher-order interaction functions increase binary Hopfield storage capacity, while exponential interactions yield 2d/2 capacity with one-update convergence but vanishing attraction radius.For polynomial interactions, the cited capacity is αn dn−1, with the stated condition cn > 2(2n −3)!!.

A.3.2 ENERGY AND UPDATE RULE FOR BINARY MODERN HOPFIELD NETWORKS · A.4 HOPFIELD UPDATE RULE IS ATTENTION OF THE TRANSFORMER · Y WKWV . (548)

The binary modern Hopfield formulation uses exponential interactions to store exponentially many patterns and recover them with one update at high probability. Its continuous-state update rule is equivalent to transformer attention, while the corresponding expression also motivates additional Hopfield layers.

  • A.3.2 ENERGY AND UPDATE RULE FOR BINARY MODERN HOPFIELD NETWORKS: Binary modern Hopfield networks store patterns with components xij ∈ {−1, +1} and define energy through an interaction function F applied to pattern-state dot products.The stored inputs are x1, …, xN, while ξ denotes the units’ current state.
  • A.3.2 ENERGY AND UPDATE RULE FOR BINARY MODERN HOPFIELD NETWORKS: For F(a) = an, the model stores αnd^(n−1) patterns, with n = 2 recovering the classical Hopfield energy.The power-law interaction generalizes the classical network’s quadratic case.
  • A.3.2 ENERGY AND UPDATE RULE FOR BINARY MODERN HOPFIELD NETWORKS: F(a) = exp(a) yields exponential storage capacity N = 2^(d/2), and one update recovers the final pattern with high probability.This result is stated for the binary model analyzed by Demircigil et al.
  • A.3.2 ENERGY AND UPDATE RULE FOR BINARY MODERN HOPFIELD NETWORKS: N = exp(αd) + 1 patterns can be stored, while Pr(∃i ∃j : Tj(exi) ≠ xij) → 0 means one update per component recovers patterns with high probability.The constraint α < I(1−2ϱ)^2 governs the trade-off between attraction radius ϱd and storage count.
  • A.3.2 ENERGY AND UPDATE RULE FOR BINARY MODERN HOPFIELD NETWORKS: As d → ∞, Pr(∃i ∃j : Tj(xi) ≠ xij) → 0, so all stored patterns are fixed points with probability converging to 1.The passage further states that α can approach I(1).
  • A.4 HOPFIELD UPDATE RULE IS ATTENTION OF THE TRANSFORMER: The Hopfield update rule is the attention mechanism used in transformer and BERT models when stored patterns act as keys and state patterns act as queries.The construction maps keys and queries into a Hopfield space of dimension dk.
  • A.4 HOPFIELD UPDATE RULE IS ATTENTION OF THE TRANSFORMER: With β = 1/√dk and projections WK, WQ, and WV, the Hopfield update becomes the transformer attention expression multiplied by WV.The matrices K, Q, and V are formed from the stored patterns and state patterns using these projections.
  • Y WKWV . (548): The left part of Eq. (548) is transformer attention, whereas its right part provides the starting point for specific Hopfield layers and other deep-network functionalities.The figure describes continuous-state generalization while retaining fast convergence and storage-capacity properties.

A.5 EXPERIMENTS · A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS

The experiments analyze transformer attention heads through Hopfield operating classes and test simplified Gaussian averaging heads. They track mode changes during BERT-small training and show that Gaussian heads can substantially reduce attention parameters.

  • A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS: Pre-trained BERT heads often exhibit large and small metastable states in middle layers.The analysis classifies heads by the minimal number k of patterns whose softmax values sum to 0.90.
  • A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS: BERT-small experiments use 12 layers, 4 heads, sequence length 128, batch size 128, and hidden dimension 256.The setup reduces the computational demands of transformer training while investigating learning dynamics.
  • A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS: The pre-training experiments use English Wikipedia and an uncased snapshot of BookCorpus because the original BookCorpus dataset is no longer publicly available.The implementation uses the Hugging Face transformers repository and aims to reproduce the Devlin et al. dataset closely.
  • A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS: Around 9,000 to 10,000 training steps, middle-layer attention heads often switch to Class IV as the second major loss drop occurs.This behavior is observed during the first 20,000 training steps.
  • A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS: Lower-layer heads often average over many patterns, motivating replacement of input-dependent self-attention with Gaussian averaging layers.The proposed simplification targets attention where fine-grained input dependence appears unnecessary according to the operating-mode analysis.
  • A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS: 2·N parameters per head are required for Gaussian averaging, compared with 2·dk·dy parameters for standard self-attention projections.The Gaussian head uses location and scale parameters for an input of N tokens.
  • A.5.1 EXPERIMENT 1: ATTENTION IN TRANSFORMERS DESCRIBED BY HOPFIELD DYNAMICS: 95.5 times more parameters are trained by the BERT-base attention mechanism than by a Gaussian head for dy = 768, dk = 64, and N = 512.This comparison uses (2·768·64)/(2·512) = 95.5.

A.5.2 EXPERIMENT 2: MULTIPLE INSTANCE LEARNING DATASETS.

Modern Hopfield networks are integrated into DeepRC for massive immune-repertoire multiple instance learning, where exponential storage capacity supports extracting rare patterns from large bags of receptor sequences. The resulting architectures outperform competing methods in average AUC across all four immune-repertoire dataset categories, while HopfieldPooling compresses variable-sized bags using a trainable query pattern.

  • Immune Repertoire Classification: DeepRC applies a modern Hopfield network to immune repertoires represented as bags containing potentially many receptor-sequence instances.The architecture is designed for immune repertoire classification, a massive multiple instance learning task.
  • Immune Repertoire Classification: Exponential storage capacity enables modern Hopfield networks to extract a few informative patterns from massive sets of immune-receptor sequences.Immune repertoire classification typically requires identifying rare sequence patterns associated with immune status.
  • Immune Repertoire Classification: Deep learning architectures with modern Hopfield networks outperform all competing methods in average area under the ROC curve across all four immune-repertoire dataset categories.The categories include simulated data, LSTM-generated data, and two real-world categories; the experiments address massive instance counts and low-frequency signals.
  • Immune Repertoire Classification: DeepRC achieves higher average AUC than all competing methods, with the SVM using a MinMax kernel or the burden test generally serving as runner-up methods across categories.Table A.1 reports average performance on simulated immunosequencing data and the remaining three dataset categories.
  • Multiple Instance Learning Benchmark Datasets: HopfieldPooling averages class-indicative embedded instances using a trainable fixed query pattern, compressing variable-sized bags into fixed-sized representations for classification.The layer receives a set of embedded instances Y and a trainable state (query) pattern Q.

A.5.3 EXPERIMENT 3: CLASSIFICATION ON SMALL UCI BENCHMARK DATASETS … 17. Stacking (Wolpert)

The experiment evaluates Hopfield layers on 75 small UCI classification datasets, where neural networks often generalize poorly relative to established machine-learning methods. Models were selected by dataset-specific grid search, and methods were compared using average test-accuracy rank.

  • A.5.3 EXPERIMENT 3: CLASSIFICATION ON SMALL UCI BENCHMARK DATASETS: Hopfield layers were motivated by their potential to identify indicative samples or learn class-relevant prototypes for query samples.
  • A.5.3 EXPERIMENT 3: CLASSIFICATION ON SMALL UCI BENCHMARK DATASETS: Modern Hopfield networks implemented through the Hopfield layer were compared with 17 groups of methods.
  • 11. Other Ensembles: The comparison included neural-network variants such as standard NN, BatchNorm, WeighNorm, MSRAinit, LayerNorm, ResNet, and Self-Normalizing Nets.
  • A.5.3 EXPERIMENT 3: CLASSIFICATION ON SMALL UCI BENCHMARK DATASETS: 75 UCI datasets with fewer than 1,000 samples were evaluated because small datasets are difficult for neural networks to generalize on.Such datasets are consistently outperformed by gradient boosting, random forests, and support vector machines.
  • 17. Stacking (Wolpert): A grid search selected the best hyperparameter setting and model separately for each dataset.The search space is listed in Table A.3.
  • 17. Stacking (Wolpert): The tested architecture used 0, 1, or 7 fully connected embedding layers with SELU activations and 32, 128, or 1024 hidden units.Embedding layers were followed by Hopfield, then a mapping to the output vector.
  • 17. Stacking (Wolpert): 25 methods were ranked by test-set accuracy per dataset, then averaged across all 75 datasets to obtain each method’s rank.Baseline scores came from Klambauer et al. (2017a).

A.5.4 EXPERIMENT 4: DRUG DESIGN BENCHMARK DATASETS

Hopfield layers were evaluated on four challenging MoleculeNet drug-design classification datasets against conventional, neural, and graph-based methods. They achieved state-of-the-art performance on SIDER and BACE, with ROCAUC values of 0.672 ± 0.019 and 0.902 ± 0.023, respectively.

  • Experimental design and implementation details: Four MoleculeNet classification datasets—HIV, BACE, BBBP, and SIDER—were used to test Hopfield layers on challenging drug-design tasks.The datasets cover antiviral screening, β-secretase binding affinity, blood-brain barrier permeability, and side-effect prediction.
  • Experimental design and implementation details: The Hopfield layer used training inputs as stored patterns or keys, training labels as values, and inputs as query states.The implementation maps the input into the softmax and the target outside it through the key and value projections.
  • Results: The model was compared with SVMs, XGBoost, random forests, DNNs, and graph neural networks including GCNs, GATs, MPNNs, and Attentive FP.Results for other methods were taken from Jiang et al. (2020).
  • Results: 0.672 ± 0.019 ROCAUC was achieved for predicting side effects on SIDER, reaching state-of-the-art performance.The reported metric is predictive performance on the test set.
  • Results: 0.902 ± 0.023 ROCAUC was achieved for predicting β-secretase activity on BACE, reaching state-of-the-art performance.The results were reported across the drug-design benchmark evaluation.

A.6.1 INTRODUCTION

This section presents Hopfield layers as PyTorch implementations and usage-ready components for memory, pooling, association, credit assignment, and attention. They can serve as plug-in replacements for attention and several other neural-network layers, using modern Hopfield networks with continuous states and increased storage capacity.

  • Applications: Potential applications include multiple instance learning, point-set processing, permutation-invariant learning, attention-based learning, language and sequence processing, and reference-data retrieval.Reference data may include training data retrieved by the model or experiences stored for reinforcement learning.
  • Layer capabilities: Hopfield layers implement associative memory, conditional pooling, association-based data combination, associative credit assignment, and attention mechanisms.These capabilities support storing and retrieving data, conditional aggregation, association-based combination, reinforcement-learning-related value estimation, and attention.
  • Layer integration: A Hopfield layer can substitute attention layers in transformer and BERT architectures as a plug-in replacement for existing layers.The described replacement targets attention layers and extends to pooling, permutation-equivariant, GRU, and LSTM layers.
  • Modern Hopfield basis: Modern Hopfield layers use continuous states and increased storage capacity, while their single-head dynamics follow energy minimization.The energy-minimization dynamics are presented as providing advantages over memory cells, associative memory, and attention mechanisms.

A.6.2 FUNCTIONALITY · A.6.3 USAGE · R Y Y

Hopfield layers extend associative-memory functionality through configurable updates, fixed-point dynamics, capacity, pattern handling, and cross-set association. They also support pooling, permutation-equivariant processing, and Transformer-compatible encoder and decoder replacements across applications.

  • A.6.2 FUNCTIONALITY: The layer adds multiple updates, variable β, adjustable associative-space dimension, static patterns, and pattern normalization to control retrieval and fixed-point behavior.These options respectively affect update precision, operating mode, storage-capacity versus averaging tradeoffs, fixed-pattern search, and norm-dependent dynamics.
  • A.6.2 FUNCTIONALITY: Hopfield layers support cross-set association for self-attention, encoder-decoder attention, sequence analysis, multiple instance learning, point sets, memory, pooling, and data-source combination.State and stored patterns are separately projected into an associative space before retrieval or averaging.
  • A.6.2 FUNCTIONALITY: The associative-space dimension trades off exponentially growing storage capacity against smaller metastable states and less averaging.Higher dimension increases capacity but reduces the number of patterns averaged in metastable states.
  • A.6.2 FUNCTIONALITY: Static state and stored patterns remain input-independent, enabling searches for particular fixed patterns among stored data and vice versa.Static patterns are determined by bias weights and remain constant across network inputs.
  • A.6.2 FUNCTIONALITY: Pattern normalization can be applied to inputs, projected associative-space patterns, or neither, controlling fixed-point dynamics through pattern norms and centers.The relevant dynamics depend on β, maximal pattern norm M, similarity spread mmax, and similarity-center norm ∥mx∥.
  • A.6.3 USAGE: With functionalities disabled, HopfieldEncoderLayer and HopfieldDecoderLayer serve as one-to-one plug-in replacements for PyTorch Transformer encoder and decoder layers.The implementation targets broad use in memory networks and Transformer models.
  • A.6.3 USAGE: A single static query pattern makes the Hopfield layer a sequence-pooling layer, while associative-memory operation supports permutation-equivariant layer replacement.The pooling weights arise from softmax applied over stored patterns, and permutation equivariance follows from the absence of dependency between sequence elements.
  • R Y Y: Figure A.7 depicts optional normalization, projection, multiple updates, and output formation from raw query and stored-key patterns, including two possible stored-input tensors.The default uses input layer normalization but no layer normalization after projection.
Loading 2008.02217v3…