Outlier-Efficient Hopfield Layers for Large Transformer-Based Models

Contribution

• OutEffHop introduces an associative memory model with outlier-efficient retrievals and interprets Softmax1 as an approximation of its retrieval dynamics.The model adds a no-op classification dimension and yields outlier-efficient Hopfield layers as alternatives to vanilla attention.Source: §1 · pp. 1–2

Theory

• The proposed Hopfield model retains fixed point convergence and exponential memory capacity while providing an outlier-efficient retrieval mechanism.These properties are stated as theoretical contributions alongside the Softmax1 interpretation.Source: §1 · p. 2

Contribution

• OutEffHop layers reduce probability assigned to low-information vectors and are positioned as alternatives to Transformer attention.The paper also establishes a generalization bound scaling with N^-1/2 log N in sample size and log(dM) in pattern dimension and memory-set size.Source: §1 · p. 2

Empirical validation

• 22+% lower average kurtosis and 26+% lower maximum infinity norm are reported across BERT, OPT, ViT, and STanHop-Net.The method ranks among the top two for outlier efficiency in 25 of 30 settings.Source: §1 · p. 2

2.1. Background

• Attention’s no-update behavior assigns high probability to small-value or low-information tokens, forcing a wide QK^T range and creating “no-op” outliers.Because softmax never reaches exact zero, these tokens continue receiving gradient signals that can magnify outliers during training.Source: §2.1 · pp. 2–3

Background

• Modern Hopfield models connect associative-memory retrieval to attention by computing memory expectations weighted by query–memory similarity.They are also characterized here by exponential memory capacity, one-step retrieval, and compatibility with deep-learning architectures.Source: §2.2 · p. 3, §2.1 · p. 3

2.2. One Dimension More

• The model adds a classification dimension that maps no-op outliers to a shared memory class whose inner product with the query is zero.Non-outlier memories receive zero in the added coordinate, while outliers receive a nonzero value; the construction assumes outliers are known or identified by assignment or thresholding.Source: §2.2 · p. 3, §2.3 · p. 3

2.3. Hopfield Energy and Retrieval Dynamics

• A refined log-sum-exponential energy assigns the no-op class a zero-energy point, so classified outliers no longer contribute to retrieval outputs.This creates an associative-memory mechanism in which the no-op pattern remains unaffected by state updates.Source: §2.3 · p. 4
• The retrieval dynamics monotonically minimize the energy and approach the nearest local minimum, while one iteration is equivalent to outlier-efficient Softmax1 attention.The construction therefore supports both iterative associative-memory retrieval and a single-step attention interpretation.Source: §2.3 · p. 4

3.1. Convergence Guarantee

• Iterative application of TOutEff produces sequences whose limit points are stationary points of the proposed energy function.The convergence analysis also establishes convergence through generalized fixed points.Source: §3.1 · p. 5

3.2. Retrieval Error Analysis

• OutEffHop retrieval dynamics have lower upper-bounded retrieval error than the original modern Hopfield model, under the stated comparison condition.The tighter bound holds when (γ + 1)/2 ≥ cos(α).Source: §3.2 · p. 5
• The model stores and retrieves patterns with capacity scaling exponentially in pattern dimension d, with a larger lower bound than the original modern Hopfield model.The informal theorem assumes memory patterns are randomly sampled from a sphere and β > 0.Source: §3.2 · p. 5

3.3. Generalization Bound

• The analysis models OutEffHop as a function of query and memory-pattern inputs under norm-bounded assumptions.The function maps memory patterns and query sequences through learned matrices into the layer output.Source: §3.3 · pp. 5–6
• The generalization error remains controllable when sample size N scales logarithmically with pattern dimension d and memory-set size M.The stated result says sequence length T does not affect generalization.Source: §3.3 · p. 6

4.1. Outlier Efficiency of OutEffHop

• OutEffHop significantly reduces maximum infinity norm versus vanilla attention across BERT, OPT, ViT, and STanHop-Net during pretraining.The comparison also reports improvements over Clipped Softmax and Gated Attention in the examined settings.Source: Figure 2 · p. 7, §4.1 · p. 8

4. Experimental Studies

• The experiments replace vanilla attention or Hopfield layers, train models from scratch, and evaluate outlier metrics including maximum infinity norm and average kurtosis.The evaluated architectures are BERT, OPT, ViT, and STanHop-Net.Source: §4.1 · p. 8

4.1. Outlier Efficiency of OutEffHop

• ∼22+% lower average kurtosis and ∼26+% lower maximum infinity norm are achieved across four test models when combining OutEffHop with comparison methods.The combination further improves the effect, with the OPT Clipped OutEffHop setting as the reported exception.Source: §4.1 · p. 8

4.2. OutEffHop Improves Hopfield-Centric Deep Learning Model: A Case Study on STanHop-Net

• OutEffHop ranks first or second for outlier efficiency in 25 of 30 STanHop-Net settings while causing only marginal model-performance sacrifice.The STanHop-Net comparison evaluates multiple Hopfield-layer variants across multivariate time-series prediction settings.Source: §4.2 · p. 9, Table 2 · p. 9

4.1. Outlier Efficiency of OutEffHop

• In BERT, OutEffHop mitigates outliers across selected layers, with the effect particularly pronounced in the final several layers.The case studies examine FFN outputs and tensor components within layer 10.Source: Figure 3 · p. 7, Figure 4 · p. 7, §4.3 · p. 9

5. Conclusion and Discussion

• ∼22+% lower average kurtosis and ∼26+% lower maximum infinity norm are reported across four models, with top-two outlier efficiency in 25 of 30 settings.The method also improves the corresponding metrics by averages of 3% and 4% against three STanHop-Net variants.Source: §5 · p. 9
• OutEffHop does not address outliers induced by LayerNorm, whose mechanisms differ from those associated with attention.The authors identify integration of these outlier types as future work.Source: §5 · p. 9

• The approach could intensify training-data biases, potentially producing unfair or discriminatory outcomes for underrepresented groups.Source: Impact Statement · p. 9
• The paper includes supplementary material covering related works, background, proofs, and additional numerical experiments.Source: Supplementary Material · p. 10

• Associative-memory research has advanced memory storage capacity and developed architectural designs for deep learning.Source: A. Related Works · p. 10
• Quantization reduces computational burden through low-bit precision, but attention outliers can cause exploding attention weights and degrade transformer quantization.Source: A. Related Works · p. 10
• Existing methods modify normalization, clipping, or feature precision, while Clipped Softmax and Gated Attention force exact-zero attention outputs.Source: A. Related Works · p. 10
• Recent theory studies bounded attention weights and bounded query and key patterns for efficient transformers and modern Hopfield models.Source: A. Related Works · p. 10

• Softmax forces each head to attend somewhere, so no-update cases assign high probability to a spurious position and introduce noise.Source: §B.1 · p. 11
• Softmax1 adds 1 to Softmax’s denominator, allowing less-relevant heads to contribute minimally instead of assigning probability mass to a token.Source: §B.1 · p. 11
• OutEffHop learns similarity scores so low-overlap patterns receive near-zero probability as zero-energy no-op memories, and its CCCP retrieval dynamics monotonically decrease energy.Source: §B.1 · p. 11, §C.1 · p. 13
• For n = [-10, -10, -10], Softmax produces approximately [0.33, 0.33, 0.33], illustrating nonzero attention for negligible inputs.Source: §B.1 · p. 11
• Softmax can concentrate weight on a low-information token while driving other token weights toward zero, requiring a wide input range and creating outliers.Source: §B.1 · p. 11
• Softmax1 instead diminishes all vector values close to zero, reducing memory usage and making the attention head outlier-robust.Source: §B.1 · p. 12

C.3. Theorem 3.2

• OutEffHop has a tighter retrieval-error upper bound than the original modern Hopfield model.Source: §C.3 · pp. 13–14
• The comparison uses retrieval-error expressions relative to a stored memory ξµ and introduces γ = exp{−βδ} with 0 < γ < 1 for β > 0 and suitable δ.Source: §C.3 · pp. 13–14, §C.4 · p. 14
• The stated condition is analyzed for normalized memory and query vectors, with cos(α) representing overlap between vectors.Source: §C.4 · p. 15
• The proposed model experiences smaller retrieval errors during the initial retrieval phase, when memories and queries have smaller overlap.Source: §C.4 · p. 15

Theorem 3.3

• The model’s memory capacity lower bound is at least M ≥ √pC for the stated successful storage and retrieval probability.This bound applies to memory patterns sampled from a sphere of radius m.Source: §C.5 · p. 15
• For all β, the proposed model has a larger memory-capacity lower bound than the original modern Hopfield model.The comparison follows from the stricter bound used for the proposed model.Source: §C.5 · pp. 15–16
• The proof establishes a well-separation condition sufficient for the OutEffHop retrieval map to preserve the relevant state space.The condition is derived for any β > 0 by selecting δ > 0 such that γ = exp{−βδ}.Source: §C.5 · p. 15
• The capacity comparison is expressed using a Lambert W solution for the bound’s auxiliary constant C.The proof defines C through the principal branch W0 and compares it with the original counterpart.Source: §C.5 · pp. 15–16

Memory Capacity

• OutEffHop outperforms Dense, Sparse, and 10th-order Hopfield baselines on MNIST and CIFAR10 memory-capacity tests, especially for large memory sets.The comparison uses β = 1 for all Hopfield models.Source: §D.2 · p. 22, Figure 5 · p. 22

Noise-Robustness

• OutEffHop excels when the signal-to-noise ratio is low, with minimal impact on error rate as injected noise increases.Noise robustness is evaluated by adding Gaussian noise with varying variance to image queries.Source: §D.2 · p. 22, Figure 6 · p. 23

Faster Convergence

• OutEffHop converges faster than Dense and Sparse Hopfield baselines across the tested CIFAR10 and CIFAR100 settings.The evaluation replaces the Vision Transformer attention layer with different Hopfield layers and compares loss and accuracy.Source: §D.2 · p. 22, Figure 7 · p. 23

Computational Resources

• The experiments compare pre-training time and average RAM usage for vanilla Softmax and OutEffHop across four model families.The setup uses two 80G A100 GPUs, a 24-core CPU, and a Slurm system.Source: §D.3 · p. 24, Table 4 · p. 24, Figure 8 · p. 25