Attention over noisy examples: generalization and memorization in the exponential Hopfield network

Francesco Alemanno

Source-linked AI summary

Attention over noisy examples: generalization and memorization in the exponential Hopfield network

Francesco Alemanno

arXiv:2610.22191v1Oct 15, 2026cond-mat.dis-nncs.LGcs.NE

TL;DR

The paper asks whether an attention layer can retrieve unseen archetypes from unlabelled noisy examples and how load and overfitting shape retrieval. A closed-form signal-to-noise analysis treats query logits as a planted random energy model, finding a load-independent generalization regime and freezing-driven memorization; on fashion-MNIST, sub-clusters make beyond-threshold readout useful.

Problem
The paper asks whether attention over unlabelled noisy examples can retrieve unseen archetypes, how load affects learning, and what overfitting looks like in an attention layer.
Source: §1 · pp. 1–2
Method
A closed-form signal-to-noise analysis derives one-step archetype overlap using an attention entropy based on how softmax weights spread over examples.
Source: §1 · p. 2
Results
The analysis finds that intermediate-temperature retrieval is load-independent, while high-temperature softmax condensation produces memorization as a random-energy freezing transition.
Source: §1 · p. 2
Takeaways & Limitations
On fashion-MNIST, sub-clusters make the class prototype a poor target, and the best accuracy lies beyond the memorization threshold where attention reads out a local sub-prototype.
Source: §7 · p. 13
The Gaussian-field approximation fails when softmax weights condense on a few examples, with numerical points up to 0.1 below its curve.
Source: §3 · p. 5, §4 · p. 9

Abstract

from arXiv · show

In Hebbian learning from examples a Hopfield network of $N$ neurons is not supplied with the $K$ archetypes it has to retrieve but with $M$ blurred examples of each, of quality $r$, and whether it can infer them depends on the load $\alpha=K/N$ (interference among them) and on the dataset entropy $\rho=(1-r^2)/(Mr^2)$ (how little the examples say about them). We transfer this protocol to the exponential Hopfield model, whose zero-temperature update is a step of softmax attention with the unlabelled examples as keys. A signal-to-noise analysis gives in closed form the overlap between the output and an unseen archetype, $m^{(1)}={\rm erf}(1/\sqrt{2\rho_{\rm eff}})$: the majority-rule law of the pairwise model at small load, with $\rho$ replaced by an attention entropy $\rho_{\rm eff}$ built from the weight the softmax gives to the correct class and the number of examples it spreads it over. Two consequences follow. First, in a window of inverse temperatures $\beta$ that widens as $\sqrt N$ the interference between archetypes is exponentially small in $N\beta$: the load drops out and the number of archetypes that can be learnt grows as $\exp(c\sqrt N)$. Second, learning fails differently: in the pairwise model through a spin-glass phase, in the exponential model through memorization: the softmax concentrates its weight on the training example closest to the input and returns it in place of the archetype. Seen from the input the softmax weights are those of the random energy model, and memorization is its freezing transition, at $\beta\sqrt{N(1-r^2)}\simeq\sqrt{2\ln M}$; it is measurable on a head whose keys are examples as the number of examples carrying weight. On MNIST the diagnostic follows the random-energy prediction; on fashion-MNIST it detects sub-clusters in the classes, and the best accuracy lies beyond the memorization threshold, where the softmax reads out a local sub-prototype.

1 Introduction

The introduction asks whether attention over unlabelled noisy examples can recover unseen archetypes, and how load and inverse temperature shape generalization and overfitting. Its analysis predicts an attention-entropy version of the majority-rule overlap, with load-independent retrieval at intermediate temperatures and memorization at high temperatures.

1 Introduction

Dense associative memories connect steep interactions to exponential storage, while zero-temperature exponential Hopfield updates correspond to softmax attention over stored patterns.Prior work also frames attention as nearest-neighbour lookup or kernel smoothing; example-based Hebbian learning instead stores noisy examples, with or without labels.Source: §1 · p. 1
Earlier example-learning work found retrieval when dataset entropy is sufficiently small and load remains below an entropy-dependent critical value; excessive noise or crosstalk leads to a spin-glass phase.In that learning setting, the spin-glass phase is described as the thermodynamic signature of overfitting.Source: §1 · p. 1
The study asks whether attention over noisy examples retrieves unseen archetypes, how load affects learning, and what replaces the pairwise model’s spin-glass overfitting.The motivating contrast is whether overfitting in an attention layer has a counterpart to the pairwise network’s spin-glass phase.Source: §1 · pp. 1–2
The one-step overlap with an unseen archetype follows m^(1) = erf(1/sqrt(2rho_eff)), replacing dataset entropy with an attention entropy determined by the softmax weights.At intermediate beta, rho_eff approaches rho for any load; at small beta, pairwise-model crosstalk is recovered.Source: §1 · p. 2
At large β, the softmax assigns essentially all weight to a single training example and the layer memorizes; from the query, this condensation is the freezing transition of a random energy model.Source: §1 · p. 2
The paper places its model at the p-to-infinity end of dense Hebbian networks, where crosstalk becomes logarithmic in K and permits closed-form analysis.The introduction says the analytical results are checked by Monte Carlo simulations and tested on MNIST and fashion-MNIST.Source: §1 · p. 2

2 Dataset and models

The section defines noisy-example datasets, compares pairwise and exponential Hebbian models, and maps the exponential model’s zero-temperature update to an attention step. It distinguishes retrieval of an unseen archetype from outputting a training example, while defining attention statistics that track pooling and concentration.

2 Dataset and models

The exponential model stores vectors as keys, uses the current state as query, and updates by taking the sign of the softmax-weighted sum of keys.One zero-temperature update is one self-attention head with stored vectors as both keys and values; beta is the inverse temperature of the Gibbs measure over keys.Source: §2 · pp. 2–3
Each of K binary archetypes has M noisy examples of quality r; archetypes are hidden, and a query is another example whose quality may be r or 1.The dataset entropy is rho = (1-r^2)/(Mr^2), and the class prototype is the sign of the empirical average of its examples.Source: §2 · p. 2
Generalization is measured by the queried archetype’s Mattis magnetization after one update or at a fixed point, whereas memorization is tracked by overlap with the closest same-class training example.An overlap of one means the state coincides with the corresponding vector.Source: §2 · pp. 2–3
The pairwise networks built on class averages or individual examples are supervised and unsupervised, respectively; the attention comparison uses the supervised network as its reference.The exponential model may use the examples themselves as keys without labels, while class averages are an alternative discussed in the text.Source: §2 · p. 2
Beta near zero is the pairwise regime, while beta tending to infinity gives nearest-key lookup; the logit scales as beta times N times the query-key overlap.The paper distinguishes beta, the inverse temperature of the key-weighting Gibbs measure, from stochasticity in the noiseless neural dynamics.Source: §2 · p. 3
W measures the softmax weight assigned to the correct class, while Meff counts its effectively contributing examples, ranging from pooling all M to concentrating on one.Condensation concerns the weights; memorization concerns the output, and the two need not coincide on structured data.Source: §2 · p. 3

3 Signal-to-noise analysis

The signal-to-noise analysis derives retrieval overlap from class signal and crosstalk, modeling exponential attention as a weighted majority vote with an effective entropy. It examines how temperature governs crosstalk suppression and attention-weight condensation, distinguishing pooling the correct class from retrieving its archetype and identifying example-level memorization.

For the pairwise network, archetypes are stable after one update when alpha(1+rho)^2 + rho is ≲ 1, and lower-quality queries shrink the stable region by q^2.The result generalizes the q = 1 formula of [13].Source: §3 · p. 3
The exponential model’s logits form a planted-block random energy model, and within-class attention freezes when x = beta sqrt(N(1-r^2)) reaches about sqrt(2 ln M).In the frozen phase, a finite number of examples carries the attention weight; for finite M, freezing is a crossover.Source: §3 · pp. 4–5
The exponential one-step overlap follows the majority-rule law with effective entropy rho_eff when the field sums many comparable terms; condensation invalidates that Gaussian approximation.The attention layer acts as a weighted majority vote, while the condensed limit returns the selected example instead.Source: §3 · p. 5
After condensation, the layer returns the training example closest to the query, with overlap approaching r rather than the archetype-retrieval prediction.The Gaussian overlap approximation fails when the effective number of pooled examples is O(1).Source: §3 · pp. 5–6
The best overlap occurs inside the generalization window, where the correct class still shares weight with wrong classes rather than eliminating crosstalk entirely.The optimum is at beta*/beta_lo = 1.3–1.8 for the explored parameters, and the optimal correct-class weight decreases with load.Source: §3 · p. 6

3 Signal-to-noise analysis

Above β_lo, crosstalk becomes exponentially small in Nβ, so it no longer sets the learning limit in that regime.The passage describes crosstalk as extensive in the pairwise model but exponentially small in the exponential model above β_lo.Source: §3 · p. 6
The prototype regime pools the correct class but does not necessarily retrieve its archetype; in the memorization region, the theory overestimates m^(1), whose actual value tends to r.In the prototype regime, ρ_eff tends to ρ.Source: §3 · p. 7
Overfitting in the exponential model is REM-like freezing of attention onto one training example, with 1-RSB rather than the pairwise model’s full RSB.The REM description applies to softmax weights viewed from a fixed query; it is not a claim of a thermodynamic phase transition of the network at finite M.Source: §3 · p. 7
With class averages as keys, the signal-to-noise estimate is m^(1) = erf(1/sqrt(2ρ_eff)); at β → ∞, it reduces to majority-rule nearest-centroid classification.The passage notes that class-average keys prevent within-class condensation, while the transformer-relevant case uses examples as keys.Source: §3 · p. 7

4 Numerical results

One-step attention retrieval remains stable as load increases, unlike the pairwise network, within the analytic temperature window. At higher temperatures, the softmax condenses onto a training example; simulations also test the predicted condensation scaling and attention-entropy relation.

Temperature scan

The temperature scan shows crosstalk at low β, prototype pooling within the analytic window, and memorization at high β, where nmax → 1 and m(1) → r.Magnetization peaks inside the window, near the majority-rule values, while Meff remains close to M.Source: Figure 3 · p. 8, §4 · p. 9

Load dependence

At ρ = 0.06, 0.25, and 0.5, attention magnetization stays at 1.0, 0.90, and 0.75 across load, while the pairwise network declines as α grows.At ρ = 0.5, attention magnetization stays flat even as correct-class weight falls from about 0.3 to 0.1; its crosstalk remains below ρ.Source: §4 · p. 8, Figure 4 · p. 9

Window scaling

The temperature-window edges collapse under different scalings, and the window widens with N, from a factor of four at N = 250 to ten at N = 4000.Rising edges collapse against βN; falling edges collapse against β√N.Source: §4 · p. 9

Condensation

Across tested sizes, dataset sizes, and qualities, Meff follows the random-energy participation ratio when plotted against x = β√N(1 − r²), with condensation essentially complete beyond xc = √(2 ln M).The departure of Meff from M begins near x ∼ 1; at the tested M values, Meff/M is 0.2–0.3 beyond xc.Source: §4 · p. 9

Attention entropy

For ρeff evaluated from the produced softmax weights, simulated one-step magnetization collapses onto erf(1/√(2ρeff)), except in the condensed regime, where deviations reach 0.1.The condensed-regime discrepancy is attributed to failure of the Gaussian approximation.Source: §4 · p. 9

5 Structured data

On MNIST and fashion-MNIST, attention over examples condenses differently: fashion-MNIST retains a tail of weights across examples, and peak accuracy occurs beyond the condensation edge. The diagnostic also reveals within-class structure that departs from the random-energy prediction.

5 Structured data

The attention-entropy diagnostic relates measured one-step magnetization to erf(1/√2ρeff), while the structured-data analysis tests the random-energy prediction using within-class overlaps.For the data analysis, the measured mean and standard deviation of query-to-example overlaps enter the window and condensation variable.Source: §5 · p. 10
On MNIST, the pooled fraction follows the random-energy prediction, while F-MNIST condenses later and retains a larger tail of example weights.At x = 4.4 and x = 10, F-MNIST has Meff/M = 0.11 and 0.023, compared with random-energy values of 0.014 and 0.006.Source: §5 · p. 10
F-MNIST’s compressed upper tail and many nearly equivalent closest examples indicate within-class sub-clusters, which the random-energy model does not capture.The text identifies this hierarchical organization with [13] and points to a hierarchical generalization of the random-energy model.Source: §5 · p. 10
The pooled-fraction diagnostic measures memorization and also detects structure in the examples.On MNIST, standardized within-class overlaps are Gaussian; on F-MNIST, they are not.Source: §5 · pp. 10–11
Peak accuracy occurs beyond the condensation edge: 0.90 on MNIST and 0.80 on F-MNIST, above prototype readouts of 0.71 and 0.62, respectively.At those peaks, Meff/M is 0.01 on MNIST and 0.04 on F-MNIST; F-MNIST's peak exceeds its closest-example limit by 0.03.Source: §5 · p. 11

6 Gradient descent versus Hebb

A gradient-trained one-head attention classifier fits the training set but performs below Hebbian attention at its best temperature in this setup. Its test accuracy peaks early and decays as keys fit example noise; weight decay limits that decay and yields more prototype-like keys.

6 Gradient descent versus Hebb

The trained layer fits the training set but has lower fresh-example test accuracy than Hebbian attention at its best β, which coincides with the nearest-centroid readout.The reference is not claimed to be optimal, and the observations are specific to the reported training setup.Source: §6 · p. 12
Without weight decay, test accuracy reaches the Hebbian value after a few tens of steps, then decays as keys fit example noise.At initialization, the cross-entropy gradient is Hebbian; the keys initially become proportional to class sums.Source: §6 · p. 12
Key weight decay of order 10^-2 brings test accuracy close to the Hebbian reference and raises learned-key cosine with archetypes from 0.4 to 0.7.The reported accuracy gaps are within 0.01–0.03 at r = 0.3 and within 0.03–0.08 at r = 0.2.Source: §6 · p. 12
The comparison uses full-batch Adam with a single learning-rate schedule and two datasets per point, not optimally tuned training.The authors limit these observations to this setup.Source: §6 · p. 12

7 Conclusions and outlook

The paper transfers Hebbian learning from examples to exponential Hopfield networks, where one attention step retrieves unseen archetypes through an attention-adjusted majority-rule law. It finds that learning is bounded by softmax memorization rather than load-driven spin-glass interference, with important limits for structured data and condensed attention.

Conclusions and outlook

The one-step magnetization for an unseen archetype follows the majority-rule form, m^(1)=erf(1/sqrt(2rho_eff)), with attention entropy determined by correct-class weight and example pooling.The update is a softmax-attention layer with training examples as keys and values.Source: §7 · p. 12
Within the temperature window, crosstalk decays exponentially in Nβ, so load no longer controls learning and the number of learnable archetypes grows as exp(c√N).The limiting boundary is the condensation edge rather than crosstalk.Source: §7 · p. 12
Overfitting becomes random-energy condensation onto one example, unlike the pairwise model’s spin-glass phase; its threshold depends on within-class overlap spread.The transition has one-step replica-symmetry breaking.Source: §7 · pp. 12–13
Within the window, examples from the correct class receive weight without labels, while the effective example count is MΦ_M(x) and crosstalk is exponentially small.The weighted-vote entropy unifies pairwise and exponential networks; iteration can instead converge to a training example unless ρ is small.Source: §7 · p. 13
A single attention step generalizes over a wider range than fixed-point iteration, whose attractors are training examples except at small ρ.The authors note this is consistent with feed-forward attention but do not claim it explains that design.Source: §7 · p. 13
The attention head’s temperature moves it from crosstalk through prototype retrieval to memorization, and its generalization window widens with model size.The two window edges scale as 1/N and 1/√N.Source: §7 · p. 13
On structured MNIST and fashion-MNIST classes, attention beyond the condensation edge reads out local sub-prototypes rather than the class prototype.The diagnostic’s departure from the random-energy curve measures deviation from the single-archetype picture.Source: §7 · p. 13
The quantitative signal-to-noise theory assumes unstructured data, one noiseless layer, and Gaussian noise; it is accurate within 0.03 while pooling but can err by up to 0.1 after condensation.The authors leave structured-data theory and learned-projection diagnostics open; gradient-descent observations suggest early stopping and weight decay track distance from the Hebbian solution.Source: §7 · p. 13

A Details of the simulations

The simulation appendix specifies how examples, queries, updates, and reported measurements were generated across the figures and datasets.

A Details of the simulations

Queries are fresh examples from randomly chosen archetypes; one parallel sign update measures one-step quantities, while convergence is run for at most ten updates for m(∞).The exponential-network field is evaluated in logspace for numerical stability.Source: A Details of the simulations · p. 14
Figures 4 and 7 sweep load and entropy grids, with 3 datasets × 6 queries per point and 4 datasets × 8 queries per point, respectively.Figure 4 uses N=4000, M=16, β=0.012; Figure 7 uses N=1000, β=0.025.Source: A Details of the simulations · p. 14
MNIST and fashion-MNIST are binarized at each pixel’s training-set mean, and figures 8–9 evaluate test queries for Hebbian attention and a trained one-head classifier.Figure 8 uses 400 test queries per sample; Figure 9 uses 500 full-batch Adam steps and 400 fresh test examples.Source: A Details of the simulations · p. 14

B Small-x expansion of ΦM

The appendix expands the finite-M participation-ratio function near zero and compares its approximation and bounds with numerical realizations.

B Small-x expansion of ΦM

Expanding the numerator and denominator gives Φ_M(x)=1−(1−M^-1)x^2+O(x^4), because the averaged cubic term vanishes.For M=16 and x=0.3, the expansion gives 0.916 versus 0.921 numerically.Source: B Small-x expansion of ΦM · pp. 14–15
At x=10, the stated bounds are 1.31, 1.40, and 1.50 for M=16, 64, and 256, respectively.Source: B Small-x expansion of ΦM · p. 15
At x=10, simulated E[P_a] is 0.82, 0.77, and 0.72 for M=16, 64, and 256, above the corresponding limits 0.76, 0.71, and 0.67.The reported reciprocal means are 1.21, 1.30, and 1.39, while MΦ_M is 1.34, 1.50, and 1.65, 11–19% above the bound.Source: B Small-x expansion of ΦM · p. 15

C Position of the maximum

The appendix locates the temperature maximizing retrieval by minimizing attention entropy, then gives a small-x approximation and its accuracy range.

C Position of the maximum

The optimum β solves a scalar ∂βρ_eff=0 condition once Φ_M is known.Source: C Position of the maximum · p. 15
For x≪1, neglecting self-coupling reduces the condition to a′=2Lbt²; at q=r, κ=(M−1)ρ²/(1+ρ), with a solution expressed using Lambert W.Source: C Position of the maximum · p. 15
The approximation is within 2% of the numerical maximum for 0.02≤ρ≤0.4, but its error reaches 19% at ρ=1.The expansion assumes x*≪1 and is not uniform as ρ→0.Source: C Position of the maximum · p. 15
Self-coupling shifts the maximizing weight only slightly at the reported finite sizes; asymptotically W* decreases as 1/ln N, while the approximation is accurate at practical sizes.Source: C Position of the maximum · p. 15