PaperlayerView exact revision on arXiv

Source-linked AI summary

Phase Kernel Lifts Capacity of Dense Associative Memory

Yifei Zhao, Ying Tang

arXiv:2610.21493v1cs.LG

TL;DR

Linear MHN kernels can yield overlapping attractor basins and spurious states under high memory load. The paper introduces nonlinear phase kernels, proves capacity and convergence guarantees under stated conditions, and instantiates them in S-Hop. S-Hop passes through 500 memories on several image datasets, with U-Hop reaching lower passing loads on CIFAR-10 and ImageNet.

  • Problem

    Linear or polynomial MHN kernels can reduce pattern separation as memory load increases, yielding overlapping attractor basins and spurious states.

    Source: §1 · p. 1

  • Method

    The paper introduces nonlinear phase kernels for MHN and instantiates the framework as S-Hop, which reshapes the energy landscape and retrieval dynamics.

    Source: §1 · p. 1, §6 · p. 9

  • Results

    S-Hop passes through 500 memories on MNIST, CIFAR-10, and ImageNet at both resolutions, while U-Hop passes through 500 on MNIST, last passes at 60 on CIFAR-10, and falls below threshold at 10 on ImageNet.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Phase-kernel design provides a practical means of improving dense associative retrieval with an analyzable energy model.

    Source: §6 · p. 9

    The capacity theorem assumes independent, uniformly distributed phases, whereas natural images have correlated pixels and uneven feature norms.

    Source: §6.1 · p. 9

Abstract

from arXiv · show

Dense associative memory underlies modern attention mechanisms; however, Modern Hopfield Networks (MHNs) with linear kernels yield overlapping attractor basins and spurious states under high memory load, limiting their theoretically exponential storage capacity in practice. We construct a class of nonlinear phase kernels that modify the energy landscape and retrieval dynamics, and prove that these kernels preserve exponential storage capacity and local fixed-point convergence under given conditions on kernel smoothness, memory separation, and memory distributions. The framework is instantiated by the Sigmoidal Phase Hopfield Network (S-Hop), which guarantees monotone energy descent and mitigates practical capacity loss. S-Hop has $8.3\times$ and $4\times$ gains in one-step noisy retrieval load over Uniform Hopfield Networks (U-Hop) on CIFAR-10 and TinyImageNet respectively, and reaches a $50\times$ gain in first-failure load for ImageNet. The results demonstrate that phase kernels provide a principled route to robust dense associative memory.

1 Introduction

Linear and polynomial MHN kernels can produce overlapping attractor basins and spurious states at high memory loads. This paper introduces nonlinear phase kernels with theoretical guarantees and evaluates S-Hop’s retrieval performance.

1 Introduction

  • Linear or polynomial kernels can reduce pattern separation as memory load rises, producing narrow, overlapping attractor basins and spurious states.The paper identifies this geometric limitation as motivation for improving separation while retaining the MHN energy structure.Source: §1 · p. 1
  • The proposed nonlinear phase-kernel family is designed to preserve Lyapunov stability and exponential capacity through verifiable structural conditions.The introduction notes that nonlinear feature substitution alone does not establish either property.Source: §1 · p. 1
  • Under stated assumptions, the kernels preserve exponential storage capacity and local fixed-point convergence, while S-Hop uses CCCP for monotone energy descent.The analysis also establishes disjoint local attraction regions under a margin condition.Source: §1 · p. 2
  • 92.07% mean success on CIFAR-10 at N = 500, versus 48.40% for U-Hop’s public avg objective, a 43.67-percentage-point gain over three seeds.On ImageNet, S-Hop passes the 90% success criterion through N = 500 at both 64×64 and 224×224 resolution.Source: §1 · p. 2
  • S-Hop introduces nonlinear phase kernels whose phase potential yields a component-wise inverse local field and an energy-derived retrieval update.The framework is presented as a way to reshape both the MHN energy landscape and retrieval dynamics.Source: §1 · p. 2
  • S-Hop supports more than 500 memories on MNIST, CIFAR-10, and ImageNet.Figure 1 summarizes retrieval-load comparisons on CIFAR-10 and ImageNet across S-Hop, U-Hop, MHN, and C-Hop.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Related work connects MHN capacity and convergence to interaction-function choice, while identifying limitations of linear kernels and distinctions from oscillator, distance-based, and other retrieval approaches.

Modern Hopfield Networks and Attention

  • MHN has exponential memory capacity and a mathematical equivalence to self-attention, while capacity and convergence are sensitive to the interaction function.Prior work also connects MHN energy formulations to diffusion models and generalized Potts models [1] [21] [23] [24] [27].Source: §2 · pp. 2–3

Limitations of Linear Kernels

  • Linear mappings can produce overlapping attractor basins, memory confusion, and spurious states at high memory loads in high-dimensional spaces [3].Biologically plausible two-layer models use threshold nonlinearities but lack continuous-state energy landscapes and convergence guarantees [31].Source: §2 · p. 3

Uniform Retrieval, Oscillatory Dynamics, and Distance-Based Variants

  • U-Hop repositions embeddings through a learned linear projection, whereas S-Hop changes both the embedding and the phase-space retrieval update [7].Oscillator models and honeycomb Kuramoto networks study stable retrieval or exponential capacity outside the attention-equivalent energy framework [33] [34] [35] [37] [38] [39].Source: §2 · p. 3
  • Distance-based overlap methods reshape basin geometry through memory-norm penalties, an axis orthogonal to feature-embedding changes [16] [34].The paper positions its separable strongly convex phase potentials and explicit contraction conditions as complementary to that direction.Source: §2 · pp. 3–4

3 Method

The method embeds phase-valued memories in a nonlinear feature space, derives a phase-space energy and CCCP update, and analyzes convergence and capacity under explicit conditions.

3 Method

  • The method constructs a phase-space surrogate Hopfield energy and derives tractable CCCP fixed-point dynamics for d-dimensional phase states and N stored patterns.The stated objective is to formalize a family of nonlinear phase kernels.Source: §3 · p. 4

3.1 Nonlinear Phase Kernel and Memory Patterns

  • Each memory pattern is represented by a phase vector and embedded into MHN feature space through a nonlinear map F.The framework requires smoothness and derivative bounds plus a structural condition for a tractable implicit phase-space step; S-Hop is one instantiation.Source: §3.1 · p. 4

3.2 Hopfield Energy in Feature Space

  • The standard feature-space MHN energy uses logits X⊤ξ to induce softmax weights, and pulling it back through ξ = F(φ) introduces the Jacobian JF.The kernel structure enables a component-wise implicit update.Source: §3.2 · p. 4

3.3 Surrogate Energy in Phase Space

  • The kernel family represents the regularizer gradient component-wise through a scalar local field, decoupling the implicit step across coordinates.For S-Hop, the local field is smooth and strictly monotone, so it has a component-wise inverse; strong convexity supports contraction analysis.Source: §3.3 · pp. 4–5

3.4 CCCP Fixed-Point Dynamics

  • S-Hop’s CCCP update treats one energy component implicitly and the other explicitly, yielding monotone energy descent with equality only at stationary points.The update preserves the MHN interaction structure.Source: §3.4 · p. 5

3.5 Theoretical Guarantees

  • Under kernel smoothness and contraction conditions with sufficient embedded-memory separation, each stored phase has an invariant ball with a unique attracting fixed point.A similarity-gap margin makes the balls pairwise disjoint, and independent uniform phase coordinates yield exponential storage capacity with probability at least 1 − δ; the exponential order matches standard MHN.Source: §3.5 · p. 6

4 Experiments

The experiments assess one-update noisy retrieval, critical load, kernel and preprocessing ablations, computational efficiency, and CIFAR-10 classification. The main retrieval comparisons use shared memories and queries, with success measured by MSE < 0.1.

4.1 Memory Retrieval Performance and Capacity

  • At N = 500, S-Hop reaches 92.07 ± 0.42% success versus 48.40 ± 3.83% for U-Hop on CIFAR-10, and 98.53 ± 0.12% versus 4.67 ± 0.12% on ImageNet.The ImageNet comparison is at 224 × 224 resolution; Figure 3 plots success and mean retrieval error.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 Critical Capacity Across Datasets

  • S-Hop passes through N = 500 on ImageNet at both resolutions, versus U-Hop’s public avg Nc < 10, a load advantage over 50×; TinyImageNet’s reported gain is 4×.On CIFAR-10, the normalized-logit sweep reaches 5000 for S-Hop, versus 870 for MHN and 880 for U-Hop (max).Source: §4.2 · pp. 7–8, Figure 4 · p. 7

4.3 Ablation of Kernel and Preprocessing

  • The similarity-only S-Hop variant fails the 90% criterion at N = 10, while restoring the joint Jacobian/inverse-field update recovers a curve close to full S-Hop through N = 500.Table 2 describes CIFAR-10 ablation and protocol-check loads as passing endpoints or first failures.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Computational Efficiency Trade-Offs

  • At N = 300 on MNIST, S-Hop takes approximately 1.5 ms per query versus 0.1 ms for MHN and 10–40 ms for U-Hop (max), with the lowest retrieval error.With U-Hop projection training amortized over 256 queries, S-Hop is approximately 7–27 times faster than U-Hop.Source: Figure 5 · p. 8, §4.4 · p. 9

4.5 Energy-Shaped Classification on CIFAR-10

  • A CIFAR-10 classifier using early S-Hop and later MHN layers achieves approximately 90% test accuracy, about 0.2 percentage points above the best baseline peak in the 100-epoch run.The S-Hop layers replace attention-based aggregation modules.Source: §4.5 · p. 9

5 Discussion

The discussion describes retrieval conditions and complementary factors that shape robustness, including PCA denoising, memory geometry, feature scale, and retrieval-map components.

5 Discussion

  • The original capacity experiments use β = 1, σ = 0.3, one update, and MSE < 0.1 success; repeated updates are connected to the convergence analysis.Appendix D.2 examines how thresholds, identification criteria, and additional updates affect retrieval.Source: §5 · p. 9
  • PCA denoising uses spectral shrinkage with σ2 PCA = 0.01 and improves retrieval at σ = 0.5 without changing the phase-kernel update.The load–error curves support preprocessing as a complementary source of robustness.Source: §5 · p. 9
  • Memory geometry affects retrieval alongside input dimension, and target-removal controls link performance to memory separation and stored-memory information.The ImageNet resolution comparison extends this observation to two input dimensions.Source: §5 · p. 9
  • Scale and component controls identify contributions from feature scale and the complete retrieval map; extending admissibility conditions to learned phase kernels could adapt retrieval to correlated data.S-Hop and P-Hop share admissibility conditions but differ in their Jacobians, inverse local fields, and finite-dimensional performance.Source: §5 · p. 9

6 Conclusion

This section derives the gradient of the log-sum-exp term in Eq. (13) and concludes by presenting nonlinear phase kernels as a way to preserve analyzable energy dynamics while improving dense associative retrieval. It also notes the independent-uniform-phase assumption in capacity theory and identifies memory selection and further theory as open directions.

6 Conclusion

  • The nonlinear phase-kernel framework reshapes MHN energy and retrieval dynamics, preserves exponential capacity under independent random phases, and gives S-Hop monotone energy descent.Separable strong convexity makes the inverse local field tractable; explicit contraction and memory-separation conditions establish local convergence.Source: §6 · p. 9
  • S-Hop passes through 500 memories on MNIST, CIFAR-10, and ImageNet at both resolutions; U-Hop passes 500 on MNIST, 60 on CIFAR-10, and falls below threshold at 10 on ImageNet.The U-Hop comparison uses public avg on MNIST and last on CIFAR-10 and ImageNet.Source: §6 · p. 9

6.1 Limitations

  • The capacity theorem assumes independent uniform phases, while natural images have correlated pixels and uneven feature norms; better-separated, norm-matched TinyImageNet memories raise passing load from 40 to over 500.The conclusion identifies correlated-distribution theory and independently sampled memory sets as extensions, alongside stricter retrieval criteria, ablations, energy-based gating, and faster kernel evaluation.Source: §6.1 · pp. 9–10

A.2 Kernel Family Viewpoint

  • The kernel-family analysis requires a separable, strongly convex phase potential with bounded Hessian and third derivative, plus a contraction condition; these checks hold for S-Hop and P-Hop.S-Hop has κ ≈1.477 and P-Hop κ ≈2.09; the linear embedding recovers standard MHN.Source: §A.2 · pp. 18–19

A.3 Verification of the CCCP Structural Identity

  • For S-Hop, the CCCP structural identity yields a component-wise invertible update, and global strict convexity of the interaction term makes it a genuine CCCP with monotone energy descent.For a general kernel in the family, the iteration is only CCCP-inspired because interaction-term convexity is not guaranteed.Source: §A.3 · p. 19

A.4 Regularity of the Kernel and Dynamics

  • On a compact phase region, the kernel dynamics are smooth, with bounded norms and Jacobians and globally Lipschitz inverse-field and kernel-composition maps.The inverse local field is globally Lipschitz by strong monotonicity of the local field.Source: §A.4 · p. 20

A.5 Strong Convexity of the Local Phase Potential

  • The local phase potential is globally strongly convex, so its gradient is strongly monotone and Lipschitz, and its inverse is globally Lipschitz on its range.The potential is separable and its Hessian has global eigenvalue bounds.Source: §A.5 · pp. 21–22

A.6 Gradient of the Log-Sum-Exp Term

  • The log-sum-exp gradient follows by applying the chain rule to the softmax weights and collecting terms, yielding Eq. (13).Source: §A.6 · p. 22

A.7 Global Strict Convexity of E2 for S-Hop

The analysis establishes how global strict convexity of S-Hop’s interaction term enables CCCP energy descent and develops local convergence and capacity guarantees through stored-pattern approximation, invariant balls, and Jacobian bounds, including contraction of the update map on an invariant ball under the stated conditions.

A.7 Global Strict Convexity of E2 for S-Hop

  • S-Hop’s interaction term E2 is globally strictly convex, making its update an instance of CCCP for the difference-of-convex energy E = E1 − E2.This is an additional guarantee beyond the contraction-based analysis that applies to the broader kernel family.Source: §A.7 · pp. 22, 24

A.8 Strict CCCP Descent for S-Hop

  • For any initialization, S-Hop’s CCCP update decreases energy, with equality only at stationarity; its energy sequence converges and every iterate-sequence accumulation point is stationary.The paper contrasts this global descent guarantee with P-Hop, for which convexity of E2 is not established and convergence relies on contraction arguments.Source: §A.8 · pp. 25–26

B Convergence, Storage Capacity, and Instantiation Comparison

  • The broader analysis combines local approximation, invariant-ball and Jacobian-contraction results with pattern separation to establish an exponential storage-capacity theorem, then compares S-Hop and P-Hop.Its comparison covers condition number, radicand positivity, and convergence guarantees.Source: B Convergence, Storage Capacity, and… · p. 26

B.1 Local Approximation at a Stored Pattern

  • Under sufficient margin, one S-Hop phase update maps a stored pattern close to itself, with feature-space error bounded by 2L_F Mε0.The bound follows by mapping the phase-space update through F.Source: §B.1 · pp. 26–27

B.2 Invariant Balls and Margin Conditions

  • Under pattern-separation and regularity conditions, the phase-space map preserves a small neighborhood around each stored pattern; these balls lie in the full phase space, where smoothness permits mean-value arguments.The neighborhoods are invariant under the stated conditions.Source: §B.2 · pp. 27–28

B.3 Jacobian Bounds and Contraction

  • A Jacobian bound establishes contraction on each invariant ball, with the S-Hop bound yielding ρ0 < 1.The associated constants depend on kernel and regularity parameters, not on the number of stored patterns.Source: §B.3 · p. 29
  • On S_i, the Jacobian norm satisfies ∥JΦ(φ)∥2 ≤ ρ0 + Cε ε + Cr r, with the stated radius bound yielding (102).Here Cε is independent of N, and r ≤ 1/(βNMLmax) for φ ∈ S_i.Source: §B.3 · p. 32
  • A uniform contraction factor follows by bounding the Jacobian norm strictly below 1 throughout S_i and applying the line-segment argument.The proof uses continuity of JΦ on the compact invariant ball to obtain a uniform ρ < 1.Source: §B.3 · p. 33

B.4 Correct Storage and Separation of Basins

Under the stated kernel and regularity conditions, each stored pattern has a unique attracting fixed point in its own disjoint basin. For random phases, the framework establishes exponential-order storage with high probability, and both S-Hop and P-Hop instantiate the analyzed kernel family.

B.4 Correct Storage and Separation of Basins

  • Each stored pattern has a unique attracting fixed point, and the phase-space balls around memories are pairwise disjoint, so initialization within a ball retrieves its corresponding memory.The feature-space trajectories also converge from mapped points in these balls.Source: §B.4 · p. 34

B.5 Exponential Storage Capacity

  • For random phases and the stated kernel conditions, storage is exponential in dimension with probability at least 1 − δ, with each phase-space ball containing a unique attracting fixed point.The result follows when the random-memory margin condition holds.Source: §B.5 · p. 35

B.6 Theoretical Comparison of P-Hop and S-Hop Instantiations

  • Both P-Hop and S-Hop share the feature dimension, CCCP structural identity, phase-space update form, and stated local-convergence and storage results, despite differing in scalar features, potentials, and regularity constants.The results apply to both kernels under their stated conditions.Source: §B.6 · p. 36

C Additional Numerical Results

  • Supplementary experiments span cross-dataset retrieval, a second kernel, alternative metrics and protocols, ablations, gating, MIL, and downstream classification, using MSE < 0.1 unless stated otherwise.Passing sweep endpoints are reported as lower bounds.Source: C Additional Numerical Results · p. 37

C.1 MNIST Retrieval and Capacity

  • On MNIST, S-Hop reaches 100% success at N = 500, while U-Hop (max) has a largest passing load of 40; the public avg objective reaches 500 with 92.93 ± 1.17% success.The protocol uses one-step retrieval with β = 1, noise σ = 0.3, and the MSE < 0.1 criterion.Source: §C.1 · p. 38, Figure 6 · p. 38, Figure 7 · p. 38

C.2 Framework-Level Validation: A Second Kernel Instance

  • On CIFAR-10, S-Hop achieves higher retrieval success and lower error than P-Hop over tested loads, while P-Hop improves on the baselines.P-Hop is a second admissible kernel instance within the framework.Source: Figure 9 · p. 39, §C.2 · p. 39

C.3 Robustness Under the U-Hop SSD Metric

The appendix aligns the SSD criterion with the main MSE threshold and evaluates retrieval under the U-Hop protocol. S-Hop remains below the aligned error threshold through the full tested load range, while the baselines cross it early.

C.3 Robustness Under the U-Hop SSD Metric

  • SSD < 0.1 · d ≈ 307 is equivalent at the pattern level to MSE < 0.1, with targets and outputs clamped to [0, 1].The comparison uses CIFAR-10, d = 3072, noise σ = 0.3, and one retrieval step.Source: §C.3 · p. 40
  • Under the aligned criterion, S-Hop passes through N = 500; the figure marks baseline bars at their threshold-crossing loads.The plotted loads follow the original endpoint/crossing convention.Source: Figure 10 · p. 40, §C.3 · p. 40
  • S-Hop remains below the aligned SSD threshold across N = 10–500, while all baselines cross it by about N = 30.Figure 11 reports perfect retrieval rate and mean SSD against stored-pattern count.Source: Figure 11 · p. 40

C.4 Alternative U-Hop Evaluation Protocol (Avg vs. Max)

Under the legacy avg protocol, S-Hop sustains retrieval through the tested load range, while U-Hop falls below the success threshold at N = 60. A centered dot-product control also supports smaller tested loads than S-Hop.

C.4 Alternative U-Hop Evaluation Protocol (Avg vs. Max)

  • S-Hop remains above threshold through N = 500 under the legacy avg protocol, while U-Hop achieves 81.7% success at N = 60.The legacy implementation averages log exp(−t∥ui − uj∥2) over memory pairs with t = 2.Source: §C.4 · p. 41, Figure 12 · p. 41
  • Under the avg protocol, S-Hop maintains above-threshold retrieval through the full tested range and lower error at larger loads, consistent with the max-protocol separation.At small loads, U-Hop’s learned separation loss can yield lower error by uniformizing memory distribution [7].Source: Figure 13 · p. 41, §C.4 · p. 42

C.5 Comparison with a Centered Dot-Product Control

  • S-Hop supports larger tested loads than the centered H-Hop− dot-product control, whose plotted threshold-crossing load is N = 40.The control retains pattern-space centering but removes the per-pattern norm penalty.Source: §C.5 · p. 42, Figure 14 · p. 42
  • Removing the norm penalty degrades H-Hop− toward the linear baseline, whereas S-Hop preserves low error across the tested range.The comparison identifies phase-kernel embedding and distance-based overlap as structurally distinct design axes [34].Source: §C.5 · p. 42, Figure 15 · p. 43

C.6 Extended Ablations and Diagnostics

Extended tests report the largest degradation when the nonlinear kernel is removed and describe retrieval gains from gating. The appendix also evaluates MIL and a hybrid classifier, with the latter slightly exceeding the best baseline peak.

C.6 Extended Ablations and Diagnostics

  • Removing the nonlinear kernel causes the largest retrieval degradation across the capacity range; PCA mainly stabilizes retrieval under moderate noise and reduces error variation under noise.Random truncation and cosine-kernel variants also yield higher error than S-Hop in these comparisons.Source: §C.6 · p. 43, Figure 16 · p. 43

C.7 Gating Wrapper: Additional Retrieval Gains

  • Gating improves retrieval robustness as load increases, with higher perfect-retrieval rates and lower mean MSE; gated S-Hop retains the largest passing load among gated variants.The wrapper mixes a proposed MHN, U-Hop, or S-Hop update with the current state through a gate.Source: §C.7 · pp. 43–44, Figure 17 · p. 44, Figure 18 · p. 44

C.8 Additional Generalization: MIL

  • The MIL benchmark reports learning curves and best test accuracy across repeated runs, but the supplied results do not state a comparative outcome.The appendix frames this as evaluation beyond the CIFAR-10 classifier.Source: §C.8 · p. 45, Figure 19 · p. 45

C.9 Energy-Shaped Classification on CIFAR-10

  • A hybrid classifier with early S-Hop layers followed by MHN layers reaches about 90% test accuracy, approximately 0.2 percentage points above the best baseline peak in the 100-epoch run.The hybrid uses two S-Hop blocks followed by four MHN blocks.Source: §C.9 · pp. 45–46, Figure 20 · p. 45

D Additional Retrieval Analyses

  • The additional retrieval analyses examine feature scale, retrieval criterion, update count, and kernel components, reporting passing sweep endpoints as lower bounds.The supplied passage specifies the scope of these analyses rather than their outcomes.Source: D Additional Retrieval Analyses · p. 46

D.1 Scale, Centering, and Minimum-Margin Controls

The controls test whether S-Hop’s retrieval advantage persists under changes to logit scale, input centering, and minimum-margin calibration. Normalized logits retain high-load success, while the two geometry controls also let S-Hop pass substantially higher loads than the baselines.

Scale controls

  • At N = 500, logit-scale matching yields 91.47±0.76% success for S-Hop, versus 3.40±0.00% for MHN and 4.40±0.69% for U-Hop.These are mean success rates with sample standard deviations over three seeds.Source: §D.1 · p. 46, Figure 21 · p. 46
  • Normalized logits have entropy near log 500 ≈ 6.215 for all methods at N = 500.Entropy and logit spread are averaged over 64 calibration queries per seed; feature norms are averaged over stored memories.Source: §D.1 · p. 46, Table 5 · p. 47
  • S-Hop passes through N = 5000 with normalized logits, versus last passing loads of 870 for MHN and 880 for U-Hop.The extended sweep reports tested-load ratios of approximately 5.75 and 5.68, respectively.Source: §D.1 · p. 47, Figure 22 · p. 48

Centering control

  • Input centering lets S-Hop and U-Hop pass through N = 1000, while MHN last passes at 500 and fails at 510.Centering only stored keys shifts all logits for each query by the same scalar, preserving softmax weights.Source: §D.1 · p. 47, Figure 23 · p. 48

Minimum-margin control

  • With signed minimum-margin matching, S-Hop passes at 790 and fails at 800, while both baselines fail at N = 10.The calibration requires nonzero margins with the same sign so its multiplier is positive.Source: §D.1 · p. 47, Figure 23 · p. 48

D.2 Retrieval Criteria and Multiple Updates

These controls show that retrieval outcomes depend on the success criterion and update dynamics, and test which S-Hop components and preprocessing choices matter. MSE success can remain high even when output identification is poor, while repeated updates at higher load degrade nearest-memory accuracy.

D.2 Retrieval Criteria and Multiple Updates

  • At MSE < 0.1, S-Hop last passes at N = 750 and first fails at 800; at stricter thresholds 0.01, 0.03, and 0.05, it fails the 90% criterion at N = 10.The sweep uses one update, σ = 0.3, and seed 42.Source: §D.2 · p. 49, Figure 24 · p. 49
  • At N = 500, S-Hop has 91.53% MSE success, 13.4% cosine-threshold success, and 1.53% output-cosine top-1 accuracy.Cosine-threshold success requires output–target cosine similarity ≥ 0.95; top-1 selects the stored memory with the highest cosine similarity.Source: §D.2 · p. 49, Figure 25 · p. 50
  • At N = 60, S-Hop nearest-memory accuracy falls from 100% to 15% by step 5 and stays there through step 20, despite a successive-update RMSE of 0.00250 at step 20.Mean MSE decreases from 0.05026 after one update to 0.04497 after two, then rises to 0.05836 at step 20.Source: §D.2 · pp. 49–50, Table 7 · p. 50, Figure 27 · p. 52
  • At N = 10, S-Hop mean MSE decreases from 0.04177 after one update to 0.01653 after 20, while nearest-memory accuracy changes from 100% to 90%.The common-norm experiment evaluates updates at t ∈ {0, 1, 2, 5, 10, 20}.Source: §D.2 · p. 49, Figure 26 · p. 51

D.3 Similarity, Dynamics, Dimension, and Preprocessing

  • The similarity-only variant falls below 90% success at N = 10, while the joint Jacobian and inverse-field variant tracks full S-Hop through N = 500.The ablation measures the combined effect of restoring those two operations.Source: §D.3 · p. 53, Figure 28 · p. 53
  • S-Hop achieves higher retrieval success than U-Hop equipped with a learned 2d × d projection.The preprocessing controls compare S-Hop with and without PCA, random truncation to 1024 coordinates, and higher query noise σ = 0.5.Source: §D.3 · p. 53, Figure 28 · p. 53, Figure 29 · p. 54

D.4 Cross-Dataset Separation and ImageNet

Cross-dataset controls examine how image separation and subset composition relate to retrieval load, followed by ImageNet tests at two resolutions. The reported results show that input dimension alone does not explain the observed TinyImageNet degradation.

Cross-dataset separation and TinyImageNet

  • On the original TinyImageNet subset, under 64 × 64 images, one update, σ = 0.3, and a 90% MSE < 0.1 threshold, S-Hop reaches retrieval load 40 versus 10 for each baseline, a 4× gain.The subset uses the first 500 class-sorted images, all from the goldfish class.Source: §D.4 · p. 54, Table 8 · p. 54
  • Resizing the same 500 TinyImageNet images from 64 × 64 to 32 × 32 raises mean MSE < 0.1 success from 50.67% to 54.40%.The original natural images have correlated content and varying feature norms, unlike the independent phase coordinates assumed in the capacity theorem.Source: §D.4 · p. 54, Figure 30 · p. 55
  • For the geometry-selected TinyImageNet subset at N = 500, S-Hop reaches 100% MSE-based success and 95.27 ± 0.46% nearest-memory accuracy.The original subset achieves 50.53% MSE-based success and 7.60% nearest-memory accuracy under the same noise seeds; the selected subset spans 183 classes and has all 500 margins positive.Source: §D.4 · p. 56, Figure 32 · p. 56

ImageNet

  • On ImageNet at N = 500, S-Hop reaches 91.4% success at 64 × 64 and 98.2% at 224 × 224, while passing through N = 500 at both resolutions.U-Hop’s largest passing load is 10; at 224 × 224, U-Hop (block) has 2.0% success at N = 500.Source: §D.4 · p. 56, Figure 33 · p. 57
  • The ImageNet results and fixed-resolution TinyImageNet subset control show that input dimension alone does not explain TinyImageNet degradation; image distribution and separation also affect retrieval.The ImageNet runs use the first 500 training images, all from the tench class.Source: §D.4 · p. 56

D.5 Public Average-Separation Baseline and Training Diagnostics

This section evaluates the public average-separation baseline against S-Hop and linear Hopfield baselines across image datasets, including scale controls, and documents the released code, reproducibility materials, and research disclosures.

D.5 Public Average-Separation Baseline and Training Diagnostics

  • Training fits the projection on the first half of each memory set for 100 SGD epochs, then fixes it; setting W = I reproduces MHN exactly in checked runs.The CIFAR-10 controls use the first 500 training images, seeds 42–44, one update, β = 1, and MSE < 0.1 as the success criterion.Source: §D.5 · p. 57
  • At N = 500 on CIFAR-10, S-Hop success is 92.07 ± 0.42% raw and matched-scale, versus U-Hop’s 48.40 ± 3.83% and 48.87 ± 3.93%; normalization raises linear baselines to 90.80% and 90.60%.Raw-logit U-Hop last passes at N = 60 and first fails at N = 100, while S-Hop passes through N = 500.Source: §D.5 · p. 57, Figure 34 · p. 58
  • At N = 500 on ImageNet, public avg reaches 7.00 ± 1.06% at 64 × 64 and 4.67 ± 0.12% at 224 × 224, versus S-Hop’s 91.47 ± 0.12% and 98.53 ± 0.12%.The warmed-up MNIST timing at N = 300 is 1.24 ms per query, including 0.17 ms for retrieval after amortizing projection training over 256 queries.Source: §D.5 · p. 57
  • Public avg reaches 92.93 ± 1.17% on MNIST at N = 500, but on TinyImageNet it fails at N = 10 and reaches 3.20% at N = 500.The MNIST timing is averaged over three seeds.Source: §D.5 · p. 57

2. Limitations

  • The authors release anonymized S-Hop and P-Hop source code and a capacity-evaluation pipeline under the MIT license, with setup instructions, pinned dependencies, commands, and protocol configurations.The materials include exact commands to reproduce Tables 1–2 and Figures 3–20.Source: §2 · p. 65
  • The research does not involve crowdsourcing, human subjects, or human-subject risks requiring IRB approval.Source: §2 · pp. 65–66
  • The core method development and theoretical derivations do not use LLMs as important, original, or non-standard components.Source: §2 · p. 66
Loading 2610.21493v1…