Information Geometric Self-Organization at the Edge of Stability in High-Capacity Kernel Associative Memories

Akira Tamamori

Source-linked AI summary

Information Geometric Self-Organization at the Edge of Stability in High-Capacity Kernel Associative Memories

Akira Tamamori

arXiv:2609.16827v1Sep 15, 2026cs.LGcs.NE

TL;DR

High-capacity KLR Hopfield networks have an empirically identified Ridge of Optimization whose geometry and optimization dynamics were unclear. The paper combines Hessian-spectrum analysis with GD trajectory analysis to show that the Ridge borders rank-1 spectral collapse and that EoS dynamics equilibrate curvature near the learning-rate stability limit. These findings place optimal memory representations at highly curved geometric boundaries rather than flat minima.

Problem
The geometry of the high-capacity Ridge of Optimization and the GD mechanism reaching it were previously unclear.
Method
The paper combines Hessian-spectrum analysis, information geometry, GD trajectory analysis, and theoretical models in KLR-trained Hopfield networks.
Results
The Ridge lies adjacent to rank-1 spectral collapse, while EoS dynamics drive maximum Hessian curvature toward the learning-rate stability limit.
Takeaways & Limitations
Optimal high-capacity memory representations are dynamically formed at highly curved boundaries of geometric singularities rather than in flat minima.
Takeaways & Limitations
The study focuses on uncorrelated random binary patterns, and its reduced-order dynamic model provides qualitative rather than rigorous high-dimensional analysis.

Abstract

from arXiv · show

High-capacity associative memories based on Kernel Logistic Regression (KLR) exhibit exceptional storage capabilities and robustness. Previous empirical studies identified a hyperparameter regime, the "Ridge of Optimization," where attractor stability is maximized. However, the geometric nature of this regime and the optimization dynamics required to reach it have remained unclear. In this paper, we investigate the static geometry of the parameter space and the learning trajectory of Gradient Descent (GD) in KLR-trained Hopfield networks. Using the eigenvalue spectrum of the Hessian, we reveal that the Ridge corresponds to a phase boundary located adjacent to a rank-1 spectral collapse, acting as a geometric singularity where the principal curvature is massively amplified. Furthermore, we demonstrate that the learning dynamics exhibit a transient self-stabilizing behavior driven by the Edge of Stability (EoS) phenomenon. Rather than seeking flat regions, the network parameters are driven toward a state where the local curvature dynamically equilibrates near the stability limit dictated by the learning rate, allowing the optimization to survive the initial instability. We provide analytical derivations for both the rank-1 asymptotic collapse and the dynamic feedback loop governing this equilibration. These findings suggest that optimal, high-capacity memory representations are not formed in flat minima, but are dynamically sculpted at the highly curved boundaries of geometric singularities.

1. Introduction

High-capacity KLR Hopfield networks overcome classical storage limits, but the geometry of their Ridge of Optimization and the GD dynamics reaching it remain unclear. This paper characterizes the Ridge through Hessian spectra and explains its emergence through Edge-of-Stability dynamics.

The Ridge of Optimization is characterized as a high-capacity regime where KLR Hopfield networks maintain stable attractors under heavy memory loads.Earlier work showed storage capacities exceeding classical limits and identified this regime empirically.
The paper investigates how GD navigates the high-dimensional parameter space to reach the Ridge, an optimization process that remains insufficiently understood.The analysis combines information geometry with dynamical-systems perspectives.
The paper combines static geometric analysis, learning-trajectory analysis, and theoretical interpretations of spectral collapse and dynamic equilibration.The later sections analyze static geometry, GD dynamics, theoretical mechanisms, implications for flat minima, and limitations.
The Ridge lies adjacent to a rank-1 spectral-collapse region, where optimal memory formation occurs near a singularity with strongly amplified principal curvature.The paper uses the Hessian eigenvalue spectrum and retrieval performance to establish this geometric relationship.
GD exhibits EoS-driven self-organization, dynamically equilibrating maximum Hessian curvature near the learning-rate-dependent stability limit.This behavior is presented as a mechanism by which optimization reaches the Ridge despite initial instability.

2. Background and Preliminaries

The paper models Hopfield retrieval with KLR, using kernel-based dual variables, Fisher geometry, and the regularized Hessian to study representation structure and GD stability. Experiments use random binary patterns, controlled optimization procedures, phase diagrams, and noisy retrieval tests.

KLR trains each neuron's dual variables by minimizing an L2-regularized logistic negative log-likelihood over kernel representations.The Gaussian RBF kernel controls locality through γ, while λ sets weight decay.
The Fisher Information Matrix G(α)=KD(α)K captures intrinsic statistical-manifold geometry through prediction variances and is used to measure spectral concentration.The diagonal entries of D are pμ(1−pμ), with pμ given by the logistic sigmoid.
The complete regularized-objective Hessian H governs GD curvature, with linear stability requiring λ_max(H)<2/η.Because λ is typically small, H≈G in most functional regimes, but H is used for stability evaluation.
Experiments use independent random binary patterns, full-batch GD with zero initialization, η=0.1 by default, λ=0.01, and up to 300 epochs.Static analyses instead use L-BFGS-B with at most 200 iterations and ftol=1e-5.
Phase diagrams span a 30×30 grid of storage loads and kernel localities, while retrieval success tests up to 10 patterns from states with 20% bit-flip noise.Success requires perfect convergence to the target within 30 synchronous retrieval steps.

3. Phenomenology: The Geometry of the Ridge

The Ridge of Optimization forms a phase boundary next to rank-1 spectral collapse rather than a flat optimum. Near this boundary, memory functionality persists with minimal spectral diversity while principal curvature becomes sharply amplified.

The functional memory regime is separated from overload by a phase boundary that coincides with geometric collapse in the Hessian spectrum.Functional points have retrieval success near 1.0, whereas overloaded points have success near 0.0.
As kernel locality decreases, spectral concentration reaches 1.0, indicating rank-1 collapse and loss of the multidimensional structure needed for pattern separation.The phase diagrams track spectral concentration, maximum curvature, and retrieval success across storage load and locality.
The Ridge near γ≈0.02 approaches the rank-1 boundary as storage load increases, preserving minimal spectral diversity while amplifying principal curvature beyond 100.It therefore balances stability and capacity immediately before pattern-separation capability is lost.
Optimal memory performance occurs adjacent to extreme spectral degeneracy, where the network remains functional at the edge of a structural transition rather than in a flat region.The dominant curvature supplies a strong restorative direction while spectral diversity is nearly exhausted.
The Ridge dynamics are tested at P/N=2.0 and γ=0.02 across η∈{0.05, 0.1, 0.2}, using normalized curvature ηλ_max(H)/2 to compare trajectories with the stability threshold.The corresponding figure also plots maximum curvature against the theoretical limit 2/η.

4. Learning Dynamics: Transient Self-Stabilization

Gradient Descent follows a consistent three-phase trajectory near the Ridge: initial instability, transient self-stabilization near the learning-rate stability limit, and eventual stable convergence. This feedback-driven behavior persists across tested learning rates and becomes longer under higher storage loads.

The learning trajectory consistently progresses through initial overshoot, transient self-stabilization, and escape into stable convergence across all tested learning rates.The three-phase structure is observed on the Ridge for η∈{0.05, 0.1, 0.2}.
An initial curvature above 50 exceeds 2/η for every tested learning rate, causing a large update and normalized curvature to spike beyond the stability threshold.Parameters start at zero, where prediction variance and curvature are maximal; normalized curvature above 1.0 indicates operation at or beyond linear instability.
After overshoot, curvature rapidly decreases and oscillates just below the limits 40, 20, and 10, while normalized curvature repeatedly reaches the 1.0 threshold.These limits correspond to η values 0.05, 0.1, and 0.2, respectively.
Gradient Descent uses transient self-stabilization to survive extreme Ridge curvature rather than passively settling into a pre-existing flat minimum.Divergent steep-direction steps dynamically reduce local curvature before the trajectory enters a stable, high-capacity representation.
Higher storage loads preserve the qualitative three-phase pattern but significantly extend the transient self-stabilization phase.

5. Theoretical Analysis of Spectral Collapse and Dynamics

Theoretical analysis links the Ridge to an asymptotic rank-1 collapse of the kernel geometry and explains self-stabilization through curvature-dependent feedback. High-capacity memory formation occurs at the boundary where restorative curvature and pattern-discriminating dimensionality remain balanced.

5.1 Geometric Origin of the Rank-1 Collapse: In the global regime γ≪1/N, a first-order Taylor expansion of the RBF kernel provides the analytic route to the rank-1 asymptotic structure.For binary patterns, the kernel depends on squared Euclidean distance, which can be expressed through pattern inner products.
5.1 Geometric Origin of the Rank-1 Collapse: As γ→0, the RBF kernel approaches the constant matrix J, causing the FIM and, with small regularization, the Hessian to become extremely spectrally degenerate.J has rank 1, so only one eigenvalue remains nonzero while the others approach zero.
5.1 Geometric Origin of the Rank-1 Collapse: The Ridge lies just before spectral collapse, where uniform restorative curvature remains strong while XX^T preserves the dimensionality needed to discriminate patterns.Memory performance is optimal when these opposing structural forces balance at the critical boundary.
5.2 Qualitative Model of Self-Stabilization: The reduced GD model updates the dominant Hessian-direction component by multiplying it by 1−ηλ_max(x_t), making local curvature determine both sign flips and magnitude changes.The model isolates the parameter component along the principal eigenvector and uses the local principal curvature as the scaling factor.
5.2 Qualitative Model of Self-Stabilization: Because logistic variance decreases as parameter magnitude grows, λ_max decreases with |x|, creating negative feedback that drives curvature toward the learning-rate stability boundary.Near zero, predictions are uncertain and variance is maximal; confident predictions at larger |x| reduce variance and curvature.
5.2 Qualitative Model of Self-Stabilization: Initial divergence pushes parameters away from the high-variance origin until curvature respects the learning-rate limit, allowing optimization to survive the Ridge’s singular geometry.The simplified one-dimensional model captures the mechanism even though the full dynamics are higher-dimensional.

6. Discussion

The discussion interprets the Ridge of Optimization as a sharply curved boundary near spectral degeneracy, where GD self-stabilizes through Edge-of-Stability dynamics rather than converging to a flat minimum.

The Ridge lies adjacent to extreme spectral degeneracy, with sharp dominant curvature, and successful learning navigates this highly curved region.The phase diagram places optimal associative-memory operation near a singularity rather than in a flat parameter-space region.
6.2 The Margin-Seeking Tendency and Spectral Concentration: A logistic-loss margin-seeking tendency plausibly drives spectral concentration by increasing weight magnitudes to separate stored patterns in feature space.The objective includes L2 regularization, but the logistic component can still promote larger weights and concentrated spectra.
Prediction variance p(1−p) supplies negative feedback: overshooting increases parameter magnitude, suppresses variance, and reduces local curvature.The mechanism provides a concrete KLR realization of self-stabilization associated with the Edge of Stability.
GD enters a transient oscillatory phase because the principal curvature initially exceeds the stability limit 2/η in the anisotropic Ridge landscape.This behavior reflects the difficulty of navigating a sharp landscape with Euclidean-metric updates.
Curvature-aware methods may help navigate the Ridge, but near-rank-deficient FIMs require damping or pseudo-inverse treatments for safe natural-gradient updates.Regularized natural-gradient methods are reported to bypass the oscillatory phase and follow a smoother Ridge trajectory.
6.5 Limitations and Future Work: The analysis is limited by independent random binary patterns and a qualitative one-dimensional dynamic model, motivating correlated-data and high-dimensional extensions.Correlated data may produce hierarchical low-rank spectra, while rigorous analysis of trailing-eigenvector dynamics remains open.

7. Conclusion

The paper combines spectral geometry and learning-trajectory analysis to explain high-capacity memory formation in KLR Hopfield networks. It finds that optimal representations form near a rank-1 geometric transition and are dynamically shaped by Edge-of-Stability self-stabilization.

The Ridge of Optimization borders extreme spectral degeneracy, while the Fisher Information Matrix asymptotically approaches rank 1 as kernel interactions become global.The functional regime balances massive principal curvature for global stability against spectral diversity for pattern separation.
GD does not converge monotonically to a flat minimum; logistic prediction variance instead drives the maximum Hessian eigenvalue toward the learning-rate stability limit.This transient self-correction lets optimization survive initial instability and form a sharp principal attractor basin.
Optimal high-dimensional kernel memory representations are dynamically formed at highly curved boundaries of geometric singularities.The conclusion also motivates curvature-aware optimization methods for degenerate parameter spaces.

Appendix A: Consistency of Self-Stabilization Across Storage Loads

Across storage loads, GD exhibits the same transient self-stabilization structure near the stability threshold. Higher loads increase initial curvature and prolong the oscillatory phase before stable convergence.

Across P/N∈{2.0, 8.0, 16.0}, trajectories self-stabilize near normalized curvature 1.0 before entering stable convergence at η=0.1.The result indicates that the three-phase learning structure persists across the tested storage loads.
At P/N=16.0, the initial maximum-Hessian-eigenvalue spike reaches orders of magnitude above the P/N=2.0 case because initial curvature scales with stored-pattern count P.This relationship is analytically derived and empirically confirmed in the appendix.
The supplementary results support EoS as a robust learning phase needed to navigate load-dependent curvature before forming deep, stable attractor basins.The authors present this phase as necessary for successful optimization in high-capacity kernel associative memories.