PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry of learning dynamics: Gradient descent versus natural gradient on the ridge of optimization

Akira Tamamori

arXiv:2609.16805v2cs.LGcs.NE

TL;DR

The paper asks how learning reaches the Ridge of Optimization, a structured regime whose learning dynamics had not been geometrically analyzed. It compares GD and NGD trajectories on the statistical manifold of a KLR-trained Hopfield network, finding two learning phases and faster, more stable NGD convergence with superior generalization.

  • Problem

    How learning navigates parameter space to reach the Ridge remains unanswered, despite its importance for understanding convergence and self-organization.

    Source: §1 · p. 1

  • Method

    The paper geometrically analyzes learning trajectories on the KLR statistical manifold, comparing GD and NGD against the e-geodesic baseline.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    NGD corrects for Ridge curvature and achieves faster, more stable convergence with superior generalization, while GD follows oscillatory, non-geodesic paths.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The authors conclude that high spectral concentration provides a setting suited to information-geometric optimization methods.

    Source: §7 · p. 8

    The analysis provides a detailed geometric picture in a specific, highly structured regime.

    Source: §6.4 · p. 8

Abstract

from arXiv · show

High-capacity associative memories based on Kernel Logistic Regression (KLR) exhibit a "Ridge of Optimization" characterized by extreme stability and a highly skewed weight spectrum. However, the dynamical process by which learning converges to this critical regime has remained unclear. This paper provides a geometric analysis of the learning trajectories on the statistical manifold of a KLR-trained Hopfield network. By comparing the paths of Gradient Descent (GD) and Natural Gradient Descent (NGD), we elucidate the mechanisms governing the optimization process. Our analysis reveals that learning on the Ridge proceeds in two distinct phases. We show that the extreme curvature of the Ridge causes standard GD to follow a highly oscillatory, non-geodesic path. In stark contrast, NGD explicitly corrects for this geometry, following the ideal geodesic path and completely overcoming the instabilities faced by GD. We demonstrate experimentally that NGD not only converges significantly faster but also achieves a solution with superior generalization performance. These results establish that the highly structured geometry of the Ridge is optimally suited for information-geometric optimization, providing a new perspective on the interplay between learning dynamics and emergent representation geometry.

1. Introduction

The paper asks how learning reaches the highly structured Ridge of Optimization and analyzes that process geometrically by comparing GD with NGD. It reports two learning phases and finds that NGD follows a more stable path, converges faster, and achieves lower validation loss.

1. Introduction

  • The unresolved question is how learning navigates parameter space to reach the Ridge, despite prior understanding of the converged states.The Ridge is a hyperparameter regime in which attractors have maximal stability and a highly concentrated, “L-shaped” weight spectrum.Source: §1 · p. 1
  • The study geometrically analyzes learning trajectories on the statistical manifold, comparing GD with NGD, which accounts for the manifold’s curvature.Source: §1 · pp. 1–2
  • On the Ridge, learning has two phases: rapid establishment of the dominant FIM spectral mode, followed by prolonged fine-tuning of the remaining modes.Source: §1 · p. 2
  • GD makes highly oscillatory, non-geodesic paths that overshoot the principal direction, whereas NGD follows the ideal e-geodesic directly to the optimum.Source: §1 · p. 2
  • NGD corrects for the Ridge’s curvature, bypasses GD instabilities, converges significantly faster, and achieves lower final validation loss.The paper interprets the lower validation loss as indicating superior generalization.Source: §1 · p. 2

2. Related Work

Related work connects the study to information geometry and to research on learning trajectories in deep networks. The paper positions its comparison of GD and NGD on the Ridge within those areas.

2. Related Work

  • The paper situates its work at the intersection of information geometry, optimization theory, and neural-network dynamics.Source: §2 · p. 2

2.1 Information Geometry and Optimization

  • NGD uses the Fisher Information Matrix to account for manifold curvature, but its practical use is often hindered by the cost of FIM inversion.The paper presents its Ridge experiments as evidence of NGD’s superiority in a high-dimensional setting and as motivation for efficient approximations.Source: §2.1 · p. 2

2.2 Learning Trajectories in Deep Neural Networks

  • Beyond lazy training, learning dynamics interact more complexly with the loss landscape; skewed curvature spectra and Edge of Stability behavior frame the paper’s comparison of oscillatory GD and stable NGD.The paper also argues that associative-memory generalization can occur in an extremely sharp basin when optimization respects the manifold’s geometry.Source: §2.2 · p. 2

3. Model and Geometric Framework

The paper frames KLR Hopfield training geometrically, treating its weights as points on a statistical manifold and defining the Ridge through FIM curvature. It compares GD and NGD under specified experimental settings.

3. Model and Geometric Framework

  • The framework combines a KLR Hopfield model with information geometry to analyze learning dynamics.Source: §3 · p. 3

3.1 Kernel Logistic Regression Hopfield Network

  • The model trains dual weights for a representative neuron on stored patterns using an L2-regularized negative log-likelihood objective and full-batch GD.Its inputs use an RBF kernel; the target bit and logistic sigmoid define the prediction terms.Source: §3.1 · p. 3

3.2 Statistical Manifold and Fisher Information

  • The FIM defines the statistical manifold’s metric, while the e-geodesic from initial to optimal weights serves as the direct-path baseline.For the logistic model, the natural parameters are e-flat coordinates, so the e-geodesic is a straight line in weight space.Source: §3.2 · p. 3

3.3 The Ridge of Optimization

  • The Ridge is the locus in (γ, P/N) where the largest FIM eigenvalue at converged weights is locally maximized.The paper describes the Ridge as an a posteriori-defined region coinciding with spectral concentration, with λ1 ≫ λ2 ≥ ··· > 0.Source: §3.3 · p. 3

3.4 Natural Gradient Descent

  • NGD preconditions the standard gradient with the inverse FIM to follow steepest descent on the curved statistical manifold.The paper uses a damped inverse for numerical stability.Source: §3.4 · p. 4

3.5 Experimental Setup

  • Unless otherwise specified, experiments use N=50 neurons, P=100 patterns, γ=0.02 for Ridge and γ=0.1 for Local, learning rate 0.1, weight decay 0.01, and NGD damping ε=10^-3.Source: §3.5 · p. 4

4. The Geometry of Learning Trajectories

The section analyzes learning dynamics on the Ridge through information-gain decomposition and projected trajectories. It finds distinct learning phases and sharply different paths for GD and NGD, with NGD closely following the geodesic.

4.1 Two-Phase Learning via Pythagorean Decomposition

  • The two-phase proposition is an interpretive model, not a strict dynamical theorem.The passage refers readers to [11] for rigorous analysis of spectral concentration and the stability limit.Source: §4.1 · p. 4
  • Ridge learning separates into an early Principal Gain spike and prolonged Tail Gain decay, whereas Local-regime gains decay concurrently.The Principal Gain spike occurs at t < 25; the figure compares Principal and Tail Gains across Ridge and Local regimes.Source: §4.1 · pp. 4–5, Figure 1 · p. 5

4.2 Learning Trajectory on the Statistical Manifold

  • On the Ridge, GD’s path ratio is ≈8.9× versus NGD’s ≈1.0×; GD oscillates away from the direct path while NGD follows the e-geodesic.The contrast is quantified by trajectory length relative to the direct initial-to-final distance.Source: Figure 2 · p. 5, §4.2 · p. 6
  • Extreme Ridge curvature makes GD overshoot and oscillate along the principal direction; inverse-FIM NGD corrects curvature and follows a direct route.The oscillatory behavior is reported as robust across storage loads and kernel localities under high memory congestion.Source: §4.2 · pp. 5–6

5. Geometric Alignment and Generalization on the Ridge

The section evaluates whether NGD’s geometric alignment improves optimization and generalization on the Ridge. NGD converges faster and achieves lower final validation loss than GD in this setting.

5. Geometric Alignment and Generalization on the Ridge

  • NGD is evaluated as a theoretical baseline rather than a computationally competitive alternative to modern adaptive optimizers; validation uses 20% of generated patterns.Source: §5 · p. 6

5.1 Overcoming Instability at the Edge of Stability

  • Despite concerns that the nearly singular FIM makes inversion ill-posed, damped NGD decreases smoothly while GD initially oscillates on the Ridge.The paper attributes NGD’s stability to inverse-FIM preconditioning, which mitigates the Edge of Stability phenomenon.Source: §5.1 · pp. 6–7

5.2 Generalization Performance: GD vs. NGD

  • NGD converges faster and achieves lower final validation loss than GD on the Ridge.In this setting, GD’s oscillations do not appear to confer a regularization benefit.Source: Figure 4 · p. 7, §5.2 · p. 7

6. Discussion

The discussion interprets GD’s oscillations as a two-phase learning heuristic, while NGD’s geometry correction better suits the sharply curved Ridge. It also identifies scope, optimizer-comparison, computational-cost, and robustness limits.

6.1 GD’s Oscillatory Dynamics as a Heuristic for Feature Learning

  • GD’s initial oscillations prioritize dominant structural features before a slower fine-tuning phase, a process interpreted as hierarchical feature learning.The discussion links this two-phase trajectory to information gain decomposition and the Edge of Stability.Source: §6.1 · p. 7

6.2 The Optimality of Natural Gradient on the Ridge

  • NGD corrects for the Ridge’s extreme curvature and achieves faster convergence and better generalization than GD.The discussion describes the Ridge FIM as nearly singular and says NGD explicitly inverts it to flatten the landscape and follow the geodesic.Source: §6.2 · p. 7

6.3 Rethinking the Role of “Flat Minima”

  • For this structured problem, NGD’s superior generalization challenges the idea that flatness alone determines solution quality; alignment with manifold curvature may matter instead.The Ridge solution is sharp in its dominant spectral direction, yet NGD generalizes better than GD.Source: §6.3 · p. 8

6.4 Limitations and Future Work

  • The study focuses on RBF-kernel KLR Hopfield networks trained on uncorrelated random patterns, leaving transfer to other kernels and structured real-world data open.The authors identify the effects of data correlations on Ridge geometry and learning trajectories as a next step.Source: §6.4 · p. 8
  • The optimizer comparison is limited to vanilla GD and exact NGD, and first-order NGD may converge slowly near the optimum.The authors propose evaluating second-order methods, including Dual Riemannian Newton, in the extreme-curvature Ridge setting.Source: §6.4 · p. 8
  • Forming and inverting the FIM or Hessian costs O(P^3) per update; low-rank and adaptive approaches are proposed as possible alternatives.The authors suggest adaptive natural-gradient algorithms could potentially reduce per-update complexity to O(P^2), but whether they reproduce exact NGD’s stable convergence remains open.Source: §6.4 · p. 8
  • The study analyzes learning dynamics, leaving robustness to adversarial attacks and data drift for future work.Source: §6.4 · p. 8

7. Conclusion

The conclusion presents learning on the Ridge as a two-phase process and contrasts GD’s oscillatory path with NGD’s geometry-corrected convergence. It argues that this analysis clarifies self-organization in kernel associative memory.

7. Conclusion

  • The trajectory comparison moves beyond static attractor analysis toward a dynamic account of how optimal memory representations form.Source: §7 · p. 8
  • On the Ridge, GD oscillates initially and prioritizes global stability before fine-tuning, while NGD corrects for geometry to converge faster, more stably, and with superior generalization.Source: §7 · p. 8
  • The findings clarify self-organization in kernel associative memory and support information-geometric optimization in regimes of high spectral concentration.Source: §7 · pp. 8–9

A. Information Gain Decomposition

The appendix briefly derives the information gain decomposition used in Section 4.1.

A. Information Gain Decomposition

A.1 Information Gain as KL-Divergence

The appendix relates one-step learning progress to KL-divergence reduction, then approximates that divergence for an infinitesimal parameter change using the FIM.

A.1 Information Gain as KL-Divergence

  • For maximum likelihood estimation, one-step learning progress is approximately the KL-divergence between the model distributions at steps t and t+1.Source: §A.1 · p. 9
  • For an infinitesimal parameter change, the KL-divergence is approximated to second order by a quadratic form using the current-point FIM.Source: §A.1 · p. 9

A.2 Choice of a Fixed Reference Metric

The analysis uses the Fisher information matrix (FIM) at the converged solution as a fixed reference metric. This projection-based simplification reveals a hierarchical, two-phase learning process on the Ridge.

A.2 Choice of a Fixed Reference Metric

  • The FIM at the converged solution, G(α*), serves as a fixed global reference metric instead of the evolving metric G(α_t).This choice linearizes the manifold geometry around the final solution and makes trajectory components globally orthogonal with respect to that geometry.Source: §A.2 · p. 9
  • Information gain is defined as the projection onto the final reference metric.It tracks how much each update step contributes to forming the principal and tail components of the final geometric structure.Source: §A.2 · p. 9
  • Although it does not capture the full nonlinear dynamics, this analysis reveals a hierarchical, two-phase learning process on the Ridge.Source: §A.2 · p. 9

B. Generality of the Oscillatory Dynamics

The supplementary experiments vary storage load and kernel locality to test the generality of oscillatory learning trajectories. They show that severe GD oscillations characterize the high-load Ridge regime, while relaxing locality yields smoother paths.

B. Generality of the Oscillatory Dynamics

Loading 2609.16805v2…