Source-linked AI summary
How Far is Adam from Natural Gradient Descent?
Vihaan Paka-Hegde
TL;DR
The geometric relationship between Adam and natural gradient descent remains contested, and the effects of empirical Fisher errors in Adam’s diagonal format are not isolated. The paper evaluates Adam’s full update rule, including momentum, across four loss landscapes using γ(∆θ). Geometric drift rises with landscape complexity, yet Adam still reaches low final loss despite high tracking error.
Problem
Whether Adam’s diagonal empirical Fisher approximations track natural gradient descent remains contested, with empirical Fisher distortions and their effects within Adam’s format not yet isolated.
Source: §1 · p. 1Method
Results
Geometric drift rises from ≈0.2 in well-conditioned linear setups to ≈103 in non-convex neural networks, while Adam reaches low final loss despite high tracking error.
Source: §6.2 · p. 6, §9 · p. 8Takeaways & Limitations
The results suggest Adam’s optimization power may be enabled by layered approximations and momentum despite imperfect geometric alignment.
Source: §9 · p. 9The directional γ(∆θ) metric cannot distinguish stable steps from divergent ones because it does not capture step-size stability or magnitude explosions.
Source: §7.4 · p. 8
Abstract
from arXiv · showhide
Adam is the standard optimizer in deep learning, yet its geometric relationship to natural gradient descent (NGD) contains unresolved questions. We study Adam's full update rule, including momentum, as a diagonal empirical Fisher approximation subject to diagonal truncation, empirical label substitution, and temporal lag. Using the scale-invariant $\gamma(\Delta\theta)$ metric, we measure Adam's geometric deviation from true NGD across four loss landscapes: well-conditioned linear regression, ill-conditioned linear regression, logistic regression, and a non-convex small neural network. Adam's geometric trajectory is context-dependent. Deviation remains low in well-conditioned settings but rises significantly under ill-conditioning, reaching misalignments of $\approx 10^3$ in the neural network. Higher geometric drift correlates with slower initial optimization but does not degrade final objective minimization; Adam consistently reaches low loss. Furthermore, the improved empirical Fisher (iEF) tracks more stable paths than the standard empirical Fisher (EF), which frequently oscillates or diverges. Our results suggest Adam's practical optimization power may stem from a balance of structural approximation errors and momentum smoothing rather than close tracking of the natural gradient path.
1 Introduction
Adam’s relationship to natural gradient descent (NGD) remains contested because empirical Fisher approximations can distort gradient directions. This study examines Adam’s complete update rule, including momentum, across varied loss landscapes and tests whether improved empirical Fisher corrections preserve stability.
1 Introduction
- Empirical Fisher approximations can distort the true gradient field far from the optimum, and inversely scaled projections are a critical failure mode.These concerns motivate examining whether iEF corrections preserve stability in Adam’s diagonal format [4].Source: §1 · p. 1
- The study applies Wu et al.’s geometric framework [4] to Adam’s complete update rule, including active momentum, across varied loss landscapes.Earlier analysis of Adam as a natural-gradient variant omitted momentum by setting β1 = 0 [5].Source: §1 · p. 1
- The optimization effects of empirical Fisher errors within Adam’s diagonal format remain unisolated.The study measures where diagonal approximations diverge from true NGD and tests iEF with active momentum.Source: §1 · p. 1
2 Background
The background connects NGD’s Fisher-based geometry to practical approximations, then frames Adam as a diagonal empirical Fisher optimizer and γ(∆θ) as a directional tracking measure.
2.1 Natural Gradient Descent
- NGD premultiplies the Euclidean gradient by the inverse exact Fisher matrix, modeling parameters as a Riemannian manifold based on output-distribution geometry.Exact NGD requires O(D^2) storage and O(D^3) inversion per step.Source: §2.1 · pp. 1–2
2.2 The Empirical Fisher Approximation
- The empirical Fisher substitutes observed data labels for model-generated predictions [3], while diagonal truncation discards off-diagonal terms and decouples interacting directions.The diagonal representation reduces computational cost to O(D).Source: §2.2 · p. 2
2.3 Adam as a Diagonal Empirical Fisher Optimizer
- Adam’s first- and second-moment moving averages yield a coordinate-wise denominator that mirrors the square root of a diagonal empirical Fisher matrix [1, 5].Adam combines diagonal truncation, empirical substitution, and temporal lag; earlier FAdam analysis assumed zero momentum (β1 = 0) [5].Source: §2.3 · p. 2
2.4 The γ(∆θ) Geometric Indicator
- The scale-invariant γ(∆θ) indicator measures directional alignment with exact NGD: zero denotes perfect alignment, while larger values indicate divergence [4].It isolates directional fidelity rather than step-size magnitude [4].Source: §2.4 · p. 2
3 Related Work
Prior work identifies distortions in empirical Fisher approximations and proposes remedies, but leaves open how these issues interact in Adam with active momentum.
3 Related Work
- Observed-label empirical Fisher approximations can misrepresent curvature and mislead updates when models are far from convergence or structurally misspecified [3].Wu et al. identified over-weighting of well-fit samples with near-zero gradients as an inversely scaled projection issue [4].Source: §3 · p. 3
- Wu et al. proposed iEF gradient rescaling to restore stability and validated it with γ(∆θ) on full-matrix variants [4].Hwang’s FAdam formalized Adam’s natural-gradient connection but assumed momentum was deactivated [5].Source: §3 · p. 3
- This paper applies Wu et al.’s geometric framework [4] to Adam’s complete update rule to evaluate approximation-error interactions under active momentum.Source: §3 · p. 3
4 Method
The method compares Adam’s parameter updates with the true natural-gradient path using γ(∆θ), across four landscapes chosen to isolate curvature, label-substitution noise, and non-convexity.
4 Method
- The evaluation computes γ(∆θ) to track directional alignment between Adam updates and the true natural-gradient path, comparing explicit optimizer updates side-by-side.Source: §4 · p. 3
- The four test landscapes are well-conditioned linear regression, ill-conditioned linear regression, logistic classification, and a non-convex multilayer neural network.They are designed to isolate curvature asymmetry, label-substitution noise, and non-convexity.Source: §4 · p. 3
- The experiments track geometric deviation and training loss to test whether landscape complexity accelerates drift and whether iEF adjustments maintain alignment with active momentum.Source: §4 · p. 3
5 Experiments
Across four progressively varied loss landscapes, the experiments track geometric deviation and training loss. Adam’s drift rises with landscape complexity yet does not prevent it from reaching identical or competitive loss to exact NGD.
5 Experiments
- Across four landscapes, Adam’s geometric drift rises with complexity without compromising final loss minimization; iEF tracks more stable paths than EF.Learning rates were chosen for baseline stability rather than absolute convergence speed.Source: §5 · p. 3
5.1 Learning Rate Summary
- NGD tolerates larger learning rates, while SGD becomes unstable under severe ill-conditioning.Table 1 reports learning rates optimized for baseline stability within each setting.Source: §5.1 · p. 4, Table 1 · p. 4
5.2 Experiment 1; Well-Conditioned Linear Regression
- In well-conditioned linear regression, Adam, SGD, and iEF show rising drift over 2,000 steps but reach lower loss floors than EF.The isotropic baseline uses 10 independent features drawn from N(0, 1).Source: Figure 1 · p. 4, §5.2 · p. 4
5.3 Experiment 2; Ill-Conditioned Linear Regression
- Under severe ill-conditioning (κ ≈ 27,146), Adam’s geometric error rises steadily, crosses SGD near step 1,250, and remains above iEF’s lowest non-baseline drift.The setting introduces a 10,000× variance asymmetry between parameter groups.Source: §5.3 · p. 4, Figure 2 · p. 5
5.4 Experiment 3; Logistic Regression
- In logistic regression, label substitution distorts curvature; Adam starts with greater geometric misalignment than SGD or iEF, while EF oscillates continuously.The experiment is binary classification with N = 200 and D = 10 using cross-entropy.Source: Figure 3 · p. 5, §5.4 · p. 5
5.5 Experiment 4; Small Neural Network
- In the non-convex neural network, Adam’s drift peaks near γ ≈ 10^3 at step 300, yet it secures the lowest final training-loss floor.The 692-parameter, two-layer ReLU network subjects optimizers to all approximation errors simultaneously.Source: §5.5 · p. 5, Figure 4 · p. 6
6 Results
Across the evaluated landscapes, Adam’s geometric drift increases with landscape complexity, but high directional error does not preclude low final loss. The results also distinguish geometric alignment from optimizer stability and compare EF with iEF.
6 Results
- Adam’s geometric drift increases alongside landscape complexity across the four evaluated environments.Table 2 reports quantitative tracking metrics and convergence thresholds.Source: §6 · p. 6, Table 2 · p. 6
6.1 Geometric Drift Across Landscapes
- Adam’s initial alignment varies by landscape: γ starts near 0.2 in the isotropic setting, degrades during ill-conditioned training, and starts at 1.5 in logistic regression.The logistic-regression baseline reflects directional distortion from empirical label substitution.Source: §6.1 · p. 6
6.2 Convergence Under High Geometric Error
- Adam’s neural-network directional error reaches ≈ 10^3 by step 300, yet it secures the lowest final training loss; momentum smoothing appears to help it navigate misaligned updates.The geometric indicator diagnoses directional compliance with NGD coordinates, not overall convergence efficiency.Source: §6.2 · p. 6
6.3 Ill-Conditioned Crossover
- In ill-conditioned regression, Adam’s γ surpasses standard SGD near step 1,250 despite faster convergence, as diagonal truncation can over-attenuate updates along minimized, high-variance directions.The crossover is absent in the isotropic experiment.Source: §6.3 · p. 7
6.4 EF vs iEF Stability
- EF can show lower absolute γ than SGD just before divergence, while iEF is more stable in Experiments 1, 3, and 4 but slower than Adam.iEF is unstable under extreme ill-conditioning, where its scaling adjustments may need momentum smoothing.Source: §6.4 · p. 7
7 Discussion
Adam’s approximation errors compound, yet momentum smoothing and stable convergence can coexist with substantial geometric drift. The directional metric γ is informative but does not capture overall optimizer health or step-size stability.
7.1 How Adam’s Approximations Interact
- Diagonal truncation removes parameter interactions, and empirical label substitution compounds the distortion by introducing noise into the curvature estimate.The exponential moving average adds temporal lag that may stabilize noisy updates by balancing directional errors from truncation and empirical noise.Source: §7.1 · p. 7
7.2 Adam and Sign Gradient Descent
- In the neural network, Adam has geometric misalignment γ ≈103 yet minimizes the objective faster than the alternatives, consistent with sign-gradient-like behavior.As β2 approaches unity, the denominator becomes a stable coordinate normalizer rather than tracking instantaneous curvature, while the first-moment average smooths direction [7].Source: §7.2 · p. 7
7.3 Trade-Offs of Exact NGD
- Exact NGD uses analytical or model-derived curvature to reach low loss in few steps, while its full-matrix storage and inversion scale poorly for large networks.Adam’s O(D) diagonal layout sacrifices fine-grained geometric alignment for scalability.Source: §7.3 · p. 8
7.4 Limitations of γ
- γ measures directional alignment, not step-size stability: the empirical Fisher can show lower γ than SGD immediately before optimization diverges.The metric therefore cannot distinguish stable steps from divergent ones.Source: §7.4 · p. 8
7.5 Practical Implications
- In three of four evaluations, Adam reaches NGD’s exact optimization floor with more steps; in logistic regression, it misses the baseline floor within 2000 steps.The authors identify increased step count, rather than degraded final loss, as the primary penalty of Adam’s geometric approximation.Source: §7.5 · p. 8
8 Future Work
The authors propose integrating iEF stability corrections into Adam’s momentum loop and extending geometric evaluations to larger architectures and datasets.
8 Future Work
- Future work could integrate iEF corrections into Adam’s momentum loop as IAdam and evaluate whether the observed tracking patterns hold on overparameterized models and large image datasets.The authors suggest CIFAR-10 as one such dataset; Wu et al. had deferred momentum integration to future studies [4].Source: §8 · p. 8
9 Conclusion
The study applies γ(∆θ) to Adam’s complete update rule across four loss landscapes. It finds that geometric deviation varies by context, while stable convergence can persist despite high misalignment.
9 Conclusion
- γ(∆θ) rises from ≈0.2 in well-conditioned linear setups to ≈103 in non-convex neural networks as approximation errors overlap.The evaluations quantify the geometric costs of active momentum under diagonal empirical Fisher approximations.Source: §9 · p. 8
- Adam regularly converges stably despite high geometric tracking error, and its momentum provides numerical robustness independent of strict geometric alignment.The results are also consistent with Adam behaving similarly to sign-gradient methods.Source: §9 · pp. 8–9
- Directional alignment is informative but incomplete: the empirical Fisher can align better than SGD while still inducing optimization divergence.The findings distinguish directional fidelity from step-size stability and suggest Adam trades strict geometry for scalability.Source: §9 · p. 9