Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View

Rodion Podorozhny

Source-linked AI summary

Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View

Rodion Podorozhny

arXiv:2608.22145v1Aug 23, 2026cs.LGmath.NA

TL;DR

The paper asks when Adam’s diagonal preconditioning can overcome ill-conditioning and when landscape structure instead causes stalls. It develops curvature and gradient-spectrum diagnostics, then applies them to INR benchmarks and FINER, finding that diagonal repair helps axis-aligned cases but not cross-coupled or saddle-dominated ones.

Problem
The paper studies why tuned Adam can reach very low loss on some ill-conditioned landscapes yet plateau above second-order methods on others.
Method
The paper analyzes raw and Adam-preconditioned Hessian conditioning, diagonal mass, negative spectral mass, and gradient-energy fractions, with worked examples and INR case studies.
Results
16×–887× condition reductions occur in axis-aligned cases, whereas cross-coupled benchmarks retain κadam≥5.5·10^5 and FINER places 0.81 spectral mass below −10^-3λmax at initialization.
Takeaways & Limitations
Adam can match second-order accuracy when ill-conditioning aligns with coordinates and negative curvature is benign, but diagonal scaling cannot repair coupled directions or provide a negative-curvature escape mechanism.
Takeaways & Limitations
Negative spectral mass estimates include both quadrature error and threshold error from estimating the spectral scale.

Abstract

from arXiv · show

Across implicit-neural-representation (INR) architectures and analytic benchmarks we observe that a thoroughly tuned Adam (especially its learning rate (lr), e.g. in a hyperparameter sweep from $lr = 0.05$ to $10^{-8}$) can potentially reach a very low loss even on ill-conditioned loss landscape or converge at a plateau far above the loss attained by second-order methods. This report defines the measured metrics that help determine if Adam can mitigate the ill-conditioning on a given loss landscape. We provide the indicators by which each outcome is determined, that are: the condition number of the Hessian and of the Adam-preconditioned Hessian $D^{-1/2}HD^{-1/2}$ (with the derivation from Adam's update rule), the diagonal mass $ρ$ that distinguishes axis-aligned from cross-coupled ill-conditioning, the negative spectral mass estimated by stochastic Lanczos quadrature, and the gradient energy fractions over curvature bands, including the flat fraction that indicates the Adam stall. A worked out $2\times 2$ example and an illustration show the reasons why a diagonal preconditioning by Adam can remove axis-aligned ill-conditioning by rescaling and why it cannot do the same if the ill-conditioning is cross coupled. In addition, we present a case study of FINER image fitting architecture that goes over the whole loss landscape analysis framework: the fitting architecture description, reasons due to which its landscape stalls Adam at saddles, the measured PSNR values through our tuned baselines to the $120$--$134$\,dB results of the blockwise second order methods, the error maps behind those numbers, and description of the benefits such image fitting accuracy gives in practice.

1 Experiment design and notation

The paper defines loss, gradient, Hessian, and gradient-energy quantities, then uses matrix-free Hessian–vector products to analyze large models without forming the full Hessian.

The training loss f has gradient g(θ)=∇f(θ) and Hessian H(θ)=∇²f(θ).
Image-fitting experiments use deterministic full-batch losses, so randomness affects only probe vectors used for matrix-free Hessian computation.
Gradient energy is squared ℓ2 magnitude resolved over an orthonormal system, with Parseval’s identity preserving total energy.
The gradient-energy partition assigns each eigendirection a fraction of ∥g∥² across spectral bands.
For architectures from roughly 200k to 91.4 million parameters, Hessian–vector products replace explicit Hessian construction.

2 Transformation of Hessian by Adam

Adam’s quasi-static diagonal scaling makes its local optimization geometry the symmetric preconditioned Hessian D^-1/2HD^-1/2. Consequently, convergence depends on the preconditioned condition number, while momentum can accelerate but cannot remove that conditioning.

Adam’s coordinatewise update is gradient descent with diagonal preconditioner D^-1 under the quasi-static approximation.
After the coordinate change φ=D^1/2θ, Adam operates on the reparametrized loss whose Hessian is D^-1/2HD^-1/2.
The preconditioned iteration has the same spectrum as a symmetric iteration involving the transformed Hessian, with stability requiring α<2/λmax.
Adam’s first-order contraction rate depends on κ(D^-1/2HD^-1/2), rather than directly on κ(H).
Momentum improves positive-definite quadratic complexity from O(κ) to O(√κ) at best but does not reduce the underlying condition number.
On stalled benchmarks, κadam is 10^5–10^6, leaving an accelerated dependence of approximately 3×10^2 to 10^3.

3 Axis-aligned versus cross-coupled ill-conditioning

Adam’s diagonal preconditioning repairs ill-conditioning when stiff directions align with coordinates, but cannot generally resolve cross-coupled curvature. The diagonal mass and Jacobi-scaled condition number indicate whether per-coordinate rescaling can remove the barrier.

A diagonal preconditioner rescales coordinate axes but cannot rotate them, so Adam’s effectiveness depends on the alignment of ill-conditioning with the coordinate basis.
ρ = 1 iff H is diagonal; small ρ indicates off-diagonal Frobenius mass and cross-coupled stiff and soft eigendirections.Large ρ characterizes axis-aligned ill-conditioning, whereas small ρ characterizes cross-coupled ill-conditioning.
3.1 A worked 2 × 2 example: For the axis-aligned example, scaling by D = diag(H) produces D^-1/2HD^-1/2 = I, reducing the condition number exactly to 1.This mechanism underlies 16×–887× condition-reduction factors on INR architectures and can make Adam perform as well as second-order methods when saddles are absent.
3.1 A worked 2 × 2 example: In the rotated 45° example, both coordinates carry the stiff direction equally, so diagonal scaling leaves the condition number essentially unchanged and the reduction factor is approximately 1.The example has the same spectrum and raw condition number as the axis-aligned case, but ρ tends to 1/2 as L/µ grows and falls much lower under dense high-dimensional coupling.
3.2 A diagonal-scaling certificate: van der Sluis’ theorem: Van der Sluis’ theorem shows Jacobi scaling is within a factor m of the best symmetric positive diagonal scaling, so a large Jacobi-scaled condition number certifies that per-coordinate rescaling cannot fully fix the geometry.A large κadam diagnoses Adam’s particular scaling, whereas the stronger impossibility conclusion requires measuring the Jacobi-scaled condition number.
3.2 A diagonal-scaling certificate: van der Sluis’ theorem: When cross-coordinate curvature remains strongly coupled, progress requires methods that represent it, such as block preconditioning, low-rank curvature models, or Krylov-based cubic-regularized steps.

4 Matrix-free estimation of the spectral features

The paper estimates Hessian spectral features without full eigendecomposition, using Lanczos quadrature and randomized probes, while gradient-started Lanczos estimates curvature-band energy directly. These estimators expose negative curvature, spectral scale, and persistent flat-band gradient energy associated with optimization stalls.

Lanczos estimation: m = 96 Lanczos steps estimate extreme eigenvalues of H or D^-1/2HD^-1/2, even when n is approximately 10^5–10^8.Ritz values converge to spectrum edges first, so moderate iteration counts suffice for λmax and λmin.
Lanczos estimation: Rademacher probes and Lanczos quadrature estimate spectral quantities under the normalized empirical Hessian spectral measure.Probe averaging converts probe-weighted estimates into estimates for the normalized empirical spectral measure.
Negative spectral mass: The negative-curvature estimate includes quadrature error and threshold error because λmax is itself estimated by Lanczos.The negative band is defined relative to the estimated spectral scale, using a cutoff proportional to −ελmax.
Negative spectral mass: Relative thresholds preserve scale invariance and separate the numerically unresolved near-zero dead band from a small number of data-determined spectral outliers.An absolute cutoff would classify the same geometry differently after simply rescaling the loss.
Gradient energy fractions: Lanczos initialized with the normalized gradient estimates gradient-weighted spectral energy fractions without eigendecomposition or probe averaging.These fractions quantify how much gradient energy lies in selected curvature bands.
Gradient energy fractions: A persistently nonzero flat fraction together with substantial negative-curvature mass indicates that gradient energy remains in directions where first-order updates have little effect.Flat-band components barely move, whereas stiff components are suppressed quickly; transient flat fractions can mark spectral-bias stall phases.

5 The measured fingerprints

The measured fingerprints separate Adam’s outcomes into axis-aligned landscapes where diagonal rescaling repairs conditioning, cross-coupled landscapes where conditioning remains a practical stall, and saddle-dominated landscapes where Adam reaches an indefinite stationary point. FINER’s error maps and optimizer baselines further show that Adam’s plateau can preserve a structured low-frequency residual despite visually accurate reconstructions.

Measurement setup: A learning-rate sweep from 0.05 to 10^-8 identifies tuned Adam values, with the reported Adam column using the best PSNR and, here, lr = 10^-4.Table 1 summarizes late-trajectory checkpoints, with initialization or mid-run diagnostics noted separately.
Axis-aligned landscapes: 16×–887× reduction factors show that Adam can remove axis-aligned ill-conditioning by Hessian rescaling, producing outcomes comparable to block second-order methods when negative curvature is benign.The 35k SIREN benchmark reaches the machine-precision floor in this category.
Cross-coupled landscapes: κadam ≥5.5 · 10^5 with reduction ≤1.3× and ρ ≤.04 marks cross-coupled ill-conditioning that diagonal preconditioning cannot repair.The resulting contraction requires about 10^5–10^6 iterations per error along soft coupled directions, producing a practical first-order stall.
Saddle-dominated landscapes: 0.81 of FINER’s spectral mass lies below −10^-3λmax at initialization, and tuned Adam reaches a stationary saddle with a vanishing update while the Hessian remains indefinite.A cubic-regularized block step escapes deterministically and reaches 120–129 dB, whereas Adam reaches 65 dB with low-frequency residual concentration.
Additional baselines: 7.3 dB is the L-BFGS plateau across memory sizes, while SOAP reaches a 77.65 dB best and remains close to tuned Adam’s 78.2 dB ceiling.These baselines support distinct negative-curvature behavior on the deterministic full-batch saddle landscape.
FINER error maps: 133.5 dB for ARC-φ1 versus 78.2 dB for extended-budget Adam accompanies residuals of approximately 2 · 10^-7 versus 10^-4, despite visually indistinguishable reconstructions.The shared error-map scale spans 10^-1 to 10^-8, revealing a nearly three-order-of-magnitude and structural difference in residuals.