PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients

Lukas Balles, Philipp Hennig

arXiv:1705.07774v4cs.LGstat.ML

TL;DR

ADAM’s behavior is difficult to explain uniformly because it combines sign-based directions with variance-adapted magnitudes. The paper disentangles these components, analyzes them separately, and finds that the sign aspect is dominant but problem-dependent, while variance adaptation yields M-SVAG as an alternative when sign-based methods fail.

  • Problem

    The paper addresses why ADAM performs inconsistently by separating its sign-based direction from its variance-adapted update magnitude.

  • Method

    The authors analyze sign and variance adaptation in isolation and transfer variance adaptation to momentum SGD through M-SVAG.

  • Results

    The sign aspect is by far the dominant component, its usefulness depends on the problem, and sign-based methods appear to harm generalization.

  • Takeaways & Limitations

    M-SVAG extends variance adaptation to momentum SGD and provides an alternative for problems where sign-based methods like ADAM fail.

  • Takeaways & Limitations

    Variance estimates are more realistic when noise is high and step sizes are small, because successive-gradient changes are then dominated by stochasticity.

Abstract

from arXiv · show

The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an estimate of their relative variance. We disentangle these two aspects and analyze them in isolation, gaining insight into the mechanisms underlying ADAM. This analysis also extends recent results on adverse effects of ADAM on generalization, isolating the sign aspect as the problematic one. Transferring the variance adaptation to SGD gives rise to a novel method, completing the practitioner's toolbox for problems where ADAM fails.

1. Introduction

The paper reinterprets ADAM as the combination of a sign-based update direction and variance-adapted coordinate magnitudes, then disentangles these aspects through alternative methods. This perspective motivates analyzing their distinct roles and extending variance adaptation beyond sign-based updates.

  • ADAM’s two aspects: ADAM separates into sign-based update directions and variance-adapted coordinate magnitudes.The sign of each stochastic-gradient coordinate determines direction, while relative variance determines its update factor.
  • Variance adaptation: Variance adaptation shortens updates in coordinates with high relative stochastic-gradient variance.The paper motivates this as adapting to differing reliability across coordinates.
  • Contrasting interpretations: The paper challenges interpreting ADAM as natural-gradient descent because its square-root denominator does not directly match the empirical Fisher approximation.It also distinguishes ADAM from approximate whitening, which would depend on standard-deviation scaling rather than the non-central second moment.
  • Alternative methods: The paper isolates the two aspects by comparing ADAM with SGD momentum, SSD, and SVAG.SSD retains signs without variance adaptation, whereas SVAG applies variance adaptation directly to stochastic gradients.
  • Scope: The introduction frames variance adaptation as applicable beyond ADAM, including momentum-based methods and variance-adapted extensions of SGD.The paper positions this decomposition as a basis for analyzing and recombining the two components.
  • Sign aspect: Sign-based methods depend on stochastic-gradient sign reliability, which is determined by relative variance under the paper’s normality assumption.The authors analyze this dependence to understand when signs can serve as useful update directions.

2. Why the Sign?

The paper compares stochastic-gradient and sign directions on stochastic quadratic problems to determine when signs are useful. The analysis links their relative performance to noise, conditioning, and eigenbasis alignment.

  • Comparison criterion: The comparison uses local expected function-value decrease under each direction’s optimal step size.The analyzed directions are SGD with z = −g(θ) and SSD with z = −sign(g(θ)).
  • Theoretical observations: For a diagonal Hessian, pdiag(Q) = 1, whereas random orientations give pdiag(Q) ≈ 1.57/d on average.The quantity can be as low as 1/d, indicating why arbitrary rotations can impair sign updates.
  • Theoretical observations: Sign-based performance has milder eigenvalue dependence because noise affects it through element-wise success probabilities rather than a noise–eigenspectrum interaction.The stochastic-gradient direction exhibits the stronger interaction in the theoretical comparison.
  • Theoretical observations: Sign updates can be advantageous on noisy, ill-conditioned problems whose Hessians are close to axis-aligned.The sign direction is intrinsically coordinate-aligned, making diagonal dominance especially relevant.
  • Experimental evaluation: On well-conditioned problems, gradient descent dominates without noise, but added noise narrows the gap between SGD and SSD.Eigenbasis orientation has little effect in this well-conditioned setting.

3. Variance Adaptation

Variance adaptation chooses element-wise scaling factors to make noisy update directions closer to their desired gradients. For Gaussian stochastic gradients, the resulting factors explain ADAM’s scaling and motivate SVAG, which applies variance adaptation without taking signs.

  • General principle: Element-wise factors γ scale an unbiased estimate ˆp to minimize its expected distance from the desired update direction p.The same framework applies either to the estimate itself or to its element-wise sign.
  • Sign adaptation: γ_i = 2ρ_i − 1, where ρ_i is the probability that a stochastic-gradient sign matches the true-gradient sign.The factor ranges from 0 with no sign information to 1 when the sign is certain.
  • ADAM: Under Gaussian noise, the sign-optimal factor is closely approximated by (1 + η_i^2)^−1/2, the factor implicitly used by ADAM.The exact expression involves the error function, while Figure 3 compares the two factors as relative standard deviation varies.
  • ADAM: ADAM approximates this variance adaptation but applies it to sign(m_t) rather than sign(g_t).The distinction between adapting g_t and m_t is addressed separately in the paper.
  • SVAG: SVAG applies variance adaptation directly to the stochastic gradient, shortening coordinates with higher relative variance.In the illustrated example, θ2 has relative variance η2_2 = 2.25 versus η2_1 = 0.25 and is shortened, reducing update-direction variance at the cost of bias.
  • Convergence: With exact adaptation factors, variance adaptation guarantees convergence without manually decreasing the global step size and recovers SGD’s O(1/t) rate for smooth, strongly convex functions.This theorem assumes a variance bound allowing nonzero variances and quadratic growth in the gradient norm.

4. Practical Implementation of M-SVAG

M-SVAG estimates variance adaptation factors from moving averages and applies them to the momentum-like update direction m_t. Its practical construction depends on an approximate stationarity assumption and uses bias corrections and implementation safeguards.

  • Method: M-SVAG applies variance adaptation to m_t, combining a momentum-like update direction with ADAM-related variance estimation.The method is presented as the practical focus of the variance-adapted methods discussed in this section.
  • Variance estimation: Moving averages provide estimates of stochastic-gradient moments when the gradient distribution is approximately constant over the effective averaging horizon.Under this assumption, m_t and v_t estimate the first and second moments of g_t.
  • Variance estimation: The assumption is more realistic with high noise and small step size, when stochasticity dominates changes in the true gradient between successive observations.The authors explicitly state that the assumption can hold only approximately.
  • Estimator design: M-SVAG uses the same moving-average constant for m_t and v_t to define a common effective range for the assumed gradient distribution.The method also corrects a systematic bias in its variance estimate.
  • Estimator design: Moving-average and within-mini-batch variance estimators produced similar performance, so the main paper uses the moving-average variant for easier implementation and correspondence with ADAM.The mini-batch estimator is reported in the supplementary material.
  • Adaptation factors: The estimated adaptation factor is intuitive but not unbiased for the exact factor, and unbiased estimation is described as intractable.Partial bias corrections were tested but had destabilizing effects.
  • Implementation: Because m_t determines both the update direction and the variance estimate, the first M-SVAG step is defined to coincide with an SGD step.The implementation also skips an update when both m_t,i and v_t,i are zero, rather than adding ADAM’s ε offset.
  • Implementation: Algorithm 1 exposes two hyperparameters, α and β, and updates θ using the element-wise product γ ⊙ m.The pseudocode includes bias-corrected moving averages, variance estimation, and adaptation-factor calculation.

5. Connection to Generalization

The paper separates ADAM’s sign behavior from its element-wise variance adaptation in the context of generalization. Its analysis indicates that sign-based updates account for the problematic construction identified for adaptive methods, while M-SVAG avoids that construction without guaranteeing favorable generalization.

  • Prior results: Prior work argued that ADAGRAD, RMSPROP, and ADAM can generalize worse than gradient descent, SGD, and momentum variants.The cited theoretical construction concerns binary least-squares classification.
  • Prior results: In the constructed problem class, non-adaptive methods converge to the max-margin solution, whereas some adaptive methods converge to solutions with arbitrarily bad unseen-data generalization.The adaptive methods considered are ADAGRAD, ADAM, and RMSPROP.
  • Sign aspect: Sign descent satisfies the same iterate proportionality θ_t ∝ sign(X^T y) established for the adaptive methods in the construction.This extends the relevant statement from adaptive methods to sign-based updates.
  • Variance adaptation: M-SVAG does not generally produce iterates proportional to sign(X^T y), so the Wilson et al. construction does not apply to it.The paper cautions that this does not imply convergence to the max-margin solution or otherwise favorable generalization.
  • Conclusion: The authors therefore identify the sign aspect, rather than element-wise adaptivity alone, as the factor impeding generalization in those examples.Their experiments are stated to support this interpretation and motivate disentangling the two aspects.

6. Experiments

Experiments disentangle ADAM’s sign and variance-adaptation components across four test problems. The sign aspect generally dominates performance differences, while variance adaptation is consistently non-harmful and sometimes beneficial.

  • Experimental design: The four methods recombine sign-based updates and variance adaptation, enabling their effects to be compared separately.The comparison uses M-SVAG, ADAM, M-SGD, and M-SSD.
  • Main observations: The sign aspect dominates performance, except on the language-modelling problem P4, where methods cluster by whether they use signs.ADAM and M-SSD show similar performance, while M-SGD and M-SVAG form the non-sign-based group.
  • Main observations: Sign-based methods outperform non-sign-based methods on training loss for P1 and P3, but plateau or underperform on P2 and P4.On P2, ADAM and M-SSD initially progress rapidly before being undercut by M-SGD and M-SVAG.
  • Main observations: Variance-adapted variants perform at least as well as their base algorithms in all experiments and often perform better.ADAM and M-SSD are identical on P3, whereas M-SVAG significantly outperforms M-SGD on P3 and P4.
  • Main observations: On CIFAR-100, ADAM achieves much lower training loss but significantly worse test performance than M-SGD, while M-SVAG improves upon M-SGD without adverse generalization effects.M-SSD behaves almost identically to ADAM in both training and test performance.

7. Conclusion

The conclusion interprets ADAM as combining sign updates with variance adaptation and reports that the sign component is dominant but problem-dependent. It presents variance adaptation applied to momentum SGD as a practical alternative when sign-based methods fail.

  • Conclusion: ADAM combines two components: taking stochastic-gradient signs and adapting update magnitudes to relative variance.The experiments identify the sign component as dominant, while its usefulness depends on the problem.
  • Practical implication: M-SVAG applies variance adaptation to momentum SGD and is proposed as an addition to the practitioner’s toolbox for problems where ADAM fails.Variance adaptation is not restricted to ADAM and can be applied to arbitrary update directions.

A.1. Network Architectures

The experiments use four neural-network settings and tune optimizer step sizes for test accuracy, with repeated runs over randomized training conditions. The appendix also formalizes sign success probabilities for Gaussian gradients.

  • Experimental protocol: Step sizes are tuned by finding the maximal stable value, searching downward, and selecting the value with maximal test accuracy within a fixed training budget.The selected configuration is replicated ten times with different random seeds and randomized training conditions.
  • Experimental protocol: The appendix lists optimizer-specific step-size grids for each problem, including M-SGD, ADAM, M-SSD, and M-SVAG.The grids span multiple orders of magnitude and include problem-dependent ranges.

B.2. Analysis on Stochastic QPs

The stochastic-quadratic analysis derives how sign updates and variance adaptation behave under noise, curvature, and momentum. It also specifies implementation choices for variance-adapted methods and the ADAM* variant.

  • Stochastic quadratic analysis: For SGD, expected quadratic progress includes both the squared mean-gradient term and a trace term involving gradient covariance.For stochastic sign descent, the expected update depends on the probability that each coordinate’s sign matches its mean gradient.
  • Stochastic quadratic analysis: Sign directions can be beneficial for noisy, ill-conditioned problems with diagonally dominant Hessians, although the extent to which this applies to real problems remains unclear.The appendix connects this intuition to axis alignment through eigenvector geometry.
  • Convergence analysis: The convergence proof analyzes the optimizer as a discrete stochastic process using conditional expectations given the history through time t.The argument uses smoothness, strong convexity, Jensen’s inequality, and a recursive error bound.
  • Implementation: The implementation estimates first and second moments with moving averages, corrects the variance estimate, and scales updates using variance-adaptation factors.The listed implementation uses γ = m2/(m2 + s) and applies γ element-wise to the gradient direction.
  • Variance-adapted methods: M-SVAG applies variance adaptation to the momentum direction mt, while SVAG applies corresponding factors directly to the stochastic gradient gt.The two methods differ in whether momentum is part of the update direction.
  • Implementation: ADAM* applies corresponding variance-estimation adjustments to the sign case while retaining factors (1 + η)^−1/2 for correspondence with original ADAM.The authors report that this choice performed almost identically to the Gaussian-optimal alternative.

C.3. Experiments

Experiments compare SVAG and ADAM* and examine mini-batch variance estimation, finding method-specific differences while supporting variance adaptation as a useful component.

  • SVAG performs better than M-SVAG on (P2), but on (P3) it progresses faster initially before plateauing and producing slightly worse training loss and test accuracy.The authors describe SVAG as a viable alternative.
  • ADAM and ADAM* are on par on CIFAR-100 (P3), while ADAM* catches up after ADAM’s marginally faster start on (P2) and reaches lower minimal training loss values.The authors conclude that the adjusted variance adaptation factors have a positive effect.
  • Mini-batch variance estimation is an alternative to moving averages; the authors report similar performance and use moving averages in the main paper for easier implementation and correspondence with ADAM.The supplementary experiments are intended to show that variance adaptation remains useful regardless of the estimator.
  • Mini-batch variance estimates could also be used for the alternative methods, but the paper does not explore this extension.

D.2. Experiments

The supplementary experiment compares mini-batch and moving-average variance estimates for M-SVAG on (P1) and (P2), finding almost identical performance.

  • The mini-batch M-SVAG variant is tested on (P1) and (P2) against the moving-average version.
  • The mini-batch and moving-average M-SVAG algorithms have almost identical performance.
  • Algorithm 5 implements M-SVAG with a mini-batch variance estimate and updates parameters using variance-adapted momentum.
  • Figure 8 presents the experimental results for the mini-batch M-SVAG variant, marked “mb” in the legend.Its plot is set up like Fig. 5.
Loading 1705.07774v4…