PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Understanding Why Adam Converges Faster Than SGD for Transformers

Yan Pan, Yuanzhi Li

arXiv:2306.00204v1cs.LGcs.AI

TL;DR

Why Adam converges faster than SGD on tasks such as transformer training remains theoretically unclear. This paper studies directional sharpness and proposes coordinate-wise clipping; it reports that clipping reduces sharpness and can improve local loss reduction and convergence. The authors suggest that adaptive coordinate-wise scaling helps explain Adam’s practical success.

  • Problem

    Why Adam often outperforms SGD on tasks such as transformer training remains unclear despite adaptive algorithms’ empirical advantages and limited theoretical explanation.

    Source: §1 · p. 1

  • Method

    The paper studies update-direction directional sharpness and proposes coordinate-wise clipping to prevent a few large-gradient, poorly smooth coordinates from dominating updates.

    Source: §1 · p. 2

  • Results

    The authors report that clipping can consistently reduce directional sharpness, often improving local function reduction and convergence speed across optimization algorithms.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that adaptive coordinate-wise scaling can balance gradient correlation and directional sharpness, helping explain Adam’s fast convergence in deep learning.

    Source: §1 · p. 2

    The authors note the observed property may be transformer-related: in ResNets, adaptive algorithms may have worse directional sharpness than SGD, and they conjecture a positive gradient-Hessian coordinate correlation.

    Source: §3.2 · p. 5, §4.2 · p. 7

Abstract

from arXiv · show

While stochastic gradient descent (SGD) is still the most popular optimization algorithm in deep learning, adaptive algorithms such as Adam have established empirical advantages over SGD in some deep learning applications such as training transformers. However, it remains a question that why Adam converges significantly faster than SGD in these scenarios. In this paper, we propose one explanation of why Adam converges faster than SGD using a new concept directional sharpness. We argue that the performance of optimization algorithms is closely related to the directional sharpness of the update steps, and show SGD has much worse directional sharpness compared to adaptive algorithms. We further observe that only a small fraction of the coordinates causes the bad sharpness and slow convergence of SGD, and propose to use coordinate-wise clipping as a solution to SGD and other optimization algorithms. We demonstrate the effect of coordinate-wise clipping on sharpness reduction and speeding up the convergence of optimization algorithms under various settings. We show that coordinate-wise clipping improves the local loss reduction when only a small fraction of the coordinates has bad sharpness. We conclude that the sharpness reduction effect of adaptive coordinate-wise scaling is the reason for Adam's success in practice and suggest the use of coordinate-wise clipping as a universal technique to speed up deep learning optimization.

1 Introduction

The paper explains Adam’s practical advantage through directional sharpness, arguing that SGD’s update directions are sharper and that a small number of coordinates drive this problem. It proposes coordinate-wise clipping to reduce sharpness and improve local loss reduction and convergence.

1 Introduction

  • The paper frames its question as why Adam can outperform SGD on tasks such as transformer training, where adaptive methods have empirical advantages but limited theoretical explanation.The cited best-known non-convex convergence rate for AMSGrad matches, but does not improve on, SGD’s.Source: §1 · p. 1
  • In a diagonal quadratic example, clipping the single high-curvature coordinate permits a larger learning rate and convergence in a few steps.The example has A_11 = 100 and A_ii = 1 for i > 1.Source: §1 · p. 2
  • Adaptive algorithms have lower directional sharpness than SGD, and low-sharpness directions can yield better local loss reduction when step sizes are chosen optimally.Directional sharpness measures curvature along an update direction; gradient correlation measures its alignment with the negative gradient.Source: §1 · p. 2
  • Coordinate-wise clipping can reduce directional sharpness and improve local loss reduction and convergence speed across optimization algorithms, with theoretical and empirical support.The authors report robustness across different tasks, models, and iterations.Source: §1 · p. 2
  • A small fraction of coordinates accounts for most of SGD’s directional sharpness, which the authors relate to imbalanced gradients and positive gradient-Hessian coordinate correlation.Clipping is intended to prevent coordinates with large gradients and poor smoothness from dominating the update direction.Source: §1 · p. 2

2 Related Work

Prior work establishes adaptive-optimizer convergence guarantees and studies settings where adaptive methods or clipping may help, but general theory does not explain Adam’s empirical speed. This paper distinguishes its explanation and clipping strategy from earlier coordinate-wise clipping work.

General Convergence Rates of Adaptive Algorithms

  • The best known non-convex convergence rate for AMSGrad matches SGD’s, leaving no general theoretical proof that Adam converges asymptotically faster.Prior work also studies convex and non-convex rates for Adagrad, Adam, and AMSGrad.Source: §2 · pp. 2–3

Faster Convergence Rates Under Certain Settings

  • Adaptive algorithms can have better asymptotic behavior with sparse stochastic gradients or sparse noise, while global clipping has been shown to outperform SGD under heavy-tailed noise.The cited work also argues Adam handles heavy-tailed noise and designs an algorithm using coordinate-wise clipping.Source: §2 · p. 3

Coordinate-Wise Clipping

  • Earlier coordinate-wise clipping work proposed CClip and ACClip for transformers, reporting that ACClip can outperform Adam on various transformers.Those methods use adaptive thresholds updated as momentum parameters.Source: §2 · p. 3
  • Prior coordinate-wise clipping work in differentially private coordinate descent selected thresholds from mean gradients, whereas this paper selects them from the gradient distribution.The stated distinction is that the latter threshold clips exactly a constant fraction of gradients.Source: §2 · p. 3
  • Unlike, this paper uses gradient-distribution-based thresholds and presents clipping as a meta-algorithm applicable to every optimizer, with directional sharpness as its explanation.The authors contrast this with ’s focus on heavy-tailed noise and designing an algorithm to outperform Adam.Source: §2 · p. 3

3 Directional Sharpness of Optimization Algorithms

The section introduces directional sharpness as a measure of curvature along an optimizer’s update direction and studies how it relates to local loss reduction. It reports lower directional sharpness for adaptive methods than SGD in transformer training, while noting the pattern may depend on architecture.

3.1 From Quadratic Taylor Expansion to Directional Sharpness

  • The local loss expansion separates update effects into gradient correlation and directional sharpness, with the former linear and the latter quadratic in step size.A global smoothness bound can be loose because the Hessian and update direction vary across algorithms.Source: §3.1 · pp. 3–4
  • Directional sharpness is v⊤∇2f(x)v for a unit direction v, measuring curvature along that direction rather than worst-case curvature.Unlike sharpness, which takes the supremum over directions, it evaluates the specific update direction.Source: §3.1 · p. 4

3.2 Directional Sharpness and Update Directions

  • Adaptive algorithms show much lower directional sharpness than SGD, which can permit larger steps and lower local loss than SGD.The study compares update directions under the same trajectory and local geometry using pseudo-update steps.Source: §3.2 · pp. 4–5
  • On machine translation, SGD’s directional sharpness can be more than 107 times Adafactor with clipping; clipping almost always improves sharpness.The comparison is reported for the same experiment and iteration as Figure 2.Source: §3.2 · p. 4, Table 1 · p. 5
  • SGD’s gradient magnitudes are imbalanced across coordinates, and a small fraction of large coordinates contributes substantially to directional sharpness.The reported histogram has half of SGD momentum coordinates between 10^-12 and 10^-6, while some reach 10^-4 to 10^-2.Source: §3.2 · p. 5
  • The adaptive-method sharpness pattern may be architecture-dependent: in a simple ResNet image-classification experiment, adaptive methods might have worse sharpness than SGD.The authors note this is consistent with adaptive methods often being slower than SGD on vision tasks.Source: §3.2 · p. 5

4 Coordinate-wise Clipping

Coordinate-wise clipping is proposed to reduce directional sharpness by limiting the influence of large gradient coordinates, with theoretical and empirical support for better local loss reduction and faster convergence. The analysis connects clipping’s benefits to imbalanced gradients and coordinate-wise smoothness.

4.1 Coordinate-wise Clipping Improves Directional Sharpness

  • The method clips the top k% of gradient coordinates by absolute value and uses the clipped gradient to update momentum.For adaptive algorithms, only numerator momentum uses the clipped gradient; denominator momentum uses the original gradient to preserve normalization.Source: §4.1 · p. 6
  • Clipping can improve local loss reduction by reducing directional sharpness, even when it decreases gradient correlation.The authors note that clipping’s effect is a trade-off between improving gradient correlation and reducing sharpness; the Figure 2 landscape uses step size normalized by update-step L2 norm.Source: §4.1 · pp. 5–7, Figure 2 · p. 6
  • Clipped optimization algorithms generally converge faster than their original counterparts in transformer training.The loss-curve comparison is presented as consistent with the local landscape analysis.Source: §4.1 · p. 7, Figure 6 · p. 8

4.2 Connection with Coordinate-wise Smoothness

  • When gradients concentrate on a small fraction of coordinates with poor smoothness, clipping redistributes their influence and can reduce dependence on the worst smoothness.The analysis bounds that dependence by O(√εL) when an ε-fraction of coordinates have smoothness L.Source: §4.2 · p. 8
  • Under the stated smoothness assumption, clipping can yield a directional-sharpness upper bound of O(√εL + ℓ), much smaller than SGD’s L bound.The assumption is that removing an ε-fraction of coordinates leaves a Hessian with spectral norm at most ℓ; the paper reports that removing at most 4% improved estimated smoothness by a constant factor.Source: §4.2 · p. 8

5 Experiment Setups

The experiments compare momentum-based optimization algorithms across transformer tasks, measuring directional sharpness, loss landscapes, and training convergence. They report that SGD has poor directional sharpness across settings, while clipping improves sharpness and is associated with faster convergence.

  • The experiments compare SGD momentum with Adam, Adafactor, Lion, signSGD, and normalized SGD, using momentum in all algorithms.Normalized SGD is included to contrast coordinate-wise scaling with weight-matrix-wise normalization.Source: §5 · pp. 8–9
  • The tasks are machine translation with T5 on OPUS Books English-French and autoregressive Python code generation with GPT-Neo on The Stack.The authors describe code generation as differing from natural-language tasks and use it to test whether observations extend across settings.Source: §5 · p. 9
  • Across the evaluated settings, SGD has poor directional sharpness, while clipping improves sharpness and often yields better local loss reduction.The comparisons cover different algorithms and local geometries.Source: §5 · p. 9
  • Clipping algorithms converge faster in practice, and better local directional sharpness and loss reduction generally track faster global convergence.The paper proposes local geometry as a good indicator of global convergence speed.Source: §5 · p. 9

6 Conclusion

The paper presents directional sharpness as an indicator of optimizer performance and proposes coordinate-wise clipping to address high sharpness. Its findings connect adaptive scaling and clipping with improved local loss reduction and faster convergence.

  • Adaptive and clipped algorithms generally achieve better directional sharpness than SGD, which the paper relates to positive coordinate-wise gradient-Hessian correlation.The authors present this correlation as an explanation for SGD’s high sharpness and slow convergence.Source: §6 · p. 9
  • Coordinate-wise clipping improves sharpness and can enable lower loss in the update direction across algorithms including SGD, normalized SGD, and Adafactor.The conclusion proposes clipping as a broadly applicable technique for speeding up deep-learning optimization.Source: §6 · p. 9
  • The authors frame their account of Adam’s empirical advantage as an insight and conjecture that further understanding could aid theoretical analysis.Source: §6 · p. 9

A Convergence of Clipping with Coordinate-wise Smoothness

Coordinate-wise clipping can reduce directional sharpness when only a small fraction of Hessian coordinates are sharp, yielding better loss decrement than ordinary SGD under stated assumptions. The result depends on gradient and Hessian conditions whose transformer-specific justification is left for future analysis.

A Convergence of Clipping with Coordinate-wise Smoothness

  • The authors leave removing the gradient assumptions to future analysis of transformer architecture, since their Appendix E results make the relevant properties transformer-specific.Assuming only a clipping fraction is insufficient because an adversarially imbalanced gradient can make the remaining clipped direction vanish.Source: A Convergence of Clipping with… · p. 13
  • The theorem assumes a non-convex L-smooth objective and a (1−ε)-fraction Hessian submatrix with spectral norm at most ℓ ≪ L.It also requires the clipped gradient norm to remain comparable to the full gradient and to the clipping threshold.Source: A Convergence of Clipping with… · p. 13
  • Clipping the top δ-fraction of coordinates, with δ > ε, gives directional sharpness O(√εL + ℓ) ≪ L and an asymptotically better loss-decrement bound than SGD.The result assumes a (1−ε)-sized Hessian submatrix has spectral norm at most ℓ, alongside conditions on the clipped and remaining gradient norms.Source: A Convergence of Clipping with… · p. 13

B.1 Tasks, Datasets, and Models

The experiments use machine translation and autoregressive modeling, with task-specific datasets, models, and training setups.

B.1 Tasks, Datasets, and Models

  • Machine translation uses the English-to-French OPUS Books dataset and T5, with 10,240 randomly selected training examples, batch size 1,024, and 10 batches per epoch.The study minimizes training cross-entropy without test or validation sets or other evaluation metrics.Source: §B.1 · p. 15, Table 2 · p. 15
  • The autoregressive task uses GPT-Neo pretrained on Code Clippy and the “the-stack-smol” version of the Stack dataset.Fixed masks with probability 0.15 are generated at training start and reused during collation.Source: §B.1 · p. 15, Table 2 · p. 15

B.2 Optimization Algorithms and Clipping Methods

The appendix specifies the optimization algorithms and clipping variants, then describes experiments for directional sharpness and convergence. Its measurement procedure uses pseudo-update steps and fixed test batches to compare update directions and landscapes.

B.2 Optimization Algorithms and Clipping Methods

  • The study includes Adam, SGD, signSGD, normalized SGD, Adafactor, and Lion, with momentum used across algorithms and coordinate-wise clipping applied to gradients.For Adam and Adafactor, clipping is applied only to the gradient in the final update numerator; clipping update steps for Adam and Lion does not significantly improve results.Source: §B.2 · p. 15

B.3 Experiment for Directional Sharpness of Optimization Algorithms

  • Directional-sharpness experiments use pseudo-update steps to compute momentum on a shared trajectory, while landscape losses are measured on a fixed random test batch of 1,024 examples.The experiments use three random seeds; a robust sharpness formula is used in rare cases where the computed sharpness is negative.Source: §B.3 · p. 18

B.4 Experiment for Convergence of Clipped Optimization Algorithms

  • For convergence experiments, learning rates are manually tuned, with their configurations reported in Table 3.The caption identifies Table 3 as the learning-rate configuration table.Source: §B.4 · p. 18, Table 3 · p. 19

C Directional Sharpness Results

The appendix presents directional-sharpness visualizations and ratios across tasks and optimizer trajectories, then discusses how sharpness, gradient correlation, and trajectory relate to optimization behavior. It also outlines a binary-classification setup and a Gauss–Newton Hessian approximation.

C.1 SGD Trajectory

  • Figures 7–10 show machine-translation landscape visualizations along the SGD trajectory at epochs 2, 5, 10, and 20, while Tables 4–5 give directional-sharpness ratios relative to SGD for machine-translation and autoregressive tasks.Table 4 covers three machine-translation experiments; Table 5 covers the autoregressive task.Source: Figure 7 · p. 20, Figure 8 · p. 21, Figure 9 · p. 22

C.3 Discussion

  • The discussion reports that directional sharpness is relatively stable across iterations for a given task, while coordinate-wise clipping improves sharpness and finds a better optimization direction.The authors describe this pattern as coherent across tasks, model architectures, iterations, and local geometries.Source: §C.3 · p. 34
  • Low directional sharpness alone is insufficient: Adafactor often has the lowest sharpness but may find a poor direction, while SGD can perform poorly despite good gradient correlation.The discussion identifies gradient correlation as another factor in convergence.Source: §C.3 · p. 34
  • On machine translation, SGD-trajectory landscapes are similar across optimizer directions, whereas Adam geometry changes after early iterations; Adafactor performs well in SGD but not Adam trajectory.The authors note that Adam generally outperforms SGD, so trajectory is not necessarily the explanation for Adam’s performance.Source: §C.3 · p. 34

D.1 Experiment Setup and Hyperparameters

  • The BERT binary-classification experiment uses IMDb; its two-dimensional output permits a rank-at-most-2k Gauss–Newton Hessian approximation for batch size k.The setup uses learning rate 10^-4 and batch size 25 for both experiments.Source: §D.1 · p. 35

D.2 Approximation of Hessian with Gauss-Newton Matrix

  • For each data point, the Gauss–Newton approximation combines the logits Jacobian with cross-entropy curvature; in the binary case, the resulting matrix has rank 1.For a batch, this structure reduces the spectral-norm computation to a p × |S| matrix.Source: §D.2 · p. 35

D.3 Results

Removing a small fraction of high-row-norm coordinates substantially improves the smoothness of the remaining function across batches and epochs, supporting optimization under the robust smoothness setting.

D.3 Results

  • Removing 1% to 2.5% of coordinates with row norms at least 4 times the mean makes the remaining function 2 to 3 times smoother across all batches and epochs.Table 8 reports the smoothness experiment; L is the full Hessian’s spectral norm, ε the fraction removed, and ℓ the remaining Hessian’s spectral norm.Source: §D.3 · p. 36, Table 8 · p. 36
  • The experiment removes coordinates whose row norm is at least 4 times the mean row norm.Source: §D.3 · p. 36
  • The authors take the consistent smoothness improvement as evidence that most coordinates can be optimized under the robust smoothness setting.Source: §D.3 · p. 36

E Directional Sharpness of ResNet

On ResNet-152 with CIFAR-10, adaptive algorithms have higher directional sharpness than SGD in the SGD trajectory, unlike the reported transformer behavior.

E Directional Sharpness of ResNet

Loading 2306.00204v1…