PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be

Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, Mark Schmidt

arXiv:2304.13960v1cs.LGmath.OC

TL;DR

Why Adam often outperforms SGD remains poorly understood. This paper varies batch size to test whether stochastic noise explains the gap, finding that Adam’s advantage grows with batch size and that its large-batch behavior resembles sign descent with momentum.

  • Problem

    Despite Adam’s success and frequent advantage over SGD, why it outperforms SGD remains poorly understood, limiting understanding of optimizer behavior and the structure it may exploit.

    Source: §1 · p. 1

  • Method

    The authors increase batch size up to the full dataset to test whether the SGD–Adam performance gap persists as subsampling noise decreases, focusing on language tasks.

    Source: §2 · p. 3

  • Results

    On language tasks, Adam’s advantage over SGD grows as batch size increases rather than shrinking with noise, and sign descent also improves with large batches.

    Source: §3.1 · p. 5, §5 · p. 7

  • Takeaways & Limitations

    For these language problems, Adam’s advantage over SGD at large batch sizes appears deterministic rather than a consequence of more robust gradient estimation.

    Source: §3.1 · p. 5

    Because matching iteration counts across batch sizes was computationally infeasible, the results may probe only the beginning of training relative to state-of-the-art workloads.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

The success of the Adam optimizer on a wide array of architectures has made it the default in settings where stochastic gradient descent (SGD) performs poorly. However, our theoretical understanding of this discrepancy is lagging, preventing the development of significant improvements on either algorithm. Recent work advances the hypothesis that Adam and other heuristics like gradient clipping outperform SGD on language tasks because the distribution of the error induced by sampling has heavy tails. This suggests that Adam outperform SGD because it uses a more robust gradient estimate. We evaluate this hypothesis by varying the batch size, up to the entire dataset, to control for stochasticity. We present evidence that stochasticity and heavy-tailed noise are not major factors in the performance gap between SGD and Adam. Rather, Adam performs better as the batch size increases, while SGD is less effective at taking advantage of the reduction in noise. This raises the question as to why Adam outperforms SGD in the full-batch setting. Through further investigation of simpler variants of SGD, we find that the behavior of Adam with large batches is similar to sign descent with momentum.

1 INTRODUCTION

The paper argues that stochastic noise is not the main cause of Adam’s advantage over SGD on transformers: the gap persists or grows as batch size increases. In full batch, sign descent can bridge most of the gap, suggesting a simpler way to study Adam’s behavior.

1 INTRODUCTION

  • Adam often outperforms SGD by a wide margin, but standard optimization and online-learning theory do not explain this empirical gap.The cited analyses under conventional assumptions mainly show Adam does not do much worse than gradient descent, while adversarial online-learning analyses require step-size decreases that are too fast for practical performance.Source: §1 · p. 1

1.1 CONTRIBUTIONS

  • The authors suggest adaptive optimizers may exploit problem structure missing from current theory; Adam’s similarity to sign descent is promising but does not explain its success.They report that this full-batch similarity had not previously been shown empirically and propose studying simpler sign descent to understand Adam and the problem classes where it works well.Source: §1 · pp. 1–2, §1.1 · p. 3

1 INTRODUCTION

  • J. Zhang et al. (2020b) hypothesize that heavy-tailed stochastic-gradient errors hurt SGD more than Adam or gradient clipping, motivating noise-robust algorithms.Their observed heavier tails and larger, more consistent optimizer gap on language tasks than image tasks are shown in Figure 1; the error distribution is described as closer to α-stable for language models and Gaussian for image models.Source: §1 · p. 2, Figure 1 · p. 2
  • Adam’s success in large-batch settings and its better batch-size scaling cast doubt on heavy-tailed-noise robustness as the primary source of its advantage.The introduction also notes that optimizer behavior on language tasks appears less variable than on image tasks, despite the proposed noise explanation.Source: §1 · p. 2

1.1 CONTRIBUTIONS

  • Removing noise with full-batch training does not reduce the SGD–Adam gap, and increasing batch size widens it because SGD benefits less from reduced noise.SGD’s progress per iteration improves less with batch size than Adam’s.Source: §1 · p. 2, §1.1 · p. 3, Figure 2 · p. 3
  • In full batch, sign descent bridges most of the gap between gradient descent and Adam, and momentum makes their performance similar.Sign descent performs poorly at small batch sizes but improves with larger batches; normalized updates also improve on SGD, though sign descent outperforms plain normalization in full batch.Source: §1.1 · p. 3

2 EXPERIMENTAL DESIGN

The experiments vary batch size to reduce subsampling noise and test whether the SGD–Adam gap persists, focusing on training loss and emphasizing language tasks while including image tasks for comparison.

Problems

  • The study covers MNIST and CIFAR-10 with a small CNN and ResNet18, PTB and WikiText-2 with a small transformer and Transformer-XL, and SQuAD with fine-tuned DistillBERT.The authors focus on language-task behavior in relation to the heavy-tail hypothesis, while including image tasks for comparison.Source: §2 · p. 3

Training performance

  • The experiments target optimization cases where SGD makes insufficient progress, use training loss to select hyperparameters, and prioritize insight from simple updates over state-of-the-art performance.A constant step size is used, although schedules can yield better performance.Source: §2 · pp. 3–4
  • The Adam–SGD performance gap grows with batch size on language models, even though fewer iterations at larger batches bias the comparison toward smaller batches.For WikiText-2 and SQuAD, smaller batches still run for more iterations despite being stopped after one epoch; the gap grows despite this bias.Source: Figure 3 · p. 4

Experimental scope

  • Hold-out results appear in Appendix C, and large-batch generalization is noted to require additional tuning and regularization (Shallue et al., 2019; Geiping et al., 2022).The authors also note practical computational limitations in comparing optimizers at large batch sizes.Source: §2 · pp. 3–4

Batch sizes and reducing noise

  • Batch sizes S, M, L, and XL represent approximately 4× increases; progress is measured per iteration, and full-batch runs remove at most 0.5% of data to divide it evenly.Larger batches run for more epochs but produce fewer iterations; disabling dropout also preserves the observed trends.Source: §2 · p. 4

Hyperparameter tuning

  • Step sizes are tuned separately for each batch size by grid search, with three random seeds and methods tested with and without momentum.Momentum is fixed at 0.9; Adam uses β2 = 0.999 and ϵ = 10−8, with other parameters at defaults.Source: §2 · p. 4

3 BEHAVIOR OF SGD AND ADAM IN LARGE BATCH

The SGD–Adam gap persists or grows as batch size increases: SGD makes little use of reduced gradient noise, whereas Adam improves and reaches similar error in fewer iterations.

3 BEHAVIOR OF SGD AND ADAM IN LARGE BATCH

  • Removing stochasticity does not reduce the SGD–Adam gap and can widen it on language tasks; even without momentum, Adam outperforms momentum SGD there.In full batch, Adam improves on CIFAR-10 while SGD worsens on PTB and WikiText-2.Source: §3 · p. 4

3.1 LIMITATIONS

  • Comparable-iteration results show the gap grows with batch size, especially on language models, though the smallest batches are stopped after only one epoch.Neither optimizer achieves reasonable training error after that single epoch, so those results need not represent standard training.Source: §3 · p. 4, §3.1 · p. 5
  • Across transformer tasks, SGD’s loss trajectory changes little with batch size, while Adam reaches similar or lower error in fewer iterations; the gap is greatest at full batch.The full-trajectory comparison addresses the concern that stopping small-batch runs after one epoch may not reflect standard training.Source: Figure 4 · p. 5, §3.1 · p. 5
  • For both optimizers, momentum improves convergence speed more at larger batch sizes.The passage relates this pattern to prior studies of momentum and Adam batch-size scaling (Shallue et al., 2019; Smith et al., 2020; G. Zhang et al., 2019).Source: §3.1 · p. 5
  • The findings suggest stochasticity is not SGD’s limiting factor on language tasks and that Adam’s large-batch advantage is deterministic, though small-batch heavy-tailed dynamics remain possible.The widening gap is inconsistent with the explanation that Adam wins because it estimates gradients more robustly.Source: §3.1 · p. 5

4 ALTERNATIVE INTERPRETATIONS

The section tests whether simpler update rules can explain Adam’s full-batch advantage. Sign descent performs poorly at small batch sizes but approaches Adam in full batch, while normalized gradient descent scales less well as batch size grows.

4.1 SIGN DESCENT AND NORMALIZED UPDATES

  • The experiments isolate simpler alternatives to Adam’s full-batch performance, comparing normalized gradients, which change step magnitude, with sign descent, which also changes direction.Both updates use heavy-ball momentum by accumulating the transformed gradient; results are shown with β = 0.9 and β = 0.Source: §4.1 · p. 6
  • Sign descent can perform worse than plain SGD at small batch sizes, but it becomes competitive with Adam in full batch; normalized GD scales better than plain GD but plateaus earlier.The relationship between sign descent and normalized GD reverses: normalized GD performs better at small batch sizes, while sign descent outperforms it in full batch.Source: §4.1 · p. 6, Figure 5 · p. 6, Figure 6 · p. 7
  • In full batch, sign descent with momentum closes most of the gap between SGD and Adam and can outperform Adam on some problems; its batch-size improvement is closer to Adam’s than normalized GD’s.A gap between Adam and sign descent remains overall.Source: Figure 6 · p. 7, §4.1 · p. 7

4 ALTERNATIVE INTERPRETATIONS

  • Sign descent’s improvement over gradient descent supports a deterministic contribution to Adam’s performance gap, with some benefit of second-moment estimation attributable to the difference between gradient descent and sign descent.The authors present this connection as a possible avenue for analyzing Adam and the impact of noise.Source: §4 · pp. 5–6, §4.1 · p. 7

5 DISCUSSION

The discussion argues that Adam’s advantage over gradient descent is not primarily explained by robustness to sampling noise: Adam performs well in full batch, and sign descent shares its improvement with larger batches. It frames sign descent as a promising but poorly understood direction and identifies theoretical and experimental limitations.

5 DISCUSSION

  • Chen et al. (2023) found that an evolutionary-search variant of sign descent with momentum outperformed Adam variants on several deep learning models.The discussion says the sign-descent perspective conflicts with Adam’s usual characterization as adaptive, and that quadratic models alone seem insufficient to explain sign descent’s performance; understanding its benefits could clarify deep-learning loss landscapes.Source: §5 · pp. 7–8
  • In full batch, Adam outperforms gradient descent, while sign descent’s performance improves with batch size and becomes competitive with Adam; sign descent can nevertheless lag at small batch sizes.Figure 7 shows sign descent moving from the worst option at medium batch size to competitive with Adam in full batch. The discussion says these findings run counter to the idea that Adam’s advantage comes from a more robust estimate of the descent direction.Source: §5 · p. 7, Figure 7 · p. 8
  • Standard smoothness analysis ties its performance guarantee to a gradient-descent step, while evidence questions whether that assumption holds even in simple deep-learning problems.The discussion points to relaxed smoothness assumptions as a way to better characterize loss landscapes and capture benefits of clipping and sign-based algorithms.Source: §5 · p. 8
  • Gradient-dependent smoothness motivates clipping by normalizing large steps, but the experiments suggest normalization alone may not close the SGD–Adam gap.The discussion suggests element-wise clipping or sign-based methods may exploit additional problem structure beyond normalization.Source: §5 · pp. 8–9
  • Whether practical assumptions can explain sign-based methods’ deterministic advantages remains open, and existing coordinate-wise analyses require many problem-specific constants.The cited analyses use element-wise smoothness or an L∞ smoothness measure; one sign-descent variant uses exponential moving averages to estimate signs.Source: §5 · p. 9
  • The large SGD–Adam gap may depend on architecture: it is less pronounced on the image tasks, and the discussion identifies a minimal explanatory model as a research need.Possible factors include language tasks’ larger vocabularies, normalization layers, skip connections, and initialization choices; the authors hypothesize Batch Normalization may explain ResNet18/CIFAR10’s different batch-size behavior.Source: §5 · p. 9, §5.1 · p. 9
  • Designing sign-descent methods that remain robust at small batch sizes and obtaining guarantees for exponential moving averages remain open problems.Existing guarantees for these averages depend negatively on β2, while majority voting has provable guarantees in distributed settings.Source: §5 · p. 9

5.1 LIMITATIONS

  • The experiments are limited by costly full-batch runs, focus on training loss rather than validation tuning, and architecture-dependent behavior that may not generalize beyond transformers.The authors use intermediate batch sizes and constant step sizes, so results may capture only early training; warm-up or decay could change SGD’s transformer performance. Validation metrics suggest training-error improvements can yield faster lower testing loss before overfitting.Source: §5.1 · p. 9

A EXPERIMENTAL DETAILS

The appendix collects experimental details on datasets and models, batch sizes and epoch counts, algorithm pseudocode, Figure 1 histograms, and grid search. Reproduction code and data generated by each run are available on GitHub.

A EXPERIMENTAL DETAILS

  • Appendix A covers datasets and models, batch sizes and epoch counts, algorithm pseudocode, Figure 1 histograms, and grid search.Source: A EXPERIMENTAL DETAILS · p. 13
  • Reproduction code and data generated by each run are available at the project’s GitHub repository.https://github.com/fkunstner/noise-sgd-adam-signSource: A EXPERIMENTAL DETAILS · p. 13

A.1 DATASETS AND MODELS

The experiments span image classification, language modeling, and question answering, using specified datasets and model architectures. Performance is evaluated on each dataset’s default test split as validation error.

A.1 DATASETS AND MODELS

  • All tasks use the default train/test split and report test-split performance as validation error; full-batch runs retain 99.5% of the dataset at each epoch.Batch size counts input-output samples: image-label pairs, sequence pairs for language models, and question/answer pairs for SQuAD.Source: §A.1 · p. 13, Table 1 · p. 14
  • MNIST uses a modified LeNet5 for image classification, with three convolutional and two linear layers, average pooling, tanh activations, and a linear output.The modified model is identified as LeNet5.Source: §A.1 · p. 13
  • CIFAR10 uses a PyTorch implementation of ResNet18, an 18-layer CNN with residual connections (He et al., 2016).Source: §A.1 · p. 13
  • PTB uses word-level language modeling with 35-token sequences and a two-layer, two-head Transformer with 200-dimensional embeddings (Vaswani et al., 2017).The model is a PyTorch tutorial example, and data processing follows Dai et al. (2019).Source: §A.1 · p. 13
  • WikiText-2 uses word-level language modeling with Transformer-XL, sequence length 128, six layers, and a target length of 128 (Merity et al., 2017; Dai et al., 2019).Model and preprocessing follow Dai et al. (2019); hyperparameters follow its ENWIK8 base experiment with modifications from J. Zhang et al. (2020b).Source: §A.1 · p. 13
  • SQuAD v1.1 uses question answering by fine-tuning pretrained DistillBERT with HuggingFace model and data-processing implementations (Rajpurkar et al., 2016; Sanh et al., 2019).The HuggingFace implementation is cited to Wolf et al. (2020).Source: §A.1 · p. 13

A.2 BATCH SIZES AND EPOCH COUNTS … B ADDITIONAL PLOTS

The appendix details how batch-size comparisons and optimizer tuning were conducted, and reports that the SGD–Adam gap grows with batch size despite iteration-count bias favoring smaller batches. It also documents implementation choices and supplementary analyses of sign descent and dropout.

A.2 BATCH SIZES AND EPOCH COUNTS

  • Batch sizes increase fourfold across S, M, L, and XL, while epochs are adjusted so successive settings’ iteration counts remain within roughly a factor of 2–8.The authors reduce epochs for larger batches because matching iteration counts would make full-batch experiments computationally infeasible.Source: §A.2 · p. 14
  • Despite stopping runs at roughly comparable—not identical—iteration counts, the SGD–Adam gap grows with batch size; on larger language datasets, fewer iterations explain performance degradation as batch size increases.Stopping times were chosen to make iteration counts similar, although the small-batch runs could only be stopped after one epoch.Source: §A.2 · p. 14, Table 2 · p. 15

SQuAD

  • Removing dropout verifies that sign descent bridges the (S)GD–Adam gap in the deterministic full-batch setting on PTB and WikiText-2.Gradient accumulation reduces peak memory for full-batch runs, WikiText-2 at batch sizes ≥L, and SQuAD at batch sizes ≥M.Source: SQuAD · p. 15

A.3 ALGORITHMS PSEUDOCODE

  • Experiments use PyTorch implementations of SGD and Adam; normalized variants (sign descent, normalized GD) use heavy-ball momentum, main step sizes are grid-searched, and momentum β/β1 is set to 0 or 0.9.Adam’s remaining hyperparameters stay at defaults: β2 = 0.999 and ϵ = 10−8.Source: §A.3 · p. 15

A.4 HISTOGRAMS IN FIGURE 1

  • Initialization histograms show stochastic-gradient-error distributions consistent with J. Zhang et al. (2020b), who also report that the pattern persists through training.For ResNet18, the average of minibatch gradients is used because batch normalization means there is no well-defined full-data gradient.Source: §A.4 · p. 16

A.5 GRID SEARCH

  • Step sizes are grid-searched from powers of ten, refined to half-powers, and selected by minimizing the maximum end-of-training training loss across three seeds.DistillBERT on SQuAD is pretrained; other runs vary data ordering and initialization across seeds.Source: §A.5 · p. 16

B ADDITIONAL PLOTS

  • Appendix B lists supplementary plots on batch-size gaps, sign descent’s relative performance, and whether findings hold without dropout.Source: B ADDITIONAL PLOTS · p. 16

B.1 GAP VS. BATCH SIZE (FIGURE 4 INCLUDING IMAGE TASKS)

As batch size increases, Adam reaches similar error in fewer iterations, while SGD generally shows little improvement beyond a batch-size threshold. These trends are less clear on MNIST and CIFAR-10.

B.1 GAP VS. BATCH SIZE (FIGURE 4 INCLUDING IMAGE TASKS)

  • Adam achieves similar error in fewer iterations as batch size increases, taking better advantage of reduced noise than SGD across most problems.The figure compares five batch sizes from the small-batch to full-batch settings; larger batches terminate earlier.Source: Figure 8 · p. 16
  • SGD follows a similar trajectory across most batch sizes, suggesting that increasing batch size past a threshold does not improve performance.Similar results would be expected from using the same batch size but terminating earlier.Source: Figure 8 · p. 16
  • The batch-size trends are less clear on MNIST and CIFAR-10; SGD performs better on CIFAR-10 than on other problems, possibly because of Batch Normalization layers.The figure shows results without momentum on top and with momentum on the bottom.Source: Figure 8 · p. 16

B.2 GAP VS. BATCH SIZE (VERSION OF FIGURE 6 INCLUDING IMAGE TASKS)

Sign descent gains more from increasing batch size than the other optimizers, becoming competitive with Adam in full batch despite poor small-batch performance. Across batch sizes, it scales best and overtakes normalized gradient descent in full batch.

B.2 GAP VS. BATCH SIZE (VERSION OF FIGURE 6 INCLUDING IMAGE TASKS)

  • Sign descent scales best as batch size increases, overtaking normalized gradient descent in full batch even though normalized gradient descent performs better at small batch sizes.Normalized gradient descent also scales better with batch size than plain gradient descent; larger batches run for fewer iterations and terminate earlier.Source: Figure 9 · p. 17

B.3 RELATIVE PERFORMANCE OF SIGN DESCENT IN MEDIUM VS. FULL BATCH (VERSION OF FIGURE 7 WITHOUT MOMENTUM)

  • In full batch, sign descent becomes competitive with Adam and closes most of the SGD–Adam gap, despite performing worse than SGD at small batch sizes.Its improvement with increasing batch size is greater than for the other optimizers; the figure shows results without momentum, with a similar trend for optimizers with momentum.Source: Figure 10 · p. 17

B.4 VERIFYING THE RESULTS HOLD WITHOUT DROPOUT … C.2.2 RESNET18 ON CIFAR10

Disabling dropout preserves the trend that sign descent closes most of the full-batch SGD–Adam gap, despite lower training error. The appendices show step-size grid searches and selected-run training curves across the listed models and datasets.

B.4 VERIFYING THE RESULTS HOLD WITHOUT DROPOUT

  • With dropout disabled, sign descent still closes most of the full-batch gap between SGD and Adam, although the models achieve lower training error.The settings match the full-batch experiments except for disabling dropout.Source: §B.4 · p. 18, Figure 11 · p. 18

C.1.1 MODIFIED LENET5 ON MNIST

  • Figure 12 shows modified LeNet5 on MNIST: grid-searched step-size outcomes and training curves for the selected step-size.The figure's top panels show end-of-training loss and evaluation metrics; its bottom panels show training trajectories.Source: Figure 12 · p. 19

C.1.2 RESNET18 ON CIFAR10

  • Figure 13 presents the same step-size grid-search and selected-run views for ResNet18 on CIFAR10.It shows end-of-training loss and evaluation metrics across evaluated step sizes, plus training curves for the selected step-size.Source: Figure 13 · p. 20

C.1.3 SMALL TRANSFORMER ON PTB

  • Figure 14 shows step-size grid-search outcomes and selected-run training curves for the Small Transformer on PTB.The figure includes end-of-training loss and evaluation metrics for evaluated step sizes.Source: Figure 14 · p. 21

C.1.4 TRANSFORMER-XL ON WIKITEXT-2

  • Figure 15 presents step-size grid-search outcomes and selected-run training curves for Transformer-XL on WikiText-2.The figure includes end-of-training loss and evaluation metrics for evaluated step sizes.Source: Figure 15 · p. 22

C.1.5 DISTILLBERT FINETUNING ON SQUAD

  • Figure 16 shows step-size grid-search outcomes and selected-run training curves for DistillBERT fine-tuning on SQuAD.The figure includes end-of-training loss and evaluation metrics for evaluated step sizes.Source: Figure 16 · p. 23

C.2.1 MODIFIED LENET5 ON MNIST

  • Figure 17 presents step-size grid-search outcomes and selected-run training curves for modified LeNet5 on MNIST.The figure includes end-of-training loss and evaluation metrics for evaluated step sizes.Source: Figure 17 · p. 24

C.2.2 RESNET18 ON CIFAR10

  • Figure 18 presents step-size grid-search outcomes and selected-run training curves for ResNet18 on CIFAR10.The figure includes end-of-training loss and evaluation metrics for evaluated step sizes.Source: Figure 18 · p. 25

C.2.3 SMALL TRANSFORMER ON PTB · C.2.4 TRANSFORMER-XL ON WIKITEXT-2 · C.2.5 DISTILLBERT FINETUNING ON SQUAD

The appendices document step-size grid searches and selected-run training curves across batch sizes for a small Transformer on PTB, Transformer-XL on WikiText-2, and DistillBERT fine-tuning on SQuAD.

C.2.3 SMALL TRANSFORMER ON PTB

  • For the small Transformer on PTB, the grid search reports end-of-training loss and evaluation metrics for tested step sizes, with initialization shown as a reference.Source: Figure 19 · p. 26
  • For the small Transformer on PTB, the figure tracks training loss and evaluation metrics over training for the selected step size at each batch size.Source: Figure 19 · p. 26

C.2.4 TRANSFORMER-XL ON WIKITEXT-2

  • For Transformer-XL on WikiText-2, the grid search reports end-of-training loss and evaluation metrics for tested step sizes, with initialization shown as a reference.Source: Figure 20 · p. 27
  • For Transformer-XL on WikiText-2, the figure tracks training loss and evaluation metrics over training for the selected step size at each batch size.Source: Figure 20 · p. 27

C.2.5 DISTILLBERT FINETUNING ON SQUAD

  • For DistillBERT fine-tuning on SQuAD, the grid search reports end-of-training loss and evaluation metrics for tested step sizes, with initialization shown as a reference.Source: Figure 21 · p. 28
  • For DistillBERT fine-tuning on SQuAD, the figure tracks training loss and evaluation metrics over training for the selected step size at each batch size.Source: Figure 21 · p. 28
Loading 2304.13960v1…