PaperlayerView exact revision on arXiv

Source-linked AI summary

On Empirical Comparisons of Optimizers for Deep Learning

Dami Choi, Christopher J. Shallue, Zachary Nado, Jaehoon Lee, Chris J. Maddison, George E. Dahl

arXiv:1910.05446v3cs.LGstat.ML

TL;DR

Optimizer comparisons in deep learning are difficult to interpret because rankings depend strongly on the hyperparameter search space and tuning protocol. The paper formalizes optimizer inclusions and evaluates them under realistic, increasingly extensive tuning, finding that adaptive methods do not underperform their included special cases.

  • Problem

    Optimizer choice affects training speed and predictive performance, but empirical comparisons lack adequate theoretical guidance and depend on methodological tuning choices.

  • Method

    The paper defines optimizers by update rules and free hyperparameters, formalizes inclusion relationships, and selects optimizer-specific search spaces through initial experiments.

  • Results

    Under exhaustive tuning, RMSPROP, ADAM, and NADAM never underperformed SGD, NESTEROV, or MOMENTUM, while rankings stabilized according to inclusion relationships.

  • Takeaways & Limitations

    Practitioners should tune all available hyperparameters of adaptive methods when feasible and report search spaces and tuning decisions in optimizer comparisons.

  • Takeaways & Limitations

    The results depend on the tuning protocols and workloads considered, and the study did not measure effects of varying batch size.

Abstract

from arXiv · show

Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

1. Introduction

Optimizer choice affects training speed and final predictive performance, yet comparisons lack adequate theoretical guidance. The paper argues that inclusion relationships and hyperparameter tuning protocols fundamentally shape empirical rankings.

  • Optimizer choice determines both training speed and final predictive performance.
  • Because no adequate theory guides optimizer selection, the field relies on empirical studies and benchmarking.
  • ADAM and RMSPROP can approximately simulate MOMENTUM when ADAM’s denominator ϵ is allowed to grow very large.
  • More general optimizers never underperform special cases in the experiments when inclusion relationships are respected through careful tuning.
  • Changing the hyperparameter tuning protocol can change optimizer rankings on a given workload, with rankings stabilizing according to inclusion relationships as tuning effort increases.
  • NADAM achieved 77.1% top-1 validation accuracy on ResNet-50 on ImageNet, exceeding MOMENTUM’s 76.5% under standard preprocessing pipelines.

2. Background and Related Work

Prior optimizer comparisons often use limited or mismatched tuning protocols, especially for adaptive methods’ ϵ hyperparameters. The paper emphasizes that realistic tuning must reflect practitioners’ choices and search effort.

  • Hyperparameter tuning is crucial to deep learning, but optimizer studies often use protocols that differ from application-focused neural network training.
  • Recent optimizer studies commonly compare ADAM and RMSPROP without tuning their ϵ hyperparameters.
  • Some prior studies tested only at most two ϵ values for ADAM, while De et al. considered a broad range.
  • Zaheer et al. and De et al. found that non-default ϵ values outperformed the default.
  • Applications have selected ϵ values up to eight orders of magnitude away from common defaults, including ϵ = 1 for RMSPROP and ϵ = 10^-6 for ADAM.
  • ADAM’s ϵ may act as a problem-dependent damping or trust-region parameter rather than merely a numerical-stability constant.

3. What is an optimizer?

The paper defines an optimizer through its update rule and tunable hyperparameters, then formalizes when one optimizer can approximate another. This taxonomy predicts performance only when tuning makes the relevant inclusions practically accessible.

  • What is an optimizer?: An optimizer is characterized by an update rule M and a hyperparameter setting φ that generate the parameter sequence θt.
  • What is an optimizer?: SGD uses a learning-rate schedule, while MOMENTUM adds a momentum parameter γ to combine the gradient with the previous parameter update.
  • What is an optimizer?: The choice of tuned hyperparameters defines the effective family of update rules, so MOMENTUM with free γ differs from MOMENTUM with γ fixed at 0.9.
  • The taxonomy of first-order methods: Optimizer inclusion means one update rule can approximately simulate another optimizer’s trajectory for every setting of the latter’s hyperparameters.
  • The taxonomy of first-order methods: SGD is a specialization of MOMENTUM because setting γ = 0 recovers the SGD update.
  • The taxonomy of first-order methods: With sufficiently tuned hyperparameters, an optimizer that includes another cannot be worse on any metric, although its additional hyperparameters may cost more to tune.
  • The taxonomy of first-order methods: In practice, inclusion may fail to predict performance if the matching hyperparameters are inaccessible, extreme, or difficult to discover.

4. Experiments

The experiments use optimizer-specific, increasingly broad hyperparameter searches across image-classification and language-modeling workloads. Across these settings, inclusion relationships consistently predict performance and training-speed comparisons, while expanding search spaces reverses prior rankings and exposes important tuning effects.

  • Experimental design: Optimizer-specific search spaces avoid assuming that similarly named hyperparameters have comparable values across optimizers.The protocol tunes optimization hyperparameters and parameterized learning-rate schedules independently for each optimizer.
  • Method validation: The final search spaces were checked for boundary optima, and the resulting error rates compared favorably with prior published results.This validation supports the authors’ claim that their methodology is competitive with expert tuning procedures.
  • Experimental design: The study evaluates optimizers on multiple image-classification and language-modeling workloads using quasi-random search with fixed feasible-trial budgets.Budgets were 10, 50, or 100 trials depending on the workload, and training speed was measured by steps to a fixed validation target.
  • Main findings: Across workloads, inclusion relationships hold for final validation error, test error, training error, and steps required to reach target validation error.A more general optimizer never underperforms its specializations within error bars; the result is not dependent on the exact tested step budgets or error targets.
  • Main findings: Optimizer choice matters unevenly: ResNet-32 on CIFAR-10 shows similar results across optimizers, whereas Transformer on LM1B shows clear predictive-performance and speed differences.On Transformer-LM1B, ADAM requires 60% as many steps as MOMENTUM to reach the target error and 25% as many steps as SGD to reach the same final result.
  • Main findings: RMSPROP, ADAM, and NADAM are not ordered by inclusion, and none consistently wins, although ADAM and NADAM generally edge out RMSPROP.NADAM reached 77.1% validation accuracy on ResNet-50/ImageNet with ϵ = 9475, while optimal ϵ values across four workloads spanned 10 orders of magnitude.
  • Reconciling prior work: Expanding the hyperparameter search space collapses or reverses earlier optimizer rankings, aligning them with inclusion relationships.Tuning additional optimizer hyperparameters and learning-rate schedules reproduces or removes prior reported advantages for RMSPROP, ADAM, MOMENTUM, and SGD.

5. Conclusions

The authors find that optimizer inclusion relationships remain meaningful under exhaustive tuning, while emphasizing that conclusions depend on workloads and tuning protocols. They recommend tuning adaptive methods broadly and reporting search spaces when comparing optimizers.

  • Conclusions: Under realistic, exhaustive tuning, more general optimizers never underperform their special cases, although incomparable optimizers show no consistent ranking.RMSPROP, ADAM, and NADAM never underperformed SGD, NESTEROV, or MOMENTUM, while some workloads lacked statistically significant ranking differences.
  • Conclusions: The study did not vary batch size, and its detailed findings are limited to similar workloads and tuning protocols.The authors specify uniform quasi-random tuning for tens to hundreds of trials over hypercube search spaces with their learning-rate schedule parameterization.
  • Conclusions: Practitioners who can afford tens or more runs should tune all hyperparameters of popular adaptive gradient methods rather than assume similarly named parameters share values.The authors emphasize that optimization hyperparameters can be coupled and that Adam’s optimal ϵ is problem-dependent.
  • Conclusions: Optimizer comparisons should report search spaces and identify which hyperparameters were tuned, especially when an optimizer underperforms one of its specializations.The paper treats search-space design as central to interpreting empirical optimizer rankings.
  • Conclusions: The update-rule hierarchy includes SGD as a specialization of MOMENTUM and NESTEROV, while MOMENTUM is included in RMSPROP, RMSTEROV, and ADAM.The paper establishes these relationships through parameter settings or limits of the corresponding update rules.
  • Conclusions: NESTEROV is included in NADAM through the limiting behavior of NADAM’s ϵ parameter.This provides the inclusion relationship used in the optimizer hierarchy.

B.1. Dataset Descriptions

The experiments span image, language, and sequence workloads with several standard neural-network architectures and dataset-specific preprocessing. Hyperparameters were tuned through quasi-random search, bootstrap uncertainty estimation, and log-scale search spaces.

  • B.1. Dataset Descriptions: The workload suite includes Fashion MNIST, CIFAR-10, ImageNet, LM1B, and War and Peace, using setups adapted from prior studies.The authors state that most setups matched Shallue et al. (2019), while War and Peace matched Schneider et al. (2019).
  • B.1. Dataset Descriptions: CIFAR experiments used standardization and, for ResNet-32 and CNN, padded random crops or horizontal reflections; VGG omitted random cropping for consistency with prior work.The preprocessing choices differed across models to match the cited experimental setups.
  • B.1. Dataset Descriptions: ImageNet training used random resized crops, horizontal reflections, color distortion, and global mean subtraction, while evaluation used a single central crop.Both training and evaluation crops were 224 × 224 pixels.
  • B.1. Dataset Descriptions: The models comprise Simple CNN, All-CNN-C, ResNet variants, VGG, LSTM, and Transformer architectures with architecture-specific normalization and regularization choices.Examples include batch normalization for ResNet and VGG, L2 regularization for CNN, and no dropout for the Transformer.
  • B.1. Dataset Descriptions: The tuning protocol sampled quasi-random hyperparameter values uniformly until a workload-specific number of feasible trials was obtained, then selected the best trial by the statistic of interest.Bootstrap samples estimated means, fifth percentiles, and ninety-fifth percentiles from the trial results.
  • B.1. Dataset Descriptions: The search spaces tuned η0, α0, 1 −γ, 1 −β1, 1 −β2, ϵ, and combinations thereof on a log scale.The paper reports both initial spaces used for refinement and final spaces used to generate plots.

D.1. CNN on Fashion MNIST

This section reports CNN experiments on Fashion MNIST, with linear learning-rate decay and tables for SGD, momentum, Nesterov, RMSProp, Adam, and Nadam.

  • D.1. CNN on Fashion MNIST: Linear learning-rate decay was used, with decay steps tuned between 0.5 and 1.0 times the training steps.The decay factor was tuned within the values shown in the tables.
  • D.1. CNN on Fashion MNIST: Table 3 reports SGD results for the Fashion MNIST CNN.
  • D.1. CNN on Fashion MNIST: Tables 4–9 report results for momentum, Nesterov, RMSProp, Adam, and Nadam.

D.2. ResNet-32 on CIFAR-10

This section reports ResNet-32 experiments on CIFAR-10 using linear learning-rate decay and optimizer-specific result tables.

  • D.2. ResNet-32 on CIFAR-10: Linear learning-rate decay was used, with decay steps tuned between 0.5 and 1.0 times the training steps.The decay factor was tuned within the values shown in the tables.
  • D.2. ResNet-32 on CIFAR-10: Tables 10–15 report results for momentum, Nesterov, RMSProp, Adam, and Nadam.

D.3. ResNet-50 on ImageNet

This section reports ResNet-50 experiments on ImageNet with linear learning-rate decay and separate tables for six optimizers.

  • D.3. ResNet-50 on ImageNet: Linear learning-rate decay was used, with decay steps tuned between 0.5 and 1.0 times the training steps.The decay factor was tuned within the values shown in the tables.
  • D.3. ResNet-50 on ImageNet: Tables 17–21 report results for momentum, Nesterov, RMSProp, Adam, and Nadam.

D.4. Transformer on LM1B

This section reports Transformer experiments on LM1B with linear learning-rate decay and optimizer-specific result tables.

  • D.4. Transformer on LM1B: Linear learning-rate decay was used, with decay steps tuned between 0.5 and 1.0 times the training steps.The decay factor was tuned over {10^-4, 10^-3, 10^-2, 10^-1, 1}.
  • D.4. Transformer on LM1B: The decay factor was tuned over five specified values in the LM1B Transformer experiments.
  • D.4. Transformer on LM1B: Tables 22–27 report results for SGD, momentum, Nesterov, RMSProp, Adam, and Nadam.

D.5.1. GRID SEARCH OVER LEARNING RATE

This section reports grid- and quasi-random tuning procedures centered on learning-rate schedules and selected optimizer hyperparameters. It also organizes the results and learning-rate ranges across optimizer-specific tables.

  • D.5.1. GRID SEARCH OVER LEARNING RATE: The experiments tune initial learning rates, with some protocols additionally tuning MOMENTUM’s γ, RMSPROP’s ϵ, and ADAM’s ϵ.The learning rate is tuned using either a shared grid or a quasi-random protocol.
  • D.5.1. GRID SEARCH OVER LEARNING RATE: The section presents separate result tables for SGD, MOMENTUM, RMSPROP, and ADAM, alongside tables listing learning-rate search ranges.Some SGD results are reused when no additional optimizer hyperparameters require tuning.
  • D.5.1. GRID SEARCH OVER LEARNING RATE: Several protocols use linear learning-rate decay, tuning decay steps over [0.5, 1.0] times training steps and decay factors from specified discrete ranges.The reported factor values include {10−4, 10−3, 10−2, 10−1}; another protocol uses values shown in accompanying tables.
  • D.5.1. GRID SEARCH OVER LEARNING RATE: Some experiments fix optimizer hyperparameters to values from Wilson et al. (2017) or Schneider et al. (2019), while Wilson-based settings use L2 regularization coefficient 0.0005.The Wilson-based protocol also decays the initial learning rate by 0.5 every 25 epochs.

E. Additional plots

The additional plots examine tuning convergence, search-space suitability, and optimizer comparisons under inclusion relationships. They report convergence after roughly two dozen trials and consistency across selected evaluation choices.

  • E. Additional plots: The search space is described as appropriate because the optimal values lie away from its boundaries.Figure 5 projects final validation error onto hyperparameter-space axes.
  • E. Additional plots: The best trial’s validation performance mostly converges with as few as 24 tuning trials for Transformer on LM1B and ResNet-50 on ImageNet.The plots show bootstrap-estimated 5th and 95th percentile shaded regions.
  • E. Additional plots: The best trial’s test performance mostly converges with as few as 23 tuning trials for a 2-layer LSTM on War and Peace.The plotted uncertainty uses bootstrap-estimated 5th and 95th percentiles.
  • E. Additional plots: Optimizer relative performance remains consistent with inclusion relationships when selecting for lowest training loss.For some ImageNet ResNet-50 optimizers, label smoothing makes loss values incommensurate with other optimizers.
  • E. Additional plots: The relevance of optimizer inclusion relationships does not depend on the exact step budgets or error targets chosen.Figure 10 reports this robustness directly.
Loading 1910.05446v3…