PaperlayerView exact revision on arXiv

Source-linked AI summary

Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers

Robin M. Schmidt, Frank Schneider, Philipp Hennig

arXiv:2007.01547v6cs.LGstat.ML

TL;DR

Choosing among hundreds of stochastic optimization methods and tuning their hyperparameters is a major practical challenge without a clearly dominant method. This paper benchmarks fifteen popular first-order optimizers to ground comparisons in empirical evidence, finding that optimizer performance varies across problems, tuning helps about as much as trying other optimizers, and ADAM remains viable without clearly dominating.

  • Problem

    Practitioners face hundreds of optimization methods and hyperparameters, while no method clearly dominates the competition.

  • Method

    The study conducts a large-scale empirical benchmark of fifteen popular first-order optimizers, comparing established and newer methods and examining default versus tuned hyperparameters.

  • Results

    Across problems, optimizer performance varies, tuning helps about as much as trying other optimizers, and ADAM remains viable without being clearly superior.

  • Takeaways & Limitations

    A reduced subset of optimizers generally delivers competitive results, with ADAM a strong baseline and RMSPROP or NAG useful alternatives on individual problems.

  • Takeaways & Limitations

    The conclusions may not generalize to workloads such as GANs, reinforcement learning, or applications where memory usage is crucial.

Abstract

from arXiv · show

Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with a conclusive ranking, then at least with evidence-backed heuristics. To do so, we perform an extensive, standardized benchmark of fifteen particularly popular deep learning optimizers while giving a concise overview of the wide range of possible choices. Analyzing more than $50,000$ individual runs, we contribute the following three points: (i) Optimizer performance varies greatly across tasks. (ii) We observe that evaluating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer. (iii) While we cannot discern an optimization method clearly dominating across all tested tasks, we identify a significantly reduced subset of specific optimizers and parameter choices that generally lead to competitive results in our experiments: Adam remains a strong contender, with newer methods failing to significantly and consistently outperform it. Our open-sourced results are available as challenging and well-tuned baselines for more meaningful evaluations of novel optimization methods without requiring any further computational efforts.

1. Introduction

Deep learning practitioners face hundreds of optimization methods and limited guidance for choosing among them. This study addresses that challenge with a large standardized benchmark of fifteen popular optimizers across representative problems and extensive runs.

  • Choosing an optimizer matters because it strongly influences training speed and the learned model’s final performance.
  • Limited theory and small empirical comparisons leave optimizer selection dependent on personal experience, informal discussion, and hype.
  • The benchmark evaluates fifteen popular optimizers on representative deep learning problems, drawing conclusions from tens of thousands of training runs.
  • The selection covers popular methods, while excluded optimizers represented less than 4% of mentions in 2020.
  • The study evaluates fifteen optimizers on eight deep learning problems using four schedules and dozens of hyperparameter settings.

2. Insights from more than 50,000 optimization runs.

Across more than 50,000 runs, optimizer performance depends strongly on the task, and no method clearly dominates all tested workloads. The results suggest that trying multiple default optimizers can be as effective as tuning one optimizer, while motivating more meaningful optimization research.

  • Optimizer performance depends highly on the problem, and no optimizer clearly dominates across all tested workloads.
  • 53,760 unique runs provide open, detailed baselines for future optimizer benchmarks and meta-learning approaches.
  • A practitioner can perform about equally well by tuning one benchmark optimizer or trying several optimizers with default settings and selecting the winner.
  • The authors argue that optimizer research should prioritize significant conceptual, functional, or performance improvements over incremental additions.
  • Prior work shows that optimizer rankings depend significantly on the hyperparameter tuning budget and search space.
  • This study targets generally effective optimizers when practitioners lack prior knowledge of problem-specific hyperparameter values.

2. Benchmarking process

The benchmark makes explicit choices about tasks, optimizer scope, tuning budgets, schedules, and evaluation metrics to keep a broad comparison feasible. It evaluates fifteen popular first-order optimizers across eight deep learning problems and 1,920 configurations, producing 53,760 training curves.

  • Benchmark scope: The benchmark covers eight diverse real-world deep learning problems and selects fifteen popular first-order optimizers rather than attempting to include every method.The optimizer selection reflects popularity, not necessarily superior performance.
  • Experimental scale: 1,920 configurations yield 53,760 unique training curves after hyperparameter search and confidence-interval estimation.Each optimizer is evaluated in 128 settings across eight problems, four budgets, and four schedules.
  • Evaluation: The analysis focuses on final test accuracy, or final test loss when accuracy is undefined, as the primary comparative metric.The selected metric is intended to reflect generalization and practical relevance.
  • Experimental factors: The experiment combines four tuning budgets with four learning-rate schedules for each optimizer and problem.The budgets include one-shot defaults and 25, 50, or 75 random-search tuning runs.
  • Tuning: Parameters are tuned with random search, using author-informed sampling distributions and refined bounds for the largest budget.The sampling distribution acts as a prior over promising parameter settings, so its design can affect performance.
  • Design decisions: The study limits schedules to learning rates and treats different choices of tunable ADAM parameters as distinct optimizer variants.This keeps the configuration space feasible while acknowledging that parameterization changes an optimizer’s practical behavior.

3. Results

Optimizer performance depends strongly on the task: no single method dominates, and rankings can change across workloads and evaluation settings. Trying several optimizers with defaults can approach the results of extensive tuning, while larger budgets and schedules improve median performance but not uniformly.

  • Default settings: The tuned AMSBOUND reaches 2.4% higher test accuracy than untuned SGD in the cited Figure 2 comparison.Figure 2 compares untuned optimizers against tuned optimizers across tasks and averages results over 10 random seeds.
  • Default settings: Trying a few optimizers with default settings can yield competitive results with less tuning effort than extensively tuning one optimizer.The comparison is suggested as a practical alternative for new tasks, though it may not hold for structurally different workloads.
  • Tuning and schedules: Roughly 3.4% median relative improvement comes from the medium tuning budget without a schedule versus default parameters without a schedule.Increasing the budget improves median performance, but returns diminish.
  • Tuning and schedules: Roughly 5.2% median relative improvement is obtained with the large tuning budget and a trapezoidal learning-rate schedule versus default parameters.Individual effects vary widely across optimizers and problems.
  • Cross-task performance: Optimizer rankings are problem-dependent, with no single method dominating across all workloads.Optimizers that perform well on one problem can perform poorly on another.
  • Optimizer-specific patterns: NAG often beats ADAM on image-classification tasks, while RMSPROP is consistently strongest for the natural-language-modeling task.These patterns suggest that optimizer suitability may differ by problem type.

4. Limitations

The benchmark is intended as a broad but feasible empirical study rather than a definitive account of optimizer performance. Its conclusions are bounded by the selected problems, fixed settings, search strategy, and omitted techniques.

  • Generalization: Results may not generalize beyond the selected DEEPOBS workloads, training parameters, software framework, and other experimental choices.The authors specifically caution that conclusions may not extend to GANs, reinforcement learning, or memory-critical applications.
  • Generalization: The benchmark excludes large-scale workloads such as ImageNet and transformer models because covering them would make the combinatorial study infeasible.Including larger models would require reducing the number of optimizers, schedules, or tuning methods tested.
  • Benchmark scope: The study fixes each problem’s L2 regularization and batch size and omits weight averaging and ensemble methods to keep computation feasible.Future work could examine how these techniques interact with different optimizers.
  • Figure 4 guide: Figure 4 reports mean test-set performance over 10 seeds for eight problems under the large tuning budget without a learning-rate schedule.Untuned ADAM and ADABOUND are reference markers, while only tuned ADAM’s standard deviation is shown.
  • Search strategy: Optimizer rankings depend significantly on the hyperparameter search space, so poorly chosen sampling distributions can make successful tuning difficult.The benchmark uses broad initial spaces, dozens of runs, and refinement for the large budget, but practitioners still inherit the search-space problem.

5. Conclusion

The benchmark finds no universally dominant optimizer: performance varies across problems, while Adam remains viable and similar optimizer distributions raise concerns about noise in new-method comparisons.

  • No optimization method clearly dominates across the competition, leaving optimizer choice problem-dependent.
  • ADAM remains viable across many problems, while NAG and RMSPROP improve performance on individual problems.
  • Tuning helps about as much as trying other optimizers in the benchmark.
  • Different optimizers show surprisingly similar performance distributions to a single method that is re-tuned or rerun with different random seeds.
  • The authors question how much insight new methods provide when they are conceptually and functionally close to existing optimizers.

A. List of optimizers and schedules considered

The benchmark covers a selected subset of optimizers and commonly used learning-rate schedules, with tables documenting the considered methods and configuration choices.

  • Table 2 lists the optimizers considered, which represent only a subset of existing deep-learning methods.
  • Schedule parameters include the initial learning rate α0, lower and upper bounds αlo and αup, switch time Δt, and decay factor k.
  • The schedule overview includes linear, triangular plus exponential, cosine with warm restarts and decay, and warmup or step-decay variants.
  • Table 4 records each selected optimizer’s hyperparameters, defaults, tuning distributions, and scheduled hyperparameters.

C. Robustness to random seeds

The benchmark evaluates seed sensitivity explicitly: tuning uses one seed, final settings use ten, and unstable learning-rate regions can produce divergent or inconsistent outcomes.

  • 10 seeds are used for each final setting to assess optimizer stability, while tuning uses a single seed to limit benchmark cost.
  • Learning-rate danger zones can produce either successful training or divergence across seeds, making tuned settings unstable.
  • Figure 5 compares an initial tuning seed with nine additional seeds while showing their mean and one-standard-deviation variability.
  • 18, 24, and 17 divergent seeds occur for the small, medium, and large budgets, respectively, roughly 0.5% of runs per budget.
  • SGD, ADAGRAD, and ADADELTA account for most divergent seeds across the three budgets.

D. Re-Tuning experiments

Re-tuning RMSPROP and ADADELTA with independent seeds changes individual outcomes, but differences tend to average out across multiple optimization problems.

  • RMSPROP and ADADELTA are re-tuned on all problems using independent seeds for tuning and ten final-setting repetitions.
  • Re-tuning can shift the distribution of results for the same optimizer across random seeds.
  • Re-tuning with different seeds can change optimizer rankings on individual problems, while rankings tend to average out across multiple problems.
  • Small performance margins may disappear after re-tuning, supporting the absence of a clearly dominant optimization method.
  • Figures 6 and 7 display mean test performance for RMSPROP and ADADELTA across eight problems, with original and re-tuned results distinguished by solid and dashed lines.

E. List of schedules selected

The benchmark uses several learning-rate schedules, including cosine decay, cosine warm restarts, and a trapezoidal schedule with warm-up and cool-down. Figures and tables document optimizer mentions and schedule choices.

  • All learning-rate schedules are scaled by the tuned or default initial learning rate.
  • Figure 8 illustrates the selected schedules over a 150-epoch training duration.
  • Cosine decay follows a half-cosine trajectory from 1 to 0.
  • Cosine warm restarts use an initial cycle length of ∆t = 10, doubling after each cycle without discounting.
  • The trapezoidal schedule uses warm-up and cool-down periods each lasting one-tenth of training.
  • Optimizer mentions in ArXiv titles and abstracts are normalized by year, with non-selected methods grouped as Other.

G. Improvement after tuning

Tuning generally improves optimizer performance, but negative within-optimizer changes can arise from poor tuning distributions, unlucky randomness, or seed variation. Increasing the tuning budget reduces these negative diagonal effects.

  • Negative diagonal entries represent cases where tuning an optimizer on its respective task reduced performance.
  • Poor tuning distributions, unlucky tuning randomness, or worse results on additional seeds can produce negative tuning changes.
  • Most negative diagonal entries vanish or diminish with more tuning runs and evaluation on the tuning seed.
  • Figures 10–13 compare untuned one-shot performance with tuned performance under small and large tuning budgets.

H. Optimizer performance across test problems

Optimizer performance varies substantially across the eight problems and remains dependent on tuning budget and learning-rate schedule. Adam and related variants are competitive, but changing optimizers across problems achieves the highest observed performance.

  • The benchmark compares mean test-set performance over 10 random seeds across eight optimization problems under multiple budgets and schedules.
  • A small-budget selection of winning untuned optimizers gives decent performance on most problems with less effort.
  • Adding tuned Adam or related variants to that selection produces a very competitive overall result.
  • The absolute top performance is achieved by changing optimizers across different problems.
  • Adding cosine or trapezoidal schedules generally increases top performance, although the medium budget need not always outperform the small budget.
  • Training-loss results preserve the cross-problem performance variation and do not reveal a clear optimizer winner; Adam and variants perform near the top.
Loading 2007.01547v6…