PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizer Benchmarking Needs to Account for Hyperparameter Tuning

Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, François Fleuret

arXiv:1910.11758v4cs.LGstat.ML

TL;DR

Optimizer studies often emphasize near-optimal settings without accounting for the effort needed to find them. The paper benchmarks optimizers using automatic hyperparameter search at specified budgets and finds that results overall favor Adam, especially when tuning budgets are low.

  • Problem

    Optimizer comparisons often emphasize best-case performance despite strong dependence on hyperparameter choices, leaving unclear how readily automatic search can find satisfactory settings.

    Source: §1 · p. 1

  • Method

    The authors compare optimizers using random search over their hyperparameter priors and estimate best performance at each budget through bootstrap resampling.

    Source: §2 · p. 3

  • Results

    Across nine diverse tasks, results overall favor Adam; at low budgets, tuning only Adam’s learning rate is likely to perform very well, while SGD variants sometimes perform best but are tedious to tune.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    Including hyperparameter-search cost in optimizer benchmarks can show practical ease of use and help practitioners choose an optimizer for their budget scenario.

    Source: §7 · p. 7

    The study does not evaluate Bayesian HPO because of its computational cost, and its univariate priors do not capture complex relationships between hyperparameters.

    Source: §6 · p. 7

Abstract

from arXiv · show

The performance of optimizers, particularly in deep learning, depends considerably on their chosen hyperparameter configuration. The efficacy of optimizers is often studied under near-optimal problem-specific hyperparameters, and finding these settings may be prohibitively costly for practitioners. In this work, we argue that a fair assessment of optimizers' performance must take the computational cost of hyperparameter tuning into account, i.e., how easy it is to find good hyperparameter configurations using an automatic hyperparameter search. Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, our results indicate that Adam is the most practical solution, particularly in low-budget scenarios.

1. Introduction

Optimizer performance depends on hyperparameter choices, yet benchmarking often emphasizes best-case performance rather than the resources needed to find good settings. The paper argues for accounting for tuning cost and reports that its nine-task comparison overall favors Adam.

1. Introduction

  • Optimizer performance depends strongly on hyperparameter values, and reported model advances have sometimes been explained by better tuning.Prior calls for fixed-budget automatic hyperparameter optimization aim to reduce bias from manual tuning.Source: §1 · p. 1
  • At large search budgets, tuning more hyperparameters becomes more useful, but tuning only Adam’s learning rate is the most reliable option across the nine tasks.The figure describes the probability of finding the best-performing configuration for a randomly sampled benchmark task.Source: Figure 1 · p. 1
  • Fair optimizer benchmarks should account for the tuning effort required to obtain good performance, not just peak performance.The authors argue that good performance with relatively little tuning can be more useful than a peak reached only after extensive tuning.Source: §1 · pp. 1–2
  • Earlier optimizer studies used widely varying experimental settings, while dataset-specific search spaces and reporting only best results leave low-budget performance unclear.The paper asks how quickly automatic search can find satisfactory settings for an optimizer.Source: §1 · p. 2, Table 1 · p. 2
  • Across nine diverse tasks, the overall results favor Adam, although a well-tuned SGD variant reaches top performance in some cases.The benchmark includes vision, natural language processing, regression, and classification tasks.Source: §1 · p. 2

2. The Need to Incorporate Hyperparameter Optimization into Benchmarking

Optimizer comparisons need to measure both how readily good hyperparameter settings can be found and the optimizer’s absolute performance. The proposed protocol incorporates search effort by evaluating performance at a given random-search budget.

2. The Need to Incorporate Hyperparameter Optimization into Benchmarking

  • A useful comparison must account for both the difficulty of finding good hyperparameters and the optimizer’s absolute performance.Considering either property alone can mis-rank optimizers: an easy-to-tune optimizer can perform poorly, while a high-performing optimizer can have a hard-to-find optimum.Source: §2 · p. 2
  • The authors define an optimizer as its update rule together with a probability distribution over its permissible hyperparameter values.This formulation incorporates prior knowledge about reasonable settings into benchmarking.Source: §2 · pp. 2–3
  • The protocol uses random search from optimizer-specific prior distributions and estimates performance after a fixed number of trials by resampling a precomputed library.Repeated bootstrap resampling estimates the distribution after budget k; mean and variance can also be computed exactly in closed form.Source: §2 · p. 3
  • The protocol avoids arbitrary dataset-specific search-space choices, characterizes performance at low and high budgets, and helps practitioners choose for their budget scenario.Low-budget performance can reveal a broad high-performing region even when another optimizer has a better attainable optimum.Source: §2 · p. 3
  • The bootstrap protocol does not apply when trials are dependent, and alternative HPO methods may introduce additional hyperparameters that can bias comparisons.The authors note that other prior-based methods, such as Bayesian approaches, could in principle be used.Source: §2 · p. 3
  • The protocol measures budget in random-search trials rather than computation time, though appendix experiments report that its conclusions remain valid using time as the budget.The trial-based choice preserves the independence assumption used for the closed-form estimates.Source: §2 · p. 4

3. Related Work

Prior optimizer studies often compare peak performance or use manually selected search spaces, limiting what they establish about tuning effort. Related work also proposes budget-aware reporting and finds evidence consistent with Adam being easier to tune.

3. Related Work

  • Wilson et al. (2017) and Shah et al. (2018) compare best configurations from manually selected grids, so they cannot quantify the effort needed to find those settings.Wilson et al. (2017) conclude that SGD and Adam require similar grid-search effort, but their grids are dataset-dependent and tunability is not clearly defined.Source: §3 · p. 4
  • Probst et al. (2019) measure performance gain over a reference configuration, but that measure ignores absolute performance and cannot distinguish optimizers B and D.Mantovani et al. (2018) instead count cases where tuned decision-tree settings outperform defaults.Source: §3 · p. 4
  • Choi et al. (2019) study optimizer hierarchies and tuning protocols, but focus on best attainable performance with manually chosen per-dataset search spaces.Their setup does not measure how easily the settings for a more general optimizer can be found.Source: §3 · p. 4
  • Dodge et al. (2019) recommend expected validation performance at a given HPO budget, while Asi & Duchi find SGD less robust to learning-rate choices than Adam on CIFAR-10.Metz et al. (2020) propose collating problem-agnostic settings across tasks and report generalization to unseen tasks.Source: §3 · p. 4

4. Optimizers and Their Hyperparameters

The paper treats an optimizer as an update rule paired with a prior over hyperparameters. It compares selected adaptive and non-adaptive variants and calibrates priors from strong settings found by random search.

4. Optimizers and Their Hyperparameters

  • An optimizer is treated as an update rule together with a dataset-independent prior over its hyperparameter search space.In the absence of designer-provided priors, the authors estimate suitable priors for Adam and SGD.Source: §4 · pp. 4–5

4.1. Parameters of the Optimizers

  • The comparison includes SGD, SGD with momentum, Adagrad, Adam, and Adam-LR, with additional SGD weight-decay and learning-rate-decay variants.Adam-LR fixes Adam’s other hyperparameters to the original paper’s recommendations; SGD-MC fixes momentum at 0.9.Source: §4.1 · p. 5
  • Training stops when validation loss plateaus for more than two epochs or the DEEPOBS maximum epoch count is reached.The authors use validation performance rather than searching over training epochs, which would add computational cost.Source: §4.1 · p. 5

4.2. Calibration of Hyperparameter Prior Distributions

  • Prior calibration retains settings within 20% of the best random-search result, fits marginal distributions, and assumes hyperparameters are independent.The learning-rate-decay exponent is excluded from calibration for computational reasons, and calibration costs are excluded from final scores because the priors are reusable across tasks.Source: §4.2 · p. 5

5. Experiments and Results

Across DEEPOBS tasks, results vary with tuning budget and the number of hyperparameters tuned. Adam-LR remains the most probable cross-task best choice, while performance differences vary by task and budget.

5. Experiments and Results

  • The experiments use the DeepOBS benchmark set, with an added IMDB sentiment-classification LSTM, and compare optimizers with different tunable-parameter sets.Table 3 lists models and datasets, and Table 4 lists optimizers and tunable parameters.Source: §5 · p. 5, Table 3 · p. 6, Table 4 · p. 6

5.1. When to Tune More Hyperparameters

  • On classification tasks, Adam-LR and SGD-MCWC average higher than their counterparts through budget 16; at high budgets, rankings vary and Adam’s first quartile remains far below Adam-LR’s.At higher budgets, both outperform their counterparts on CIFAR-100 and FMNIST on average and in the second quartile; Adam also outperforms Adam-LR on IMDB in those measures.Source: §5.1 · p. 6
  • On regression tasks, tuning more hyperparameters helps only SGD-MW on MNIST-VAE; elsewhere it degrades small-budget performance and yields similar performance at high budgets.Source: §5.1 · p. 6

5.2. Summarizing across datasets

  • SGD variants are 20% worse than Adam-LR early, narrowing to 10% at larger budgets, while Adam ends 2–3% behind Adam-LR.The relative-performance curves flatten at high budgets for SGD variants, suggesting they are unlikely to improve further with larger budgets.Source: Figure 3 · p. 6, §5.2 · p. 7
  • Adam-LR remains the most probable optimizer to yield the best results across tuning-budget scenarios, while Adam becomes a better relative option as budget increases.Source: §5.2 · p. 6
  • Cross-dataset summaries compare each optimizer’s expected performance with the task-best after k hyperparameter-search trials, averaging relative performance across test problems.Source: §5.2 · p. 7

6. Discussion

The results favor Adam in low-budget tuning because adaptive methods are easier to tune, while tuning more than Adam’s learning rate becomes more useful as budgets grow. The study also notes limits in its HPO procedure and calls for optimizer designers to provide sampling priors.

6. Discussion

  • In low-budget tuning, Adam is easier to tune than non-adaptive methods and often performs close to their best configurations.Tuning only Adam’s learning rate is described as a good choice, though it does not guarantee the best possible performance.Source: §6 · p. 7
  • As tuning budgets grow, tuning Adam’s additional hyperparameters becomes more useful, but Adam-LR remains the safer choice because full tuning can have high variance.Adam approaches Adam-LR at large budgets; tuning all hyperparameters can also yield better performance.Source: §6 · p. 7
  • The study does not test Bayesian HPO because of its computational cost, and univariate hyperparameter priors do not account for possible parameter relationships.The authors describe the study as not exhaustive.Source: §6 · p. 7
  • SGDElrW improves lower-budget HPO by exploiting historically successful (γ, µ) pairs, but the literature lacks probabilistic models incorporating their causal relationships.The authors attribute the lower-budget improvement to exploiting these historically successful pairs.Source: §6 · p. 7
  • The authors recommend that optimizer designers provide priors for sampling hyperparameters, alongside studying efficacy and convergence.They identify this as a component of an optimizer’s practical value.Source: §6 · p. 7

7. Conclusion

The conclusion proposes including hyperparameter optimization in optimizer benchmarks to show both peak performance and ease of use in practical scenarios. Figure 4 plots five optimizers across different hyperparameter-search budgets.

7. Conclusion

  • Optimizer benchmarks should include the hyperparameter-optimization process, not only peak performance.The authors argue that this also showcases ease of use in practical scenarios.Source: §7 · p. 7
  • A more holistic assessment includes the cost of hyperparameter search when evaluating optimizer performance.The authors hope future optimizer studies will adopt this perspective.Source: §7 · p. 7
  • Figure 4 plots Adam-LR, Adam, SGD-MCWC, SGD-MW, and SGD-MCD across hyperparameter-search budgets.Some plots are truncated for readability.Source: Figure 4 · p. 8

A. Architectures of the Models Used in Experiments

The experiments add a word-level LSTM and the IMDb sentiment-classification dataset to the architectures and tasks considered. The LSTM architecture is specified in Table 5.

A. Architectures of the Models Used in Experiments

B. Performance Analysis

Across problem-specific HPO runs, Adam-LR is often the best option at a given budget, though SGD variants lead on two problems at later stages. The appendix also provides full optimizer performance plots and extends the probability analysis to each problem.

B. Performance Analysis

C. How Likely Are We to Find Good Configurations?

  • The per-problem plots extend the problem-agnostic analysis of the chance of finding an optimal hyperparameter setting.The analysis covers each problem considered in Section 5.Source: C. How Likely Are We to Find Good… · p. 11
  • Adam-LR is the best option for four problems across nearly all budgets, while SGD variants become better options for CIFAR-100 and Char-RNN at later HPO stages.The probability of finding the best configuration is computed from repeated HPO runs at each budget.Source: C. How Likely Are We to Find Good… · pp. 11–12

D. Computing the Expected Maximum of Random Samples

The section estimates the expected maximum and variance of randomly sampled optimizer performances, then proposes aggregating incumbent performance across tuning budgets. Its CPE weighting emphasizes early search performance, where Adam-LR is reported as most competitive across tasks.

D. Computing the Expected Maximum of Random Samples

  • For N sampled values, the expected maximum of a random subset of size S is computed from the distribution of that maximum.The section also describes computing its variance for discrete distributions and using the estimates to assess Random Search performance at a given budget.Source: D. Computing the Expected Maximum of… · p. 12

E. Aggregating the Performance of Incumbents

  • The proposed aggregation function combines incumbent performances into a comparable scalar for ranking optimizers while accounting for ease of use.It is intended to complement the existing peak-performance metric in benchmarking software such as DEEPOBS.Source: E. Aggregating the Performance of… · p. 12, §E.1 · p. 13

E.1. Aggregating Function

  • The ω-tunability function is a convex combination of incumbent performances, with positive weights summing to one.The weights can emphasize peak performance, one-shot performance, a particular budget, or earlier tuning stages.Source: §E.1 · p. 13
  • Adam-LR fares best across tasks under Cumulative Performance-Early (CPE), and is competitive even when it is not the best-performing optimizer.The authors recommend CPE for benchmarking because it emphasizes ease of tuning alongside peak performance.Source: §E.1 · p. 13, Table 6 · p. 13

F. Results for Computation Time Budgets

The section tests whether conclusions change when HPO budgets are measured in computation time rather than trial counts, and examines optimizer performance across learning-rate values. The time-budget results show nearly the same pattern, while learning-rate surface widths help explain low-budget comparisons.

F. Results for Computation Time Budgets

  • The computation-time comparison simulates 1,000 HPO runs per test problem and averages their best performance at each budget interval.The maximum budget is set from the minimum number of update steps any optimizer needed to finish 100 trials.Source: F. Results for Computation Time Budgets · p. 13
  • Using computation time rather than hyperparameter-trial count as the budget unit leaves the conclusions unchanged, with nearly the same pattern as the trial-budget results.The time-budget simulation accounts for differing numbers of update steps needed to complete trials.Source: F. Results for Computation Time Budgets · p. 13

G. Plotting Hyperparameter Surfaces

  • On MNIST VAE, FMNIST VAE, and Tolstoi-Char-RNN, Adam-LR performs near its optimum across a wider learning-rate range than SGD-MCWC, yielding better expected performance at budgets k = 1 and 4.On CIFAR10, the learning-rate maxima have similar widths, and SGD-MCWC's slightly higher maximum gives it better high-budget performance.Source: G. Plotting Hyperparameter Surfaces · p. 14

H. Interplay between momentum and learning rate

The section tests an SGD variant that samples learning rate and effective learning rate independently, deriving momentum to account for their interplay. SGD-LReff improves over SGD-MW in low-budget settings but falls increasingly behind at larger budgets, while SGD variants remain below Adam-LR across budgets.

H. Interplay between momentum and learning rate

  • SGD-LReff improves over SGD-MW in low-budget regimes, particularly on classification tasks, but performs increasingly worse at larger budgets; SGD variants remain substantially below Adam-LR across budgets.The authors attribute the low-budget gain to exploiting historically successful learning-rate and momentum pairs, and the larger-budget decline to SGD-MW's greater chance of exploring new configurations.Source: H. Interplay between momentum and… · p. 14
  • SGD-LReff samples learning rate and effective learning rate independently from calibrated lognormal priors, then derives momentum to account for their interplay.The effective learning rate combines learning rate and momentum and quantifies a minibatch's contribution to overall training.Source: H. Interplay between momentum and… · p. 14
Loading 1910.11758v4…