PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepOBS: A Deep Learning Optimizer Benchmark Suite

Frank Schneider, Lukas Balles, Philipp Hennig

arXiv:1903.05499v1cs.LGstat.ML

TL;DR

Deep learning optimizer research lacks a generally agreed-upon protocol for quantitative, reproducible evaluation, complicated by stochasticity, tunability, and generalization. The paper proposes benchmarking practices and DeepOBS, an open-source package that automates realistic tests and supplies baselines for fair comparisons. Its baseline analyses show that no optimizer is best on every problem, with MOMENTUM notably outperforming the others on CIFAR-100.

  • Problem

    Deep learning lacks a generally accepted protocol for quantitatively and reproducibly evaluating optimization strategies across stochasticity, tunability, and generalization.

  • Method

    The paper defines benchmarking practices and implements DeepOBS, an extensible open-source TensorFlow package that automates realistic optimizer tests and provides tuned baselines.

  • Results

    No optimizer is optimal across all test problems; ADAM compares favorably on most, while MOMENTUM is significantly better on CIFAR-100.

  • Takeaways & Limitations

    DeepOBS provides a common, reproducible basis for comparing deep-learning optimizers and helps practitioners assess algorithm choices.

  • Takeaways & Limitations

    ImageNet is not included in the automatic procedure because registration and dataset size make it impractical for many users.

Abstract

from arXiv · show

Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning. We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization. As the primary contribution, we present DeepOBS, a Python package of deep learning optimization benchmarks. The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking. The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on ImageNet or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10. The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments. It comes with output back-ends that directly produce LaTeX code for inclusion in academic publications. It supports TensorFlow and is available open source.

1 INTRODUCTION

Deep learning optimization lacks a commonly accepted, reproducible evaluation protocol despite many proposed methods. DeepOBS addresses this gap with standardized practices, realistic benchmarks, automation, and tuned baselines for fair comparisons.

  • 1 INTRODUCTION: DeepOBS proposes an extensible open-source benchmark suite that automates reproducible evaluation of optimization methods on deep learning architectures.The TensorFlow package provides ready-to-use problems, dataset and model setup, and publication-ready LaTeX output.
  • 1 INTRODUCTION: The benchmark protocol addresses stochasticity, tunability, and generalization by recommending multiple runs, reporting key metrics, and evaluating competition fairly.It recommends final performance, speed, and tunability as key indicators, and reporting training and test loss and accuracy.
  • 1 INTRODUCTION: The suite includes realistic tasks spanning application domains, from simple stochastic functions to contemporary deep networks and language or image problems.The collection is extensible and includes examples such as ImageNet-scale image classification and natural language processing.
  • 1 INTRODUCTION: DeepOBS provides four small and four large core problems so researchers can iterate quickly before testing algorithms on more demanding settings.The suite also supplies realistic baseline results for popular optimizers, reducing repeated computational effort and comparison bias.
  • 1.1 RELATED WORKS: Deep learning optimization research lacks a shared benchmark protocol, while existing benchmarks primarily target hardware, frameworks, or holistic systems rather than optimizers.Unit tests cover small-scale problems, whereas holistic benchmarks combine architecture, hardware, and training procedure.

2 BENCHMARKING DEEP LEARNING OPTIMIZERS

The proposed protocol treats optimizer evaluation as a noisy, multi-objective measurement problem involving stochasticity, generalization, speed, and hyperparameter tunability. It prescribes repeated evaluation, multiple metrics, practical runtime accounting, and comparable tuning budgets.

  • 2.1 STOCHASTICITY: Repeated runs with different random seeds are needed because mini-batch sampling and initialization make optimizer performance noisy.Reporting means and standard deviations exposes variability and helps assess whether performance differences are statistically meaningful.
  • 2.2 CHOICE OF PERFORMANCE METRIC: Training loss alone is insufficient: optimizers should be assessed using training loss, test loss, training accuracy, and test accuracy.A train evaluation set reduces mini-batch noise and supports evaluation-mode measurements such as disabling dropout.
  • 2.3 MEASURING SPEED: Speed is measured as the time to reach an acceptable problem-specific performance defined from tuned SGD, MOMENTUM, and ADAM baselines.Wall-clock time is practical but hardware-dependent, so relative per-iteration cost can complement iteration-based reporting.
  • 2.4 HYPERPARAMETER TUNING: Tunability is characterized by reporting the best hyperparameters for each problem and the relative performance of settings explored during tuning.Using multiple test problems reveals how widely the best settings vary without requiring a full sensitivity analysis.
  • 2.4 HYPERPARAMETER TUNING: Fair comparisons require equal computational budgets across methods, especially when optimizers expose different numbers of hyperparameters.Fixed, diverse test problems reduce cherry-picking and make it harder to overfit an optimizer to familiar datasets or architectures.

3 BENCHMARK SUITE OVERVIEW

DEEPOBS provides an end-to-end, reproducible workflow for benchmarking deep learning optimizers, from data and models through training, baselines, runtime estimation, and standardized visualization.

  • 3 BENCHMARK SUITE OVERVIEW: DEEPOBS provides the full stack for rapid, reliable, and reproducible optimizer benchmarking, including data loading, models, runners, baselines, and visualization.Its baselines currently cover SGD, MOMENTUM, and ADAM, while the visualization script produces LaTeX output.
  • 3 BENCHMARK SUITE OVERVIEW: Results obtained with the same MAJOR.MINOR DEEPOBS version will be directly comparable, while PATCH updates will not affect benchmark results.MAJOR versions retain the same problems even when benchmark-set selection changes between major releases.
  • 3.1 DATA LOADING: The data-loading module downloads, preprocesses, splits, shuffles, batches, and augments datasets, and can support models not yet included in DEEPOBS.Excluding IMAGENET, automatically downloaded datasets require less than one GB of disk space.
  • 3.2 MODELS: The suite spans diverse optimization problems, including image classification, natural language processing, generative models, toy functions, and an ill-conditioned quadratic problem.Problems range from simple two-dimensional tests to contemporary deep neural networks and are intended to expand as hardware and research progress.
  • 3.3 RUNNERS: Runners handle training and log optimizer statistics, requiring only hyperparameters for optimizers that follow TensorFlow’s standard optimizer API.Templates and an example support more sophisticated optimizers outside that API.
  • 3.4 BASELINES: DEEPOBS supplies realistic baselines for SGD, MOMENTUM, and ADAM, enabling comparisons with competing algorithms without repeating costly baseline experiments.Baselines currently cover all problems in the small and large benchmark sets; additional optimizers can be added when authors provide code.
  • 3.5 ESTIMATE RUNTIME: The runtime estimator compares a new optimizer with SGD using wall-clock ratios, with a default five-run, three-epoch MNIST measurement that can be adapted for amortized startup costs.The setting can be changed when an optimizer has a high initial cost that amortizes over many epochs.
  • 3.6 VISUALIZATIONS: The visualization module standardizes results and generates pgfplots-based LaTeX files containing learning curves for the proposed optimizer and relevant baselines.This reduces preparation overhead while supporting dense presentation of benchmark results.

4 INSIGHTS FROM THE BASELINES

DEEPOBS evaluates SGD, MOMENTUM, and ADAM across eight small and large test problems using tuned learning rates and repeated random-seed runs. The results show problem-dependent optimizer rankings, non-universal generalization behavior, and somewhat easier tuning for ADAM.

  • 4 INSIGHTS FROM THE BASELINES: The baselines tune each optimizer and problem separately over 36 logarithmically spaced learning rates, then repeat the selected setting ten times with different random seeds.The evaluation covers four small and four large benchmark problems.
  • 4 INSIGHTS FROM THE BASELINES: MOMENTUM is significantly better than SGD and ADAM on CIFAR-100, demonstrating that no optimizer is best across all test problems.ADAM compares favorably on most problems, but other optimizers can be considerably better in particular cases.
  • 4 INSIGHTS FROM THE BASELINES: Optimizer rankings differ between training and test loss, and no optimizer universally generalizes better or maps loss to accuracy across problems.The reported relationships among the four learning metrics are non-trivial, including on P6, P7, P3, and P6 respectively.
  • 4 INSIGHTS FROM THE BASELINES: ADAM is somewhat easier to tune: its optimal learning rates are usually 10^-4 to 10^-3, whereas SGD and MOMENTUM show a slightly wider spread.The three optimizers do not differ significantly in learning-rate sensitivity overall.

5 CONCLUSION

DEEPOBS provides open-source standards and implementation for more reproducible and fair deep-learning optimizer evaluation. Its benchmark reports performance, speed, and tuneability for baseline optimizers across eight test problems.

  • DEEPOBS provides an open-source implementation of benchmarking standards intended to make deep-learning optimizer evaluation simpler, more reproducible, and fairer.
  • The package establishes common ground for comparing methods, aiming to speed optimizer development and aid practitioners’ algorithm choices.
  • Table 2 evaluates SGD, MOMENTUM, and ADAM across eight problems using performance, speed, and tuneability measures.Performance uses test accuracy when available or test loss otherwise; speed counts iterations to convergence, with averages over ten runs.

A EXPERIMENTAL SETUP

The benchmark suite contains eight test problems spanning stochastic quadratics, autoencoders, convolutional classifiers, a wide residual network, and character-level language modeling. Problems vary in datasets, architectures, batch sizes, and training durations.

  • Eight test problems comprise the small and large benchmark sets, covering synthetic optimization, image modeling and classification, and character-level language modeling.
  • P1 Quadratic Deep uses a 100-dimensional stochastic quadratic with an ill-conditioned eigenspectrum, trained for 100 epochs with batch size 128.
  • The small-set image problems include MNIST and FASHION-MNIST variational autoencoders plus FASHION-MNIST and CIFAR-10 convolutional classifiers.
  • The larger image problems use All-CNN-C on CIFAR-100 and Wide ResNet-16-4 on Street View House Numbers for image classification.
  • P8 Tolstoi uses a two-layer 128-unit LSTM for character-level language modeling on War and Peace, with sequence length 50, batch size 50, and 200 training epochs.
Loading 1903.05499v1…