PaperlayerView exact revision on arXiv

Source-linked AI summary

SGDR: Stochastic Gradient Descent with Warm Restarts

Ilya Loshchilov, Frank Hutter

arXiv:1608.03983v5cs.LGcs.NEmath.OC

TL;DR

Training deep neural networks is computationally costly, and learning-rate scheduling remains a central difficulty. The paper proposes stochastic gradient descent with warm restarts, achieving competitive CIFAR results two to four times faster and reaching 3.14% error on CIFAR-10 and 16.21% on CIFAR-100.

  • Problem

    Training large-dataset deep neural networks is computationally costly, while learning-rate scheduling is a central training difficulty.

  • Method

    SGDR periodically raises the learning rate while retaining the current solution, then schedules it downward to simulate warm restarts.

  • Results

    SGDR achieved competitive CIFAR-10 and CIFAR-100 results roughly two to four times faster, reaching 3.14% and 16.21% error, respectively.

  • Takeaways & Limitations

    The proposed learning-rate schedule is competitive with default scheduling even without restarts and requires only the initial learning rate and total epochs.

  • Takeaways & Limitations

    Further improvement may require longer runs and hyperparameter tuning, and the results should be tested with alternative network structures.

Abstract

from arXiv · show

Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR

1 INTRODUCTION

Training deep neural networks on large datasets is a major computational bottleneck, while learning-rate scheduling remains a central difficulty in stochastic-gradient training. The paper proposes periodically restarting SGD’s learning rate and reports comparable or better results using substantially fewer epochs.

  • Training deep neural networks on large datasets often takes several days even on high-performance GPUs, making speedups valuable.
  • DNN training formulates parameter optimization as minimizing f: IRn → IR using stochastic gradient information from relatively small data batches.
  • Learning-rate scheduling and L2 weight-decay regularization are identified as the main training difficulties, with schedules commonly reducing a constant rate at fixed intervals.
  • The proposed method periodically simulates SGD warm restarts by reinitializing the learning rate and scheduling it to decrease during each restart.Four different instantiations of the schedule are visualized in Figure 1.
  • 2× to 4× fewer epochs are required by SGD with warm restarts to achieve comparable or better results than currently used learning-rate schedules.

2 RELATED WORK

Related work spans gradient-free niching for multimodal optimization, gradient-based restart schemes for convergence acceleration, and learning-rate methods related to SGDR. Prior approaches differ in scalability, restart triggers, and whether learning rates increase or decrease at restarts.

  • Gradient-free multimodal optimization: Niching-based gradient-free methods cover search spaces with dynamically allocated local-optimizer niches but usually handle only small spaces, such as n < 10.Their suitability depends on the function landscape and function-evaluation budget.
  • Gradient-based restarts: Gradient-based methods such as BFGS can restart for multimodal functions, while gradient information provides an n-fold speedup over gradient-free approaches when n is 10^3–10^9.Warm restarts are generally used to improve convergence rather than address multimodality.
  • Gradient-based restarts: Function and gradient restart schemes trigger when the objective increases or when momentum forms an obtuse angle with the negative gradient, respectively.The gradient scheme resembles Powell’s conjugate-gradient approach.
  • Gradient-based restarts: Simple restart schemes accelerate optimization on smooth functions and can be adapted to accelerate methods such as FISTA on nonsmooth functions.O’Donoghue and Candès reported this result, which follow-up work confirmed.
  • Stochastic subgradient restarts: Yang and Lin’s restarted stochastic subgradient method achieves linear convergence for polyhedral-epigraph objectives, but decreases learning rates geometrically and resets to the previous epoch’s averaged solution.This contrasts with approaches that increase the learning rate to perform restarts.

3 STOCHASTIC GRADIENT DESCENT WITH WARM RESTARTS (SGDR)

SGDR performs warm restarts of stochastic gradient descent by increasing the learning rate while retaining the previous solution, then decays the rate within each run using cosine annealing. To improve anytime performance, it expands restart intervals and selects a run-end incumbent without requiring a validation set.

  • Warm restarts: SGDR restarts SGD after Ti epochs by increasing ηt while retaining the previous xt as the initial solution.The increase controls how much previously acquired information, such as momentum, is retained.
  • Learning-rate schedule: Within each run, SGDR decays the learning rate batch-by-batch using cosine annealing.Tcur measures epochs since the last restart and is updated at each batch, allowing fractional values.
  • Learning-rate schedule: At Tcur = 0, the learning rate is ηi_max, while at Tcur = Ti it reaches ηi_min.The cosine schedule decreases the learning rate across the run.
  • Restart intervals: SGDR can begin with a small Ti and multiply it by Tmult at every restart to improve anytime performance.The paper illustrates T0 = 1, Tmult = 2 and T0 = 10, Tmult = 2.
  • Incumbent recommendation: After the first restart, SGDR recommends the solution at the end of the latest run when ηt = ηi_min rather than always using the latest xt.Keeping ηi_min constant across runs reduces the number of hyperparameters, and the recommendation strategy avoids requiring a separate validation set.

4 EXPERIMENTAL RESULTS

Experiments on CIFAR-10 and CIFAR-100 show that SGDR improves anytime performance over standard learning-rate schedules, while wider networks and snapshot ensembles further reduce test error. Results on EEG and downsampled ImageNet also support the benefits of snapshots and SGDR.

  • CIFAR experimental setup: Experiments train WRN architectures on 32×32 CIFAR-10 and CIFAR-100 images using standard preprocessing and moderate augmentation.Both datasets contain 50,000 training and 10,000 test images, spanning 10 and 100 classes, respectively.
  • CIFAR experimental setup: The reproduced default results match prior WRN-28-10 results, with η0 = 0.05 outperforming η0 = 0.1 in these experiments.The η0 = 0.05 setting is used in later experiments.
  • SGDR learning-rate schedules: SGDR with T0 = 1 or T0 = 10 and Tmult = 2 reaches approximately 4% CIFAR-10 and 20% CIFAR-100 test error 2-4 times faster than the default schedule.The schedule doubles the maximum epoch count after each restart to improve anytime performance.
  • Network width: Widening the network from WRN-28-10 to WRN-28-20 improves test error by about 0.25% on CIFAR-10 and 0.5-1.0% on CIFAR-100.WRN-28-20 requires roughly three-four times more computation, while SGDR’s aggressive schedule reaches good performance faster.
  • Snapshot ensembles: 16 SGDR runs with three snapshots per run achieve 3.14% CIFAR-10 and 16.21% CIFAR-100 test error.Three runs already reach 3.25% and 16.64%, respectively, outperforming Huang et al. (2016a).
  • Additional datasets: SGDR improves the approximately 9% median EEG reference error by 1-2% with one run’s snapshots and by 2-3% using snapshots from both hyperparameter settings.On downsampled ImageNet, SGDR reaches 39.24% top-1 and 17.17% top-5 error, matching AlexNet’s original-image results of 40.7% and 18.2%.

5 DISCUSSION

The discussion emphasizes SGDR’s goal of improving anytime performance, while noting that its aggressive schedule is competitive even without restarts and requires only two hyper-parameters. SGDR also supports cost-free ensemble construction, though no effect related to multi-modality is claimed.

  • Learning-rate schedule: The aggressive learning-rate schedule remains competitive with the default schedule on WRNs trained on CIFAR-10 and CIFAR-100, even without restarts.The example configuration is T0 = 200, Tmult = 1.
  • Learning-rate schedule: The proposed schedule requires only two hyper-parameters: the initial learning rate and the total number of epochs.The discussion identifies these as the only parameters that need to be defined in practice.
  • Anytime performance: SGDR’s anytime performance remains similar when shorter epochs are considered.The paper refers to supplementary Section 8.1 for this result.
  • Purpose and scope: SGDR’s main purpose is improving anytime performance, not addressing multi-modal functions; the authors explicitly make no claim of observing multi-modality effects.The authors distinguish the stated optimization objective from a possible use of restarts for multi-modal functions.
  • Ensembling: Intermediate SGDR models can build efficient ensembles at no cost, making SGDR attractive when ensemble construction is considered.This reproduces a finding by Huang et al. (2016a).

6 CONCLUSION

The conclusion presents SGDR as a simple learning-rate scheduling mechanism that accelerates DNN training and achieves competitive or state-of-the-art results. It also identifies promising evidence from EEG and downsampled ImageNet experiments, alongside directions for evaluating alternative network structures and compression methods.

  • 6 CONCLUSION: SGDR simulates warm restarts by scheduling the learning rate and achieves competitive CIFAR-10 and CIFAR-100 results roughly two to four times faster.The mechanism is presented as a way to accelerate DNN training.
  • 6 CONCLUSION: SGDR achieves new state-of-the-art results mainly by using wider WRNs and ensembles of snapshots from SGDR’s trajectory.The supplied passage introduces these factors as the main contributors to the reported state-of-the-art results.
  • 6 CONCLUSION: Preliminary EEG results improve with more restarts and model snapshots, while downsampled ImageNet results suggest SGDR might reduce the learning-rate-selection problem.These are described as preliminary or suggestive findings rather than definitive conclusions.
  • 6 CONCLUSION: Future work should test SGDR on additional datasets, replace WRNs with more memory-efficient networks, and evaluate whether deep compression reduces time and memory costs.The passage specifically mentions SVHN, ImageNet, MS COCO, alternative network structures, and deep compression methods.

8 SUPPLEMENTARY MATERIAL

The supplementary material reports additional WRN and dataset experiments, including median results, training and test metrics, and ImageNet top-5 error comparisons. It also notes that image flipping doubles examples per epoch, potentially changing the meaning of epoch-indexed hyperparameters.

  • WRN-28-1 supplementary results: Figure 6 reports median results from five runs for the best learning-rate settings tested on WRN-28-1.
  • Data augmentation: Image flipping doubles the number of training examples per epoch, potentially affecting results when hyperparameters depend on epoch index.The augmentation code is inherited from the Lasagne Recipe for ResNets.
  • CIFAR results: Figure 7 presents training cross-entropy plus regularization loss, test loss, and test error for CIFAR-10 and CIFAR-100.
  • Downsampled ImageNet: Figure 8 compares top-5 test errors for SGD with momentum and SGDR on downsampled ImageNet using T0 = 1, Tmult = 2, and initial learning rates 0.050, 0.015, and 0.005.The experiment uses WRN-28-10 and 32 × 32 images from all 1000 classes.
Loading 1608.03983v5…