Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates

Leslie N. Smith; Nicholay Topin

Source-linked AI summary

Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates

Leslie N. Smith, Nicholay Topin

arXiv:1708.07120v3Aug 23, 2017cs.LGcs.CVcs.NEstat.ML

TL;DR

The paper addresses the open question of why SGD works well by studying super-convergence, where large learning rates and cyclical schedules enable much faster neural-network training. It combines one-cycle training, learning-rate range tests, and a Hessian-free learning-rate estimate, and reports higher accuracy in far fewer iterations across multiple settings, with larger gains when labeled data is limited.

Problem
Understanding why stochastic gradient descent works well and why deep-network solutions generalize remains an open research question.
Method
The paper combines cyclical learning rates, learning-rate range tests, a one-cycle policy, and a simplified Hessian-free method for estimating learning rates.
Results
Super-convergence trains networks in an order of magnitude fewer iterations and to higher final test accuracy than piecewise-constant training across multiple datasets and architectures.
Takeaways & Limitations
Large learning rates regularize training, so reducing other regularization can preserve balance and make super-convergence effective, especially with less labeled data.
Takeaways & Limitations
The paper does not fully evaluate its Hessian-free method for estimating optimal adaptive learning rates, leaving that assessment for future work.

Abstract

from arXiv · show

In this paper, we describe a phenomenon, which we named "super-convergence", where neural networks can be trained an order of magnitude faster than with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with one learning rate cycle and a large maximum learning rate. A primary insight that allows super-convergence training is that large learning rates regularize the training, hence requiring a reduction of all other forms of regularization in order to preserve an optimal regularization balance. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. Experiments demonstrate super-convergence for Cifar-10/100, MNIST and Imagenet datasets, and resnet, wide-resnet, densenet, and inception architectures. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. The architectures and code to replicate the figures in this paper are available at github.com/lnsmith54/super-convergence. See http://www.fast.ai/2018/04/30/dawnbench-fastai/ for an application of super-convergence to win the DAWNBench challenge (see https://dawn.cs.stanford.edu/benchmark/).

1 Introduction

The paper introduces super-convergence, showing that carefully selected very large learning rates can accelerate neural-network training while improving accuracy. It argues that this phenomenon also informs questions about SGD and generalization.

Motivation: Super-convergence uses very large learning rates with cyclical learning rates to speed training by as much as an order of magnitude.The authors present this as both a practical training method and evidence relevant to understanding SGD and generalization.
Empirical example: 92.4% test accuracy was reached after 10,000 iterations, versus 91.2% after approximately 80,000 iterations with piecewise-constant training on Cifar-10.Both comparisons use Resnet-56.
Empirical example: 92.1% final test accuracy exceeded typical training's 91.2% after only 6,000 iterations.As total iterations increased from 2,000 to 20,000, final accuracy improved from 89.7% to 92.7%.
Contributions: The paper identifies large-learning-rate regularization, an optimal-learning-rate estimate, and stronger gains with fewer labeled examples as central contributions.It also reports systematic investigation of a training methodology with improved speed and performance.

2 Background

The background frames SGD's effectiveness and generalization as open research questions, contrasting standard piecewise-constant schedules with broader discussions of optimization and noise. It also situates the paper alongside learning-rate range tests.

Standard training: Standard training uses a global learning rate for many epochs, then repeatedly decreases it by a factor of 0.1 after test accuracy plateaus.This is the paper's definition of typical, standard, or piecewise-constant training.
Related work: The paper's use of large learning rates contrasts with literature suggesting a maximum learning-rate value.It uses a simplification of a second-order Hessian-free method in this context.
Open questions: Understanding why SGD works well and why deep-network solutions generalize remains an open research area.The paper connects this question to research on loss-function topology, adaptive learning rates, and SGD noise.
Learning-rate tests: Learning-rate range tests are presented as part of the methodological background for selecting learning-rate schedules.The supplied background identifies Figure 2 as comparing learning-rate range-test results.

3 Super-convergence

Super-convergence combines cyclical learning rates with a learning-rate range test and a one-cycle schedule using unusually large maximum learning rates. The paper emphasizes balancing this regularization by reducing other regularizers.

Cyclical learning rates: Cyclical learning rates specify minimum and maximum boundaries plus a stepsize, with one increasing and one decreasing step per cycle.The paper describes linear changes as the simplest CLR policy and relates its philosophy to curriculum learning and simulated annealing.
Learning-rate range test: The learning-rate range test increases the rate linearly from zero or a small value to identify a peak accuracy and set the maximum CLR bound.The network initially converges, then accuracy decreases when the learning rate becomes too large.
Learning-rate range test: Learning rates up to 3.0 remained consistently effective for Cifar-10 Resnet-56, an order of magnitude above typical values.This unusual range motivated experiments with much higher learning rates and was considered indicative of super-convergence potential.
One-cycle policy: The 1cycle policy uses one cycle shorter than the total training duration, followed by a learning rate several orders of magnitude below the initial rate.The authors report that this modification improved accuracy.
Regularization balance: Large learning rates act as regularization, so weight decay, dropout, batch size, and other regularizers must be balanced for each dataset and architecture.Reducing other regularization while using very large learning rates makes training more efficient.

4 Estimating optimal learning rates

The paper estimates useful learning rates without forming the full Hessian by tracking curvature along the steepest-descent direction and rewriting the estimate using sequential weight updates.

Gradient descent: Gradient descent updates network weights using the negative gradient scaled by the learning rate.The learning rate is the step size used to update parameters.
Hessian-free approximation: The Hessian-free method avoids computing the full Hessian, which has Ω(N^2) elements, by estimating curvature along SGD’s steepest-descent direction.This reduces the curvature information needed for learning-rate estimation to the direction SGD traverses.
Adaptive learning-rate estimate: The method converts the Hessian-based estimate into an adaptive learning rate for each neuron using weight differences from three sequential iterations.The derivation rewrites the estimate in terms of sequential parameter updates.
Empirical estimate: 2–6 is the estimated optimal learning-rate range from moving-averaged calculations over constant 0.1 and CLR 0.1–3 runs.The calculations used Equation 8 on weights from every iteration, with CLR stepsize 5,000 iterations and moving-average coefficient α = 0.1.
Scope: The paper uses this approximation illustratively rather than fully evaluating its effectiveness for estimating optimal adaptive learning rates.A comprehensive assessment is left for future work.

5 Experiments and analysis

Experiments show super-convergence across datasets and architectures, with large learning rates and 1cycle schedules enabling faster training and often higher accuracy. Its benefits increase with limited data, while reduced weight decay helps support large learning rates on ImageNet.

Resnet depth: Super-convergence gains were larger for Resnet-20 than Resnet-110: 90.4% versus 88.6%, compared with 92.1% versus 91.0%.The reported comparison used CLR against a piecewise-constant learning-rate schedule.
Batch size: Large batch sizes were more effective for super-convergence, with a small performance improvement and approximately equivalent generalization gaps across batch sizes.The experiments associate large batches with the ability to use large learning rates.
Other architectures: Super-convergence also accelerated wide-ResNet and DenseNet training: wide32 reached 91.9% in 100 epochs versus 90.3% in 800, while DenseNet reached 92.8% in 150 versus 92.7% in 400.The reported wide-ResNet result used a 1cycle schedule with learning-rate bounds from 0.1 to 1.0; DenseNet used bounds from 0.1 to 4.0 with cyclical momentum.
ImageNet: On ImageNet, reducing weight decay enabled 1cycle training with large learning rates, producing 67.6% top-1 accuracy for ResNet-50 and 74.0% for Inception-ResNet-v2 in 20 epochs.The standard ResNet-50 result was 63.7% after the conventional training regime, while the standard Inception comparison reached 67.6% after 100 epochs.

6 Conclusion

The paper presents super-convergence as training networks in far fewer iterations while achieving higher final test accuracy than piecewise-constant training. It suggests that studying this phenomenon may improve understanding of deep networks, optimization, and loss landscapes.

Super-convergence trains networks in an order of magnitude fewer iterations and reaches higher final test accuracy than piecewise-constant training.
Its gains increase as the available labeled training data becomes more limited.
Super-convergence is demonstrated across a variety of datasets and architectures when large-learning-rate regularization is balanced by reducing other regularization.
Deeper study of super-convergence may improve understanding of deep networks, their optimization, and their loss function landscape.

A Supplemental material

The appendix supplements the main paper with intuitive explanations, literature connections, and experimental details that could not fit within the paper's space limits.

The appendix provides an intuitive explanation for super-convergence, discussion connecting evidence to prior literature, and experiment details.

A.1 Intuitive explanation for super-convergence

The intuitive explanation describes super-convergence as traversing a flat loss valley with a learning-rate cycle that starts and ends small but is large in the middle.

Early training uses a small learning rate so optimization progresses in an appropriate direction.
During traversal of the flat valley, a large learning rate enables faster progress.
The learning rate is reduced again at the end so training can settle into a local minimum.

A.2 Large learning rate regularization

The paper presents large learning rates as a source of training regularization, supported by decreasing test loss or error as learning rates increase.

Increasing the learning rate from approximately 0.2 to 2.0 raises training loss while lowering test loss for Cifar-10 with Resnet-56.
Because regularization is intended to reduce generalization error, the paper treats large learning rates as regularizing.
Figure 7 visualizes loss-function topology, with the z axis representing the loss potential.
Figure 8 reports decreasing generalization error as the learning rate increases from 0.3 to 1.5.

A.3 Relationship of super-convergence to SGD and generalization

Super-convergence contributes empirical evidence to debates about SGD noise, generalization, and the geometry of minima. Its results motivate a broader framework for understanding interacting sources of regularization and noise.

Super-convergence adds evidence to ongoing efforts to explain why SGD solutions generalize well.The paper supports some existing theories, contradicts others, and identifies a need for further theoretical understanding.
The reported alignment with wide, flat minima is qualified because small learning rates remain necessary at the end of training.The authors state that this implies the minimum reached locally is narrow and requires reconciliation.
The method studies time-varying high noise during training rather than treating SGD noise as fixed.The noise is described as proportional to learning rate, gradient variance, and inversely related to batch size.
The authors report a curriculum-like schedule: begin with little noise, increase it to a peak, then reduce it near the end.They contrast this schedule with a proposal to warm-start using small batches and then increase batch size.
The results suggest that very large learning-rate regularization can permit much shorter training lengths.This contrasts with work treating longer training as a form of regularization that improves generalization.
The paper calls for a unified framework covering architectural noise, hyper-parameter noise, gradient noise, weight noise, and input diversity.Such a framework would help reconcile conflicting claims about the value of different regularization sources.

A.4 Hardware, software, and architectures

The supplemental experiments use Caffe and multi-GPU systems to evaluate several residual and convolutional architectures across image datasets. ResNet-56 is specified as a staged residual architecture with downsampling between stages.

Experiments were run with Caffe, CUDA 7.0, CuDNN, and multi-GPU systems including Titan Black and Tesla P100 clusters.Most experiments used 64 nodes with eight Titan Black GPUs per node; others used 32 Power8 nodes with four Tesla P100 GPUs per node.
ResNet-56 repeats residual blocks within three stages and changes block structure between stages to reduce spatial dimensions.Batch Normalization layers were followed by scaling layers to achieve true batch-normalization behavior.
The evaluated datasets include CIFAR-10, CIFAR-100, MNIST, and ImageNet.ImageNet contains 1.24 million training images across 1,000 classes in the described setup.
The architectures include ResNet, Wide ResNet, DenseNet, and Inception-ResNet variants.The Wide ResNet increases channels by a factor of four, while the described DenseNet uses 40 layers.

A.5 Hyper-parameters and adaptive learning rate methods

The experiments examine batch normalization, learning-rate ranges, adaptive optimizers, dropout, batch size, momentum, and weight decay. They consistently emphasize that super-convergence depends on suitable hyper-parameter settings and reduced competing regularization.

A batch-normalization moving-average fraction of 0.999 updates too slowly for super-convergence’s short training runs.The authors found that faster updating of accumulated global statistics was needed when training very quickly.
A maximum learning rate of approximately 3 performed well across the tested CLR range bounds from 1.0 to 3.5.These experiments used CLR range tests to assess suitable maximum learning rates.
Adaptive methods in piecewise-constant training did not discover large learning rates or produce super-convergence-like behavior.The tested methods were Nesterov momentum, AdaDelta, AdaGrad, and Adam.
With the 1cycle policy, Nesterov, AdaDelta, and AdaGrad allowed super-convergence, whereas Adam did not in these experiments.For Nesterov, super-convergence reached 92.1% final test accuracy after 10,000 iterations.
68.6% versus 59.8% final test accuracy produced an 8.8% improvement for super-convergence over piecewise-constant training on CIFAR-100.The comparison used ResNet-56 and is reported for the supplemental CIFAR-100 experiment.
Dropout with ratio 0.2 produced a small improvement over super-convergence without dropout.The comparison covered both learning-rate range tests and 10,000-iteration training results.
Larger mini-batch sizes produced better final accuracies in the reported super-convergence experiments.Most main-paper results used a total mini-batch size of 1,000.
Weight decay of 10^-3 prevented super-convergence, while smaller values did not; 10^-4 performed well.Momentum changes from 0.8 to 0.95 caused only small result changes, with 0.9 somewhat better.