Source-linked AI summary
Cyclical Learning Rates for Training Neural Networks
Leslie N. Smith
TL;DR
Choosing a suitable learning rate and schedule is difficult because poor values slow convergence or cause divergence. The paper introduces cyclical learning rates, which vary the rate between estimated bounds and use a short linear increase to estimate those bounds. CLR achieves near-optimal classification results, often with fewer iterations, while requiring essentially no additional computation; its full application range and theoretical basis remain unexplored.
Problem
Learning-rate selection requires experiments because rates that are too small slow convergence and rates that are too large cause divergence.
Method
CLR cyclically varies the global learning rate between minimum and maximum bounds, with a short linearly increasing run estimating reasonable bounds.
Results
Near-optimal classification results are achieved, often with fewer iterations, while CLR requires essentially no additional computation.
Takeaways & Limitations
CLR reduces guesswork in setting learning rates and provides practical guidance for when to drop rates and stop training.
Takeaways & Limitations
The work does not explore the full range of applications or provide a theoretical analysis of CLR methods.
Abstract
from arXiv · showhide
It is known that the learning rate is the most important hyper-parameter to tune for training deep neural networks. This paper describes a new method for setting the learning rate, named cyclical learning rates, which practically eliminates the need to experimentally find the best values and schedule for the global learning rates. Instead of monotonically decreasing the learning rate, this method lets the learning rate cyclically vary between reasonable boundary values. Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. This paper also describes a simple way to estimate "reasonable bounds" -- linearly increasing the learning rate of the network for a few epochs. In addition, cyclical learning rates are demonstrated on the CIFAR-10 and CIFAR-100 datasets with ResNets, Stochastic Depth networks, and DenseNets, and the ImageNet dataset with the AlexNet and GoogLeNet architectures. These are practical tools for everyone who trains neural networks.
1. Introduction
Training neural networks requires choosing a learning rate carefully because values that are too small slow convergence and values that are too large cause divergence. The paper proposes cyclical learning rates to reduce this tuning burden while preserving or improving accuracy.
- Too-small learning rates slow convergence, while too-large learning rates cause divergence, requiring experiments over rates and schedules.
- Cyclical learning rates vary the global learning rate within a band instead of keeping it fixed or monotonically decreasing.
- 81.4% baseline accuracy was matched by CLR within 25,000 iterations instead of 70,000 iterations.
- The method provides a way to set global learning rates without numerous experiments and with essentially no additional computation.
2. A surprising phenomenon is demonstrated - allowing
The paper argues that increasing and decreasing the learning rate can benefit training overall, even when increases temporarily hurt performance. It demonstrates CLR across several network architectures and image-classification datasets.
- Increasing the learning rate can temporarily harm performance yet benefit training over the longer term.
- CLR is demonstrated with ResNets, Stochastic Depth networks, and DenseNets on CIFAR-10 and CIFAR-100, plus AlexNet and GoogLeNet on ImageNet.
2. Related work
Related work covers practical guidance and adaptive methods for learning-rate selection. The paper distinguishes CLR from adaptive learning rates while noting compatibility and computational differences.
- Prior work and practical resources provide guidance on reasonable learning-rate ranges and hyper-parameter settings.
- CLR is computationally simpler than adaptive learning rates and can be combined with them.
- Adaptive methods estimate learning rates from gradients or related curvature statistics, including AdaGrad, RMSProp, AdaDelta, and Hessian-based approaches.
- The paper contrasts its use of increasing learning rates with prior work that appears to limit such increases to non-stationary problems.
3. Optimal Learning Rates
The triangular learning rate policy cycles linearly between minimum and maximum bounds, with an LR range test providing practical estimates for those bounds. Experiments indicate robust performance across cycle lengths and improved training efficiency when training ends at cycle boundaries.
- Cyclical Learning Rate Policy: Increasing the learning rate can temporarily hurt performance yet produce longer-term benefits, motivating cyclic variation rather than fixed or monotonically decreasing schedules.
- Cyclical Learning Rate Policy: The triangular policy linearly varies the learning rate between minimum and maximum boundaries, using stepsize as half the cycle length.The paper selected the triangular window because it is the simplest policy that captures beneficial cyclic variation.
- Cyclical Learning Rate Policy: The triangular2 policy halves the learning-rate difference after each cycle, while exp range reduces both boundaries by an exponential factor.
- Cycle Length: Cycle lengths are generally robust, with stepsize often set to 2–10 times the iterations per epoch; eight times the epoch length was only slightly better than twice.For CIFAR-10, 500 iterations constitute one epoch with batchsize 100.
- Cycle Length: Training with at least three cycles captures most of the network-weight training, while four or more cycles can improve performance; stopping at cycle ends is recommended.Accuracy peaks when the learning rate reaches its minimum at the end of a cycle.
- Boundary Estimation: An LR range test linearly increases the learning rate for several epochs to identify reasonable minimum and maximum boundaries for a new architecture or dataset.Choose the lower bound where accuracy starts increasing and the upper bound where accuracy slows, becomes ragged, or falls.
4. Experiments
Experiments across CIFAR-10, CIFAR-100, and ImageNet architectures show that cyclical learning rates generally match or improve fixed-rate accuracy, often in fewer iterations. Results also show benefits with adaptive methods and when starting from a best-guess learning rate.
- Caffe’s CIFAR-10 architecture: CLR matched 81.4% CIFAR-10 test accuracy in 25,000 iterations, compared with 70,000 iterations using standard hyper-parameters.The triangular2 policy used base lr = 0.001 and max lr = 0.006.
- Caffe’s CIFAR-10 architecture: CLR improved CIFAR-10 accuracy from 60.8% with a fixed rate to 72.2% for an architecture using sigmoids and batch normalization.The comparison used training accuracy from a Caffe-provided architecture.
- Caffe’s CIFAR-10 architecture: For adaptive methods, CLR sometimes reached the accuracy of 70,000-iteration runs after 25,000 iterations, though benefits varied by method.Nesterov with CLR reached 81.3% in 25,000 iterations, while Adam with and without CLR required 70,000 iterations.
- ResNets, Stochastic Depth, and DenseNets: Across ResNets, Stochastic Depth networks, and DenseNets on CIFAR-10 and CIFAR-100, CLR produced similar or better accuracy than fixed learning rates.DenseNet CIFAR-10 accuracy averaged 93.33% with CLR from 0.1 to 0.3, versus 91.67%, 92.17%, and 92.46% at fixed rates 0.1, 0.2, and 0.3.
5. Conclusions
Cyclical learning-rate policies estimate bounds with a short linear-increase run, then vary the rate between those bounds to obtain near-optimal classification results, often with fewer iterations. The paper reports broad practical benefits but leaves the full application range and theoretical understanding for future work.
- 5. Conclusions: A few epochs of linearly increasing the learning rate can estimate boundary values for cyclical learning-rate policies.The policy then varies the learning rate cyclically between these bounds.
- 5. Conclusions: Cyclical learning-rate policies provide near-optimal classification results, often with fewer iterations, while requiring essentially no additional computational expense.The paper describes the policy as easy to implement and contrasts its computational cost with adaptive learning-rate methods.
- 5. Conclusions: Cyclic learning-rate functions improve performance across a range of architectures and provide guidance on when to drop learning rates and stop training.The paper states that these effects reduce guesswork in setting learning rates.
- 5. Conclusions: The work does not explore the full range of applications for cyclical learning-rate methods or provide a theoretical analysis of them.The authors identify recurrent architectures and theoretical analysis as future directions.
A. Instructions for adding CLR to Caffe
Adding CLR to Caffe requires modifying the solver’s learning-rate calculation and extending SolverParameter with CLR-related configuration fields. The implementation includes triangular and triangular2 policies based on iteration, step size, base learning rate, and maximum learning rate.
- A. Instructions for adding CLR to Caffe: Modify SGDSolver::GetLearningRate() in sgd_solver.cpp to add CLR policy handling.The implementation instructions identify this solver method as the modification point.
- A. Instructions for adding CLR to Caffe: The triangular policy computes the learning rate from the base and maximum rates using the current iteration and step size.Its code derives a cycle and normalized position, then scales the base-to-maximum rate difference.
- A. Instructions for adding CLR to Caffe: The triangular2 policy applies cycle-dependent decay to the triangular rate increase.The implementation divides the normalized triangular component by 2^cycle before combining it with the base learning rate.
- A. Instructions for adding CLR to Caffe: When the CLR iteration condition is not active, the solver uses the base learning rate.The fallback branch explicitly assigns the base learning rate.
- A. Instructions for adding CLR to Caffe: Add start_lr_policy and max_lr fields to the Caffe SolverParameter message.The max_lr field is documented as the maximum learning rate for CLR policies.