Don't Decay the Learning Rate, Increase the Batch Size

Samuel L. Smith; Pieter-Jan Kindermans; Chris Ying; Quoc V. Le

Source-linked AI summary

Don't Decay the Learning Rate, Increase the Batch Size

Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le

arXiv:1711.00489v2Nov 1, 2017cs.LGcs.CVcs.DCstat.ML

TL;DR

Learning-rate decay is standard, but large batches can reduce updates and improve parallelism while often hurting test accuracy. The paper shows that increasing batch size can usually reproduce learning-rate schedules, preserving test accuracy over epochs and enabling faster large-batch training, with some momentum-related trade-offs.

Problem
Large-batch training offers parallelism and fewer updates but often lowers test accuracy, while the empirical equivalence to learning-rate decay had not been established.
Method
The paper converts learning-rate decay schedules into increasing-batch-size schedules and scales batch size with learning rate or momentum to control optimization noise.
Results
The schedules achieve near-identical test performance over the same epochs with fewer updates, including 76.1% ImageNet validation accuracy for ResNet-50 in under 30 minutes.
Takeaways & Limitations
Existing training schedules can be repurposed for large-batch training without hyper-parameter tuning, reducing updates and training time.
Takeaways & Limitations
Scaling batch size with momentum can slightly reduce test accuracy, and large-batch initialization bias correction can cause early-training instabilities.

Abstract

from arXiv · show

It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times. We can further reduce the number of parameter updates by increasing the learning rate $ε$ and scaling the batch size $B \propto ε$. Finally, one can increase the momentum coefficient $m$ and scale $B \propto 1/(1-m)$, although this tends to slightly reduce the test accuracy. Crucially, our techniques allow us to repurpose existing training schedules for large batch training with no hyper-parameter tuning. We train ResNet-50 on ImageNet to $76.1\%$ validation accuracy in under 30 minutes.

1 INTRODUCTION

The paper proposes replacing learning-rate decay with increasing batch size, preserving learning curves while reducing parameter updates. This creates a direct route to efficient large-batch training without fine-tuning.

Motivation: Large batches can be parallelized across many machines, reducing training time, but often suffer lower test accuracy.The paper targets this accuracy trade-off while retaining the parallelism benefits of large-batch training.
Core proposal: Increasing the batch size during training can reproduce learning-rate decay with near-identical test performance over the same number of epochs.The replacement follows existing schedules: when the learning rate drops by α, the batch size increases by α.
Large-batch scaling: The method further reduces parameter updates by increasing the learning rate and scaling B ∝ ε.Scaling the batch size with the learning rate supports larger steps while following the proposed schedule.
Large-batch scaling: Increasing momentum and scaling B ∝ 1/(1-m) can reduce updates further, although test accuracy decreases slightly.This trade-off is reported separately from the learning-rate and batch-size equivalence.
Novelty: The work addresses a previously unshown empirical equivalence between increasing batch size and decaying learning rate.Earlier studies discussed increasing batch size, but had not demonstrated quantitative equivalence between the two schedules.
Evaluation: The experiments cover CIFAR-10 and ImageNet and include SGD, Momentum, and Adam.The paper presents the approach as applicable across multiple datasets and optimizers.

2 STOCHASTIC GRADIENT DESCENT AND CONVEX OPTIMIZATION

The paper interprets SGD through gradient-noise dynamics and argues that reducing noise by increasing batch size can substitute for learning-rate decay. This reframes a conventional convergence condition as dependent on batch-size scheduling.

Background: SGD introduces gradient noise that can obstruct optimization, motivating learning-rate decay for convergence.The discussion begins from the standard conditions used to reach the minimum of a strongly convex function.
Convergence criteria: The learning-rate conditions do not require decay when batch size varies, because the relevant convergence condition assumes constant batch size.The paper explicitly notes that the second condition holds only under a constant, or suitably bounded, batch size.
Noise-scale interpretation: The stochastic differential-equation view models mini-batch gradient estimation as Gaussian random noise in parameter dynamics.The noise has zero mean and covariance shaped by gradient fluctuations between parameters.
Noise-scale interpretation: The noise scale g depends on learning rate, training-set size, and batch size, and controls random fluctuations in training dynamics.The supplied expression identifies g as proportional to ε(N/B − 1).
Proposed schedule: Increasing batch size at constant learning rate can reduce the noise scale like learning-rate decay, enabling efficient use of large batches until B ∼ N/10.After this point, the paper reverts to decaying learning rates.

3 SIMULATED ANNEALING AND THE GENERALIZATION GAP

The paper connects learning-rate decay to simulated annealing: reducing gradient noise during training can aid convergence while preserving generalization. It argues that batch-size scheduling can produce the same noise-scale reduction.

Generalization gap: Small-batch training often generalizes better than large-batch training, producing a recognized generalization gap.Prior work attributed an optimal batch size to an optimal noise scale at constant learning rate.
Scaling rule: The optimal batch size was reported to scale with learning rate and training-set size, and this rule enabled earlier large-batch ImageNet training.Goyal et al. used a linear batch-size–learning-rate scaling rule with batches of 8192 images.
Noise and generalization: Gradient noise may benefit non-convex optimization by helping SGD escape sharp minima that generalize poorly.The paper presents this as a proposed explanation for the generalization behavior of noisy optimization.
Scope of classical criteria: Because deep learning often uses early stopping, the paper questions whether classical strongly-convex convergence conditions fully explain its learning-rate schedules.Despite this uncertainty, learning-rate decay remains empirically successful in deep learning.
Simulated annealing: Sharper decay schedules can be interpreted as more rapid annealing of the optimization noise scale.The paper relates this interpretation to the increasing use of cosine decay and step-function drops.

4 THE EFFECTIVE LEARNING RATE AND THE ACCUMULATION VARIABLE

With momentum, the effective learning rate and the accumulation timescale determine how batch-size scaling affects training dynamics. Higher momentum can suppress or delay parameter updates, requiring care in schedule design.

The effective learning rate is ϵ/(1−m), and batch size can be scaled as B ∝ ϵ/(1−m) to reduce parameter updates.This extends the noise-scale analysis from vanilla SGD to momentum methods.
Increasing the learning rate while scaling B ∝ ϵ performs well, whereas scaling B ∝ 1/(1−m) with higher momentum slightly reduces test accuracy.
The accumulation A tracks an exponentially decaying average of gradient estimates and is initially initialized to zero.The mean gradient per training example is estimated on a batch of size B.
The accumulation approaches steady state over approximately B/(N(1−m)) training epochs, suppressing parameter updates and reducing convergence during this period.
Higher momentum lengthens the accumulation’s memory timescale, which can prevent adaptation to changes in the loss landscape and require additional training epochs.The issue is especially relevant where the noise scale decays.

5 EXPERIMENTS

Across CIFAR-10 and ImageNet experiments, increasing batch size reproduces learning-rate-decay performance while reducing parameter updates, including across several optimizers. Scaling the effective learning rate and batch size reduces updates further, while higher momentum trades additional efficiency for lower accuracy.

5.1 SIMULATED ANNEALING IN A WIDE RESNET: Increasing batch size and learning-rate decay produce nearly identical training and test curves on Wide ResNet CIFAR-10.This equivalence holds for SGD with momentum, Nesterov momentum, vanilla SGD, and Adam.
5.2 INCREASING THE EFFECTIVE LEARNING RATE: The batch-size schedule preserves CIFAR-10 accuracy while substantially reducing parameter updates compared with the original schedule.The original schedule uses approximately 80000 updates, versus approximately 29000 with increasing batch size.
5.2 INCREASING THE EFFECTIVE LEARNING RATE: Increasing the effective learning rate and scaling the initial batch size reduces CIFAR-10 training to under 6500 updates without lowering final accuracy.The increased-initial-learning-rate schedule reaches 94.5% final test accuracy, compared with 94.3% for the original schedule.
5.2 INCREASING THE EFFECTIVE LEARNING RATE: Increasing momentum reduces updates further, but the less-than-2500-update schedule reaches lower final test accuracy of 93.3%.Across five additional runs, the median accuracy for this schedule is 93.5%.
5.3 TRAINING IMAGENET IN 2500 PARAMETER UPDATES: On ImageNet, increasing batch size achieves similar accuracy to learning-rate decay while reducing Inception-ResNet-V2 updates from just over 14000 to below 6000.The two increasing-batch-size runs reached 78.1% and 76.8%, compared with 78.7% and 77.8% for decaying learning rate; the difference was similar to run-to-run variance.
5.4 TRAINING IMAGENET IN 30 MINUTES: ResNet-50 reached 76.1% ImageNet validation accuracy in under 30 minutes after increasing the batch size during training.The fixed-batch baseline reached the same accuracy in under 45 minutes, while doubling the initial learning rate produced 75.0% in 22 minutes.

6 RELATED WORK

Prior work established several batch-size scaling rules and alternative optimizers for efficient training, but their accuracy and generalization trade-offs remained relevant comparisons.

Goyal et al. observed B ∝ ε, while Hoffer et al. proposed B ∝ √ε.
Bottou et al. showed that SGD converges to strongly convex minima in similar numbers of training epochs when B ∝ ε.
LARS trained ImageNet in 14 minutes but reached a lower final accuracy of 74.9%.
Wilson et al. argued that adaptive optimization methods tend to generalize less well than SGD and SGD with momentum.

7 CONCLUSIONS

The paper shows that increasing batch size can often replace learning-rate decay across optimizers, reducing updates and enabling efficient large-batch training without hyper-parameter tuning.

Increasing batch size often achieves the benefits of learning-rate decay across SGD, Momentum, and Adam experiments on CIFAR-10 and ImageNet.
77% ImageNet validation accuracy was reached with Inception-ResNet-V2 in under 2500 parameter updates using batches of 65536 images.
76.1% ImageNet validation set accuracy was achieved for ResNet-50 on TPU in under 30 minutes.
Scaling B ∝ ε/(1−m) by increasing learning rate and momentum further reduces parameter updates, although higher momentum slightly reduces test accuracy.
Existing hyper-parameter choices can be converted directly for large-batch training without hyper-parameter tuning.

A THE GROWTH OF THE ACCUMULATION AT THE START OF TRAINING

At training start, momentum accumulation grows from zero, temporarily suppressing parameter updates and creating an effective loss of training epochs; this loss grows rapidly with momentum.

The accumulation variable starts at zero and grows toward the mean gradient, initially suppressing parameter updates and reducing the effective learning rate.
The accumulation variable grows exponentially during the initial training phase.
The effective number of lost training epochs depends on the accumulation growth during startup.
N_lost ∝ ε/(N(1−m)^2), so the lost-epoch count rises most rapidly when the momentum coefficient increases.
The startup loss can be addressed by adding training epochs or ensuring it remains negligible relative to training before the noise scale decays.

B INCREASING THE INITIAL LEARNING RATE

The initial learning rate can be increased while scaling batch size proportionally and preserving the number of training epochs, but accuracy eventually declines beyond a threshold.

Increasing the learning rate while scaling B ∝ ε preserves the same number of training epochs.
For ε = 0.1, the baseline batch size is B = 128 throughout training.
The initial learning rate can be increased only to approximately 0.4 before final test accuracy starts to fall.
Each plotted result is the median of five runs.