Source-linked AI summary
Large Batch Training of Convolutional Networks
Yang You, Igor Gitman, Boris Ginsburg
TL;DR
Large-batch training can reduce accuracy and make conventional linear learning-rate scaling with warm-up unstable. The paper proposes Layer-wise Adaptive Rate Scaling (LARS), enabling AlexNet and ResNet-50 training at batch sizes up to 32K, while accuracy beyond 32K remains open.
Problem
Increasing computational workers enlarges batch sizes, but large batches can reduce accuracy and conventional high-learning-rate training may diverge.
Method
LARS assigns separate learning rates to layers by using layer-specific weight and gradient norms to stabilize large-batch optimization.
Results
For B=8K, AlexNet and AlexNet-BN matched the B=512 baseline, while AlexNet-BN lost 0.9% at B=16K and 2.6% at B=32K.
Takeaways & Limitations
LARS extends large-batch training of AlexNet and ResNet-50 to B=32K, but training above 32K without accuracy loss remains an open problem.
Takeaways & Limitations
Reported baselines are lower than published state-of-the-art because the experiments used minimal augmentation, one model, and one central crop at testing.
Abstract
from arXiv · showhide
A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.
1 INTRODUCTION
Scaling CNN training across more computational units increases batch size, but large batches can reduce accuracy and make linear learning-rate scaling unstable. The paper proposes LARS, which adapts learning rates by layer to stabilize large-batch training.
- Large-batch data-parallel SGD speeds CNN training by distributing each global mini-batch across computational workers.
- Larger batches can negatively affect model accuracy, motivating improved large-batch optimization methods.
- Fewer updates at fixed epochs motivate linear learning-rate scaling, but larger learning rates can make optimization difficult and cause divergence.
- For AlexNet, linear scaling reduced accuracy from 57.6% at B=256 to 53.1% at B=4K and 44.8% at B=8K; Batch Normalization reduced the B=8K gap from 14% to 2.2%.
- LARS uses separate layer-wise learning rates and controls update magnitude relative to weight norms, enabling AlexNet-BN and ResNet-50 training with B=32K without accuracy loss.
2 BACKGROUND
Large-batch training enables parallel computation but requires careful optimization because fewer updates and high learning rates can destabilize training. Prior work established linear scaling with warm-up, while generalization concerns remained unresolved.
- A mini-batch is divided across N parallel units, allowing larger batches to scale computation without reducing each unit’s workload.
- At fixed epochs, increasing batch size produces fewer weight-update steps, motivating proportionally larger learning rates.
- Linear learning-rate scaling trained AlexNet at B=1K with approximately 1% accuracy loss but became difficult above B=2K because larger learning rates caused divergence.
- Learning-rate warm-up addresses initial instability by gradually increasing the learning rate before switching to the regular policy; with linear scaling, it trained ResNet-50 at B=8K without accuracy loss.
- Large-batch methods were also associated with a generalization gap attributed to convergence toward sharp training-loss minimizers, but tested remedies did not produce a working solution.
3 ANALYSIS OF ALEXNET TRAINING WITH LARGE BATCH
AlexNet training exposed severe accuracy loss and divergence under large-batch linear scaling, while Batch Normalization improved stability and accuracy. The remaining B=8K loss was not associated with a larger training–testing loss gap.
- At B=4K, AlexNet’s best accuracy was 53.1%, while at B=8K it was 44.8%; training diverged when the learning rate exceeded 0.06 for B=4K.
- Batch Normalization enabled larger learning rates: AlexNet-BN reached 58.9% at B=4K and 58% at B=8K, versus a 60.2% B=512 baseline.
- Table 1 compares AlexNet and AlexNet-BN at B=4K and B=8K, emphasizing that Batch Normalization permits larger learning rates.
- The B=8K AlexNet-BN accuracy gap was 2.2%, and the training–testing loss gap did not differ significantly from B=256, indicating the loss was not attributed to a generalization gap.
4 LAYER-WISE ADAPTIVE RATE SCALING (LARS)
LARS addresses instability in large-learning-rate training by assigning each layer a local learning rate based on its weight-to-gradient norm ratio. The method is designed to accommodate substantial differences across layers and batch sizes.
- Motivation: Large learning rates can cause divergence when an update exceeds the weight norm, making early training sensitive to initialization and the initial learning rate.
- Motivation: The weight-to-gradient norm ratio varies substantially across layers, motivating a separate learning rate for each layer.The supplied table passage identifies AlexNet-BN norm measurements at the first iteration.
- LARS method: LARS replaces one shared learning rate with local learning rates λ_l for individual layers, while retaining a global learning-rate factor.The local rate is defined through a trust coefficient η < 1.
- LARS method: The trust coefficient η controls how much each layer is allowed to change during one update.The formulation is also extended to balance local learning rate and weight decay.
- Implementation: Algorithm 1 specifies SGD with LARS together with momentum, weight decay, and polynomial learning-rate decay.The algorithm initializes training parameters and iterates over layers while updating momentum and weights.
- Behavior: The local learning rate depends strongly on both layer and batch size.
5 TRAINING WITH LARS
LARS enables large-batch training by adapting learning rates across layers, extending AlexNet and ResNet-50 experiments to batches as large as 32K. Accuracy remains close to baseline at several large batch sizes, though degradation appears at the largest AlexNet-BN setting.
- LARS matched the B=512 baseline accuracy for AlexNet and AlexNet-BN at B=8K.AlexNet-BN lost 0.9% at B=16K and 2.6% at B=32K.
- A broad learning-rate interval produced approximately 59.3% accuracy for AlexNet-BN at B=16K.Learning rates from [13;22] gave the reported accuracy.
- A broad learning-rate interval produced approximately 57.5% accuracy for AlexNet-BN at B=32K.Learning rates from [17,28] gave the reported accuracy.
- The ResNet-50 experiments used LARS with five epochs of warm-up and polynomial learning-rate decay.Training used SGD with momentum 0.9 and weight decay 0.0001 for 90 epochs.
- LARS scaled ResNet-50 to B=32K with a 0.7% accuracy loss relative to the baseline.The reported baseline top-1 accuracy was 73%.
6 LARGE BATCH VS NUMBER OF STEPS
With very large batches, fewer parameter-update steps can limit accuracy even when LARS and large learning rates stabilize training. The AlexNet-BN results indicate that longer training can recover the baseline accuracy.
- At B=32K, LARS with a large learning rate did not initially reach baseline accuracy, but longer training recovered it completely.The paper attributes this to very large batches producing stochastic gradients close to true gradients.
- The B=32K accuracy-versus-duration results are presented for AlexNet-BN.
7 CONCLUSION
The paper concludes that large learning rates can make large-batch CNN training diverge, while LARS adapts the learning rate per layer and extends scaling to batch size 32K. Accuracy-preserving scaling beyond 32K remains open.
- Large learning rates can cause divergence, especially during the initial training phase, even with learning-rate warm-up.
- LARS adapts the learning rate for each layer to address the optimization difficulties of large-batch training.
- LARS extended AlexNet and ResNet-50 scaling to a batch size of 32K.
- Training these networks above B=32K without accuracy loss remains an open problem.