PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Small Batch Training for Deep Neural Networks

Dominic Masters, Carlo Luschi

arXiv:1804.07612v1cs.LGcs.CVstat.ML

TL;DR

Large batches offer greater computational parallelism, but the evidence base for their learning-rate scaling and training behavior remains contested against small-batch advantages. The paper reviews these assumptions and experimentally compares mini-batch sizes, finding that small batches generally provide more stable convergence and better test performance, with best results at m = 32 or smaller. These benefits must be weighed against reduced computational parallelism and batch-normalization limitations at very small sizes.

  • Problem

    Large batches improve computational parallelism, while the relative training stability, generalization, and learning-rate behavior of different mini-batch sizes require systematic comparison.

  • Method

    The paper reviews assumptions about training duration and learning-rate scaling, then empirically evaluates mini-batch SGD across network architectures and datasets.

  • Results

    Across the experiments, small batches achieved the best training stability and generalization performance, with best results at m = 32 or smaller and often m = 2 or m = 4.

  • Takeaways & Limitations

    Small-batch training supports a broader stable-learning-rate range and better test performance for a given computational cost or number of epochs.

  • Takeaways & Limitations

    Small batches reduce available computational parallelism, and very small batches can make batch-normalization statistics noisy and less representative of testing statistics.

Abstract

from arXiv · show

Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footprint, which might also be exploited to improve machine throughput. In this paper, we review common assumptions on learning rate scaling and training duration, as a basis for an experimental comparison of test performance for different mini-batch sizes. We adopt a learning rate that corresponds to a constant average weight update per gradient calculation (i.e., per unit cost of computation), and point out that this results in a variance of the weight updates that increases linearly with the mini-batch size $m$. The collected experimental results for the CIFAR-10, CIFAR-100 and ImageNet datasets show that increasing the mini-batch size progressively reduces the range of learning rates that provide stable convergence and acceptable test performance. On the other hand, small mini-batch sizes provide more up-to-date gradient calculations, which yields more stable and reliable training. The best performance has been consistently obtained for mini-batch sizes between $m = 2$ and $m = 32$, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.

1 INTRODUCTION

Mini-batch training presents a trade-off between the parallelism and efficiency of large batches and the generalization, convergence, and memory advantages of small batches. The paper therefore revisits batch-size-dependent learning-rate assumptions and experimentally compares training behavior across batch sizes.

  • Large mini-batches increase SGD parallelism for more efficient processors and distributed processing.
  • Small mini-batches can improve generalization and optimization convergence while requiring a significantly smaller memory footprint.
  • Maintaining generalization with large batches by matching the number of SGD updates introduces computational overhead proportional to mini-batch size.
  • With summed mini-batch gradients, a fixed learning rate keeps the expected weight update per training example constant across batch sizes.
  • Holding the expected update per gradient calculation constant makes weight-update variance increase linearly with batch size.
  • The experiments compare training and generalization across a wider batch-size range, including values as small as 2 or 4.

2 BACKGROUND: BATCH TRAINING AND BATCH NORMALIZATION

This section defines mini-batch SGD and examines how learning-rate scaling, update variance, gradient staleness, and Batch Normalization depend on batch size. It argues that larger batches trade parallelism for progressively different optimization dynamics and may make normalization estimates less reliable.

  • Stochastic Gradient Optimization: Mini-batch SGD estimates the loss gradient from a subset of training examples and updates network parameters using that stochastic approximation.The empirical loss averages per-example losses over the training set, while each update uses a mini-batch of m examples.
  • Learning-Rate Scaling: A constant expected weight update per training example requires learning rate η to increase linearly with batch size m.This follows because the expected update per unit computation is proportional to η/m.
  • Learning-Rate Scaling: Under linear scaling, keeping the base learning rate constant causes weight-update variance to increase linearly with batch size m.The variance scales with ˜η^2 · m, so larger batches alter update noise even when the mean update per training example is preserved.
  • Learning-Rate Scaling: Large-batch training approximates small-batch training by replacing successive fresh updates with more stale gradients when the base learning rate is held constant.The approximation becomes weaker when batch size or base learning rate grows, especially early in training while parameters change rapidly.
  • Learning-Rate Scaling: Smaller batches calculate gradients from more up-to-date weights and have lower update variance, potentially supporting faster and more robust convergence.This challenges the assumption that larger batches necessarily provide more useful gradient estimates or permit larger learning rates.
  • Batch Normalization: Batch Normalization estimates feature-wise means and variances over the batch, then scales and shifts normalized activations using learned parameters γ and β.For convolutional layers, the effective estimation sample size is m · p · q; very small batches can make estimates noisy and less representative of testing statistics.

3 BATCH TRAINING PERFORMANCE

Across CIFAR-10, CIFAR-100, and ImageNet experiments, smaller batch sizes generally produced better test performance and more reliable convergence. Increasing batch size narrowed the stable-learning-rate range and degraded training, with the strongest results typically at small batch sizes.

  • Experimental setup: Experiments varied batch size and base learning rate across AlexNet and ResNet models on CIFAR-10, CIFAR-100, and ImageNet using standard SGD.Momentum was excluded to isolate the interaction between batch size and learning rate.
  • 3.2 PERFORMANCE WITHOUT BATCH NORMALIZATION: For CIFAR-10 models without BN or augmentation, best accuracy occurred at m = 8 or smaller for reduced AlexNet and at m = 4 or m = 2 for ResNet-8 and ResNet-20.Increasing batch size at constant ˜η reduced generalization performance and could lead to instability.
  • 3.2 PERFORMANCE WITHOUT BATCH NORMALIZATION: For fixed ˜η = 2−8, batch sizes 4 ≤ m ≤ 64 showed similar ResNet-32 convergence, whereas larger batches degraded training performance.At higher learning rates, analogous curves also showed divergence for large batch sizes.
  • 3.3 PERFORMANCE WITH BATCH NORMALIZATION: Across BN and augmentation settings, increasing batch size consistently degraded performance, with best ResNet-32 results at m = 4 or m = 8.The optimum base learning rate was achievable only for m = 16 or smaller on CIFAR-10 and m = 8 or smaller on CIFAR-100.
  • ImageNet results: ImageNet likewise favored small batches: ResNet-50 achieved best validation accuracy at m = 16 to m = 64, while larger batches behaved less predictably.Batches m = 16 and m = 32 performed best across a continuous range of base learning rates, and small batches provided reliable convergence over a wider range.
  • 3.6 DIFFERENT BATCH SIZE FOR WEIGHT UPDATE AND BATCH NORMALIZATION: Using smaller BN batches alongside larger SGD batches improved results, with BN batch sizes m = 4 or m = 8 often best across tested SGD batch sizes.The authors suggest distributing both BN and SGD over multiple workers with a small batch size per worker.

4 DISCUSSION

Across datasets and architectures, the strongest results generally come from small batches, while larger batches narrow the stable learning-rate range and can lose performance even with warm-up. Batch normalization and hardware efficiency introduce important trade-offs in choosing batch sizes.

  • Test performance: Best CIFAR-10 test performance is consistently achieved below m = 32, with performance sometimes improving down to m = 2 without BN.With BN, performance appears maintained for batch sizes as small as m = 4 and m = 8.
  • Trade-offs: Small batches provide more up-to-date gradient information, while their reduced computational parallelism creates a trade-off between hardware efficiency and test performance.The paper also reports that overall SGD batch size contributes more to performance than BN batch size, with best BN batch sizes often between m = 4 and m = 8.
  • Test performance: Best ImageNet performance is achieved with batch sizes between m = 16 and m = 64, although m = 64 is acutely sensitive to learning-rate choice.For m ≤8, performance depends on batch-normalization effects.
  • Training strategies: Gradual warm-up improves large-batch performance but does not fully recover the performance achieved with the smallest batches.
  • Stable convergence: Increasing batch size progressively reduces the learning-rate range that provides stable convergence across the evaluated architectures and datasets.The results associate this narrowing with increased weight-update variance and highlight limitations of linear learning-rate scaling.

5 CONCLUSIONS

The conclusions report that small batches deliver the best stability and generalization across the experiments, while larger batches face narrower usable learning-rate ranges. Batch normalization and larger datasets can make somewhat larger batches useful, but computational parallelism remains a trade-off.

  • Small batch sizes achieve the best training stability and generalization performance for a given computational cost across a wide range of experiments.
  • The best results are obtained with batch sizes m = 32 or smaller, often as small as m = 2 or m = 4.
  • With BN and larger datasets, batch sizes up to m = 32 or m = 64 can be useful, while the best BN batch size is often smaller than the overall SGD batch size.
  • Larger batch sizes significantly reduce the usable base-learning-rate range, sometimes preventing use of the optimal learning rate.The paper suggests this is attributable to a linear increase in weight-update variance with batch size.
  • Small batches benefit both stable-convergence range and test performance for a given number of epochs.

A ADDITIONAL RESULTS WITH BATCH NORMALIZATION

Additional CIFAR-10 ResNet-32 experiments vary training length while retaining scheduled learning-rate reductions. Longer training improves performance, but it does not eliminate the performance gap between large and small batches.

  • The experiments repeat CIFAR-10 ResNet-32 training with data augmentation for 41 and 164 epochs, using learning-rate reductions at 50% and 75% of training.
  • Increasing training length consistently improves performance, with larger gains at lower base learning rates for both very small and large batches.
  • At base learning rates corresponding to the best test accuracy, longer training benefits large batches mainly, but large-batch performance remains inferior to small-batch performance.
  • Figures 17 and 18 report test performance for reduced and increased epoch counts in the CIFAR-10 ResNet-32 experiments.
Loading 1804.07612v1…