PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

Aditya Devarakonda, Maxim Naumov, Michael Garland

arXiv:1712.02029v2cs.LGcs.CVcs.DCstat.ML

TL;DR

Training neural networks requires balancing the faster convergence of small batches against the computational efficiency of large batches. AdaBatch progressively increases batch size during training while adapting the learning rate, and experiments across CNN architectures and datasets report up to 6.25× speedups with less than 1% accuracy difference from fixed-batch baselines.

  • Problem

    Static batch sizes force a trade-off between the faster convergence of small batches and the parallelism and computational efficiency of large batches.

  • Method

    AdaBatch progressively increases batch size during training and adapts learning rate so the α/r ratio remains constant.

  • Results

    Across AlexNet, ResNet, and VGG on CIFAR-10, CIFAR-100, and ImageNet, adaptive batch sizes achieve up to 6.25× speedup with less than 1% accuracy difference from fixed-batch baselines.

  • Takeaways & Limitations

    Adaptive batch sizing can preserve the better test accuracy associated with small batches while obtaining the higher performance associated with large batches.

  • Takeaways & Limitations

    The derivation assumes the number of training samples is divisible by the batch size and assumes grouped small-batch updates are approximately similar to large-batch updates.

Abstract

from arXiv · show

Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a new training approach that, rather than statically choosing a single batch size for all epochs, adaptively increases the batch size during the training process. Our method delivers the convergence rate of small batch sizes while achieving performance similar to large batch sizes. We analyse our approach using the standard AlexNet, ResNet, and VGG networks operating on the popular CIFAR-10, CIFAR-100, and ImageNet datasets. Our results demonstrate that learning with adaptive batch sizes can improve performance by factors of up to 6.25 on 4 NVIDIA Tesla P100 GPUs while changing accuracy by less than 1% relative to training with fixed batch sizes.

1 INTRODUCTION

Static batch sizes force a trade-off: small batches tend to converge in fewer epochs, while large batches improve parallelism and computational efficiency. AdaBatch resolves this by progressively increasing batch size during training, preserving accuracy while improving runtime.

  • 1 INTRODUCTION: Small batches generally converge in fewer epochs, whereas large batches provide greater data parallelism and computational efficiency.Static batch-size training requires choosing between these competing benefits.
  • 1 INTRODUCTION: AdaBatch begins with small batches for early convergence, then progressively increases batch size to expose more parallelism later in training.The experiments double batch size at selected intervals while adapting learning rate so α/r remains constant.
  • 1 INTRODUCTION: The approach was evaluated across AlexNet, ResNet, and VGG on CIFAR-10, CIFAR-100, and ImageNet.

2 RELATED WORK

Prior work studied learning-rate scaling, variance-based selection, scheduled batch increases, and replacing learning-rate decay with batch-size growth. AdaBatch extends this line by evaluating adaptive batch sizing on popular CNNs and combining it with fixed large-batch techniques.

  • 2 RELATED WORK: Earlier studies used learning-rate warmup, linear scaling, or layer-wise scaling to train with fixed batch sizes as large as 32,768.Those approaches kept batch size fixed throughout training, unlike AdaBatch.
  • 2 RELATED WORK: Other work adaptively increased batch sizes for variance reduction or optimization convergence, mainly in convex or second-order settings.The cited studies include fixed schedules, gradient-variance criteria, and analyses on convex problems or convolutional networks.
  • 2 RELATED WORK: A recent study showed that batch-size increases can replace learning-rate decay across SGD, momentum SGD, and Adam.It also explored changing momentum alongside batch-size increases.
  • 2 RELATED WORK: AdaBatch complements fixed large-batch methods and can combine with them to obtain larger speedups while preserving similar test-error performance.The paper positions adaptive batch sizing as complementary to existing large-batch training techniques.

3 ADAPTIVE BATCH SIZING AND ITS EFFECTS

AdaBatch links batch size and learning rate so that increasing batch size can mimic learning-rate decay, while preserving per-epoch work under linear computation scaling. It is designed to improve hardware utilization without sacrificing the accuracy advantages associated with smaller batches.

  • 3.1 LEARNING RATE: The setup partitions training data into batches and assumes the sample count is divisible by the batch size, with padding or truncation handling required otherwise.
  • 3.1 LEARNING RATE: Doubling batch size while scaling learning rate preserves the relationship needed for comparable epoch-level updates when grouped batches produce similar gradients.The derivation compares q small-batch iterations with q/β large-batch iterations and uses α = ˜α/β under the stated approximation.
  • 3.1 LEARNING RATE: Increasing batch size can mimic learning-rate decay, and the experiments keep effective learning rates fixed for fair fixed-versus-adaptive comparisons.The method uses fixed schedules for batch-size increases and can combine batch-size growth with learning-rate decay.
  • 3.2 TEST ACCURACY AND PERFORMANCE: Larger batches improve GPU throughput and hide communication costs across GPUs, but tend to reduce test accuracy by converging to sharper minima.Adaptive sizing progressively exposes more parallelism while targeting the accuracy benefits of smaller batches.
  • 3.3 WORK PER EPOCH: Under linear forward-and-backward computation, increasing batch size raises flops per iteration but leaves flops per epoch unchanged.For a fully connected layer, βr batches require O(mnβr) flops per iteration and still O(mnrq) per epoch.

4 EXPERIMENTAL RESULTS

Experiments across CIFAR-10, CIFAR-100, and ImageNet evaluate adaptive batch sizes with VGG, ResNet, and AlexNet. Adaptive schedules preserve test accuracy while improving training efficiency and scaling to large batches.

  • 4.1 FIXED VS. DYNAMIC BATCH SIZES: Adaptive batch sizes attained CIFAR-10 test errors within 1% of the smallest fixed batch size and lower errors than the largest fixed batch sizes.The comparison covered VGG19, ResNet-20, and AlexNet.
  • 4.1 FIXED VS. DYNAMIC BATCH SIZES: Adaptive batch sizes on CIFAR-100 likewise remained within 1% of the smallest fixed batch size while improving efficiency and scalability without sacrificing test error.The experiments used the same VGG19, ResNet-20, and AlexNet network settings, with Figure 2 reporting lowest test error over five trials.
  • 4.1 FIXED VS. DYNAMIC BATCH SIZES: 6.25× speedup was achieved for adaptive 1024–16384 batches on ResNet-20, with less than 2% test-error difference from the baseline.VGG19 achieved a 3.54× average speedup under the same CIFAR-100 evaluation.
  • 4.2 MULTI-GPU PERFORMANCE: Adaptive schedules converged to test errors similar to fixed settings on CIFAR-100, including when combined with gradual learning-rate warmup.The reported difference from fixed counterparts was less than 1%.
  • 4.3 IMAGENET TRAINING WITH ADABATCH: On ImageNet, AdaBatch closely matched fixed batch size 4096 while eventually training with batch size 16,384; fixed 8192 and 16,384 did not reach the same test errors.With learning-rate warmup, adaptive settings also showed convergence similar to small fixed batches and lower test errors than large fixed batches.

5 CONCLUSION

The paper develops adaptive batch sizes that dynamically vary during training, preserving small-batch accuracy while improving performance associated with large batches.

  • Adaptive batch sizes preserve the better test accuracy of small batches while obtaining higher performance often associated with large batches.
  • 6.25× speedups on 4 NVIDIA P100 GPUs were achieved with less than 1% accuracy difference from fixed-batch baselines.
  • ImageNet experiments attained batch sizes up to 524,288 without altering test error performance.
  • Future work will explore alternative resizing schedules, including shrinking batch sizes to improve convergence properties.

A APPENDIX: DETAILED ANALYSIS OF LAYERS

The appendix reviews convolutional, batch-normalization, and fully connected layers used to construct the networks studied in this work.

  • The networks combine convolution, batch normalization, and fully connected layers.

A.1 LOSS FUNCTION

The appendix assumes cross entropy loss and introduces its sample- and batch-level notation, softmax output, and one-hot target behavior.

  • The analysis assumes cross entropy loss as the loss function used in this paper.
  • The loss notation indexes batches by t and elements within each batch by s.
  • The output uses a softmax function.
  • For one-class targets, only one component of the target vector is non-zero.

A.2 FULLY CONNECTED LAYER

The fully connected layer maps inputs through an affine transformation and activation, with forward and backward computation whose cost grows linearly with batch size.

  • A fully connected layer maps x ∈ R^n through y = Wx + b and z = f(y), with weights, biases, and component-wise activation.
  • The layer's backward pass computes weight-gradient components from error-gradient terms and activation derivatives.
  • For batch size r, the weight update uses learning rate α.
  • Forward and backward computation requires O(mnr) flops, so cost depends linearly on batch size.

A.3 CONVOLUTION LAYER

The convolution layer maps a padded input through a kernel with specified strides to produce an activated output, with forward and backward computation scaling linearly with batch size.

  • Convolution dimensions: A 2D convolution produces an m′×n′ output from an m×n input and k1×k2 kernel, using vertical and horizontal strides s1 and s2.Padding is assumed so the output dimensions are m′ = (m − k1)/s1 + 1 and n′ = (n − k2)/s2 + 1.
  • Forward propagation: The convolutional layer applies an activation function to the convolution result plus a bias matrix.
  • Backward propagation: Backward propagation computes weight-gradient components and uses stride-aware indexing, kernel rotation, and permutation or selection matrices.The index switch accounts for strides and boundary conditions, with out-of-range summation elements treated as zero.
  • Computational cost: Forward and backward convolution computation both require flops that depend linearly on batch size.The stated costs are O(k1k2m′n′r) for forward propagation and O(k′1k′2mnr) for backward propagation.

A.4 BATCH NORMALIZATION LAYER

The batch normalization layer transforms an entire batch using batch statistics and has forward and backward computation whose cost grows linearly with batch size.

  • Batch normalization: Batch normalization operates on the entire batch, mapping a batched input X ∈ Rm×r to an output Z ∈ Rm×r.
  • Batch statistics: The normalization transformation uses row-wise batch means and variances, together with diagonal weights, bias, and an activation function.The cited formulation identifies the mean and variance through matrix expressions and defines the component-wise function f(x) = √x + ϵ.
  • Backward propagation: Backward propagation derives weight updates from error-gradient components, including terms involving the Kronecker delta and Hadamard products.
  • Computational cost: Forward and backward propagation for batch normalization requires O(mr) flops, so computation depends linearly on batch size.
Loading 1712.02029v2…