Source-linked AI summary
Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
Sergey Ioffe
TL;DR
Batchnorm can perform poorly with small or non-i.i.d. minibatches because training activations depend on minibatch statistics and differ from inference activations. Batch Renormalization adds a correction that matches training and inference activations, improving performance in these settings while retaining batchnorm’s efficiency.
Problem
Batchnorm’s minibatch-dependent activations differ between training and inference and can harm models trained with small or non-i.i.d. minibatches.
Method
Batch Renormalization augments batchnorm with a per-dimension affine correction whose minibatch-derived parameters are treated as constants during gradient computation.
Results
Batch Renormalization significantly improves training on small or non-i.i.d. minibatches compared with batchnorm, without extra cost and while matching training and inference outputs.
Takeaways & Limitations
Batch Renormalization preserves batchnorm’s implementation simplicity and training/inference speed while reducing dependence on the entire minibatch.
Takeaways & Limitations
Replacing minibatch moments with moving averages during training can cause unbounded parameter growth without improving the loss, so minibatch moments must remain in backpropagation.
Abstract
from arXiv · showhide
Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.
1 Introduction
Batch normalization accelerates deep-network training but becomes problematic when minibatches are small or non-i.i.d. Batch Renormalization addresses this by aligning training and inference activations and reducing minibatch dependence.
- Batch normalization stabilizes internal activation distributions, permits higher learning rates, reduces initialization sensitivity, and accelerates training.It has also enabled architectures such as residual networks.
- Batchnorm normalizes each activation using minibatch mean and variance, so an example’s normalized output depends on other examples in the minibatch.Moving averages of minibatch statistics are typically used during inference instead.
- Training–inference activation differences are small for large i.i.d. minibatches but can harm models when minibatches are small or non-i.i.d.Small batches yield inaccurate statistics, while related-example sampling can make the model overfit minibatch composition.
- Existing alternatives may require careful nonlinearities analysis, change the representable function class, or increase training expense.These approaches also may not guarantee normalized activations outside a fixed minibatch.
- Batch Renormalization extends batchnorm so training activations depend only on individual examples and match inference activations.The paper reports improved training on small or non-i.i.d. minibatches without extra cost.
2 Prior Work: Batch Normalization
Batch normalization incorporates minibatch statistics into both the forward computation and backpropagation, producing training benefits but creating a mismatch with inference statistics. Using moving averages during training would remove minibatch dependence but can destabilize optimization.
- Batch normalization computes sample mean and variance from minibatch activations, then applies a trainable affine transformation.The normalization is performed independently for each dimension or convolutional channel.
- The minibatch mean and variance are functions of model parameters and therefore participate in backpropagation.Batchnorm’s backpropagation formulas follow directly by the chain rule.
- Batchnorm’s scale invariance reduces sensitivity to weight initialization and permits larger learning rates.This follows when activations arise from a linear transform, ignoring the small numerical-stability constant.
- Batch normalization can improve gradient flow through normalized layers, helping prevent gradients from exploding or vanishing.This intuition relies on independence and linearity assumptions that do not fully hold in practice.
- During inference, batchnorm uses moving-average statistics instead of minibatch statistics, so its activations differ from those produced during training.The inference normalization depends only on a single input example rather than a whole minibatch.
- Using moving averages during training can cause optimization and normalization to counteract each other, producing unbounded parameter growth without improving the loss.Therefore minibatch moments must remain part of the computation and be differentiated through.
3 Batch Renormalization
Batch Renormalization augments batch normalization with a fixed per-minibatch affine correction so training can use moving-average statistics and match inference activations. The method gradually relaxes correction bounds and uses an additional backpropagation projection to preserve the benefits of batch normalization.
- Batch Renormalization: Batch Renormalization retains the correction factors r and d as constants during gradient computation, forming a per-dimension affine transformation of normalized activations.The correction accounts for differences between minibatch and population statistics.
- Batch Renormalization: The correction is identity in expectation when the moving averages equal expected minibatch statistics, with E[r] = 1 and E[d] = 0.Batch normalization is the special case r = 1 and d = 0.
- Batch Renormalization: Batch Renormalization uses moving averages µ and σ during training to correct minibatch normalization toward the activations generated by the inference model.Unlike batch normalization, the moving averages participate in the training-time correction rather than being reserved for inference.
- Batch Renormalization: Correction bounds initially constrain r and d to 1 and 0, then gradually relax to increase the allowed correction during training.This warm-up begins with batch normalization alone before introducing the correction.
- Batch Renormalization: The backpropagation interpretation is to normalize with moving averages and apply an extra projection step onto the null-space associated with minibatch shifts and scalings.The projection follows the gradient rescaling described by the Batch Renormalization formulas.
- Batch Renormalization: Batch Renormalization preserves batch normalization’s insensitivity to initialization and efficient training with large learning rates while aligning training representations with inference representations.The alignment ensures upper layers are trained on internal representations used during inference.
4 Results
Batch Renorm preserves batchnorm performance on standard minibatches while improving training with small or non-i.i.d. minibatches. Its advantages include faster convergence and recovery of validation accuracy under biased sampling, though larger normalization groups remain beneficial and some alternatives have scope limits.
- 4.2 Small minibatches: With normalization groups of 4, Batch Renorm reached 76.5% validation accuracy at 130k steps, versus batchnorm’s 74.2% at 210k steps.The gradient was still aggregated over 1600 examples per step, while normalization used groups of 4 rather than 32.
- 4.2 Small minibatches: Normalizing over groups of 32 examples performed better than normalizing over groups of 4, so Batch Renorm did not eliminate the benefit of larger groups.Batch Renorm improved small-minibatch training but did not fully remove the accuracy advantage of larger normalization groups.
- 4.3 Non-i.i.d. minibatches: Under non-i.i.d. sampling, batchnorm achieved 67% test accuracy, while training accuracy reached 72.8% and later dropped consistently with overfitting.Each minibatch sampled 16 labels and two images per label, creating a biased distribution relative to ordinary i.i.d. sampling.
- 4.3 Non-i.i.d. minibatches: Splitting biased minibatches into two more i.i.d. halves improved batchnorm test accuracy to 77.4% at 140k steps, but the method applies only when examples per label are few.The number of examples per label determines how many microbatches the minibatch must be split into.
- 4.3 Non-i.i.d. minibatches: Batch Renorm achieved 78.5% test accuracy at 120k steps on non-i.i.d. minibatches, matching the equivalent i.i.d. result and avoiding image-set overfitting.The model used ordinary minibatches of size 32, enabling inference that effectively classifies individual images.
5 Conclusions
Batch Renormalization addresses batchnorm’s limitations on small or non-i.i.d. minibatches by making outputs depend only on individual examples during training and inference. It preserves batchnorm-like implementation and speed while introducing additional hyperparameters and leaving several applications for future exploration.
- Batch Renormalization ensures model outputs depend only on individual examples rather than the entire minibatch during both training and inference.
- A per-dimension correction matches activations between training and inference, is identity in expectation, and participates directly in training.
- Batch Renormalization runs at the same speed as batchnorm during training and inference and is comparably easy to implement.
- The method introduces extra hyperparameters for moving-average updates and correction-limit schedules, whose effects require more investigation.
- Batch Renormalization may improve models normally using batchnorm, including residual networks and generative adversarial networks.The passage presents these applications as potential benefits rather than established results.
- Recurrent networks remain an area for further exploration, where shared running averages could normalize all timesteps.