Source-linked AI summary
Group Normalization
Yuxin Wu, Kaiming He
TL;DR
Batch Normalization becomes unreliable with small batches, constraining memory-intensive vision models and transfer to detection, segmentation, and video tasks. This paper introduces Group Normalization, which normalizes channels within groups independently of batch size, and finds it competitive with or better than Batch Normalization across ImageNet, COCO, and Kinetics, including 10.6% lower error at batch size 2.
Problem
Batch Normalization requires sufficiently large batches for accurate statistics, limiting memory-constrained model training and computer vision tasks that use small batches.
Method
Group Normalization divides channels into groups and normalizes features within each group without using the batch dimension.
Results
GN is competitive with or outperforms BN across ImageNet, COCO detection and segmentation, and Kinetics video classification, achieving 10.6% lower error for ResNet-50 at batch size 2.
Takeaways & Limitations
GN is a competitive alternative to BN across image classification, object detection, segmentation, and video classification, while remaining stable across batch sizes.
Takeaways & Limitations
GN-based systems may not be optimally configured because existing architectures and hyperparameters were designed for BN.
Abstract
from arXiv · showhide
Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.
1. Introduction
The introduction identifies BN’s dependence on sufficiently large per-worker batches as a limitation for memory-intensive vision tasks and presents GN as a batch-independent alternative. GN groups channels for normalization, remains stable across batch sizes, and transfers effectively across training settings and tasks.
- Motivation: BN requires sufficiently large per-worker batches because small batches produce inaccurate statistics and sharply increase model error.The paper defines batch size as samples per worker, with statistics computed independently on each worker.
- Motivation: Detection, segmentation, and video recognition often use very small batches because higher-resolution or spatiotemporal features increase memory demands.Fast/er and Mask R-CNN use batch sizes of 1 or 2 images, where BN is frozen.
- Method: GN divides channels into groups and normalizes features within each group without exploiting the batch dimension.The paper presents GN as a simple alternative to BN and relates its group-wise design to classical features such as SIFT and HOG.
- Results: 10.6% lower error: GN outperforms its BN counterpart for ResNet-50 on ImageNet at batch size 2.With regular batch sizes, GN is comparably good as BN, with a gap of ∼0.5%, and outperforms other normalization variants.
- Related work: GN avoids the limited visual-recognition success reported for Layer Normalization and Instance Normalization and can replace both methods.LN and IN are described as effective for sequential or generative models, while GN presents better visual-recognition results.
2. Related Work
Prior normalization methods either rely on batch statistics or avoid the batch dimension with alternative normalization axes, but each has limitations in small-batch or changing-distribution settings. GN instead inherently avoids batch-statistics computation while remaining a generic layer unrelated to group convolutions.
- Normalization: BN’s precomputed inference statistics can become inconsistent when target distributions change, while smaller batches dramatically affect estimated batch statistics.These issues create inconsistency across training, transferring, and testing.
- Normalization: LN, IN, and WN avoid batch-dimension issues by normalizing channels, individual samples, or filter weights, respectively.The cited passage states that these methods avoid batch-dimension problems but does not complete its comparison of their accuracy.
- Addressing small batches: BR improves BN accuracy for small batches by constraining estimated means and variances, but remains batch-dependent and degrades as batch size decreases.BR introduces two extra parameters to reduce mean and variance drift.
- Addressing small batches: Synchronized BN computes statistics across multiple GPUs, but shifts small-batch requirements to engineering and hardware demands and prevents asynchronous solvers.The required number of GPUs is proportional to BN’s requirements.
- Addressing small batches: GN inherently avoids batch-statistics computation rather than modifying or distributing that computation.This distinguishes GN from approaches such as BR and synchronized BN.
- Group-wise computation: Although GN divides channels into groups like several grouped-convolution methods, it does not require group convolutions and functions as a generic layer in standard ResNets.Related methods include ResNeXt, MobileNet, Xception, and ShuffleNet.
3. Group Normalization
Group Normalization treats channels as structured, potentially interdependent groups and normalizes features using statistics computed within each group rather than across the batch. It generalizes Layer Normalization and Instance Normalization through the group count while retaining per-channel scale and shift parameters.
- Motivation: Visual features can exhibit group-wise structure and interdependent channels arising from orientations, frequency, shapes, illumination, and textures.Classical descriptors use group-wise normalization, while higher-level feature coefficients may remain dependent within groups.
- Group Norm: Pixels in the same group share the same normalization statistics, while GN learns per-channel γ and β to restore representational flexibility.GN is defined within the general feature-normalization formulation using the group-specific set S_i.
- Group Norm: GN divides channels into G groups and computes mean and standard deviation over spatial dimensions and the C/G channels within each group.The default setting is G = 32, and channels are assumed to be stored sequentially along the channel axis.
- Relation to Prior Work: GN operates independently along the batch axis, becomes LN when G = 1, and becomes IN when G = C.Unlike IN, GN can exploit channel dependence because groups contain multiple channels.
- Implementation: GN can be implemented in a few lines in PyTorch and TensorFlow by specifying the axes used to compute mean and variance.Automatic differentiation is supported in both frameworks.
4. Experiments
Experiments show that GN matches BN when batch statistics are reliable and remains substantially more stable as batch sizes shrink. GN also transfers effectively to detection, segmentation, and video classification, often outperforming BN-based alternatives.
- Feature normalization methods: At batch size 32, GN approaches BN with only 0.5% higher validation error, while LN is 1.7% worse and IN is 4.8% worse than BN.All normalization methods converge in this regime, where BN provides a strong baseline.
- Small batch sizes: 10.6% lower error: at batch size 2, GN reaches 24.1% error versus BN’s 34.7%, while GN remains stable from batch sizes 32 to 2.BN becomes considerably worse with small batches because batch mean and variance estimates are stochastic and inaccurate.
- Normalization variants: At batch size 4, GN achieves 24.2% error, outperforming BR at 26.3% and BN at 27.3%.With two channels per group, GN also beats IN, yielding 25.6% versus 28.4% error.
- Deeper models: On ResNet-101 with batch size 2, GN has 23.0% error, 8.9% better than BN’s 31.9%, while at batch size 32 it is only 0.4% worse.The batch-size-32 validation errors are 22.4% for GN and 22.0% for BN.
- VGG models: On VGG-16, GN is 0.4% better than BN, with qualitatively similar feature distributions and lower training error.Both normalized variants differ substantially from the no-normalization model.
- Object detection and segmentation: GN improves detection by 0.9 AP in the box head and another 0.5 AP from the backbone, while the final long-schedule model gains 2.2 box AP and 1.6 mask AP over BN*.GN normalization in the head accounts for a substantial portion of the improvement, and longer training increases iterations from 180k to 270k.
- Video classification: In Kinetics with 32-frame inputs and batch size 4, GN reaches 72.8 / 90.6 top-1/top-5 accuracy versus BN’s 72.1 / 90.0.GN’s accuracy remains similar from batch size 8 to 4, whereas BN decreases by 1.2% and shows a noticeable error-curve gap.
5. Discussion and Future Work
The authors conclude that GN is effective across applications but may benefit from systems and hyper-parameters redesigned specifically for it. They propose studying GN in recurrent, generative, and reinforcement-learning settings.
- Limitations and future work: GN is an effective normalization layer that does not exploit the batch dimension, but existing systems and hyper-parameters designed for BN may not be optimal for GN.The authors suggest that redesigning systems or searching for new GN-specific hyper-parameters could improve results.
- Limitations and future work: GN’s relationship to LN and IN motivates future study in recurrent and generative models, along with representation learning for reinforcement-learning tasks.The authors note that BN is important for training very deep reinforcement-learning models.