PaperlayerView exact revision on arXiv

Source-linked AI summary

Micro-Batch Training with Batch-Channel Normalization and Weight Standardization

Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, Alan Yuille

arXiv:1903.10520v2cs.CVcs.LG

TL;DR

Micro-batch training is important for memory-constrained vision tasks, but BN is limited when batches are small. The paper proposes WS and BCN to bring BN’s smoothing and protection from elimination singularities to this setting. Across reported tasks, the methods improve micro-batch performance and can match or outperform large-batch BN.

  • Problem

    BN’s effectiveness is limited for micro-batch training, although batches of only 1–2 images are needed for memory-constrained vision tasks.

    Source: §1 · p. 1

  • Method

    WS standardizes convolutional weights, while BCN combines batch and channel normalization using estimated activation statistics to address loss smoothness and elimination singularities.

    Source: §1 · pp. 1–2

  • Results

    WS and BCN improve micro-batch training across the reported tasks, and micro-batch WS-based methods can match or outperform large-batch BN.

    Source: §1 · p. 2, §9 · p. 13

  • Takeaways & Limitations

    GN+WS with batch size 1 can match or outperform large-batch BN on ImageNet, while replacing GN with BCN further improves results in both batch settings.

    Source: §1 · p. 2

    The actual smoothing effect of WS’s Lipschitz reductions depends on the statistics of the weights and gradients.

    Source: §5.2.1 · p. 5

Abstract

from arXiv · show

Batch Normalization (BN) has become an out-of-box technique to improve deep network training. However, its effectiveness is limited for micro-batch training, i.e., each GPU typically has only 1-2 images for training, which is inevitable for many computer vision tasks, e.g., object detection and semantic segmentation, constrained by memory consumption. To address this issue, we propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to bring two success factors of BN into micro-batch training: 1) the smoothing effects on the loss landscape and 2) the ability to avoid harmful elimination singularities along the training trajectory. WS standardizes the weights in convolutional layers to smooth the loss landscape by reducing the Lipschitz constants of the loss and the gradients; BCN combines batch and channel normalizations and leverages estimated statistics of the activations in convolutional layers to keep networks away from elimination singularities. We validate WS and BCN on comprehensive computer vision tasks, including image classification, object detection, instance segmentation, video recognition and semantic segmentation. All experimental results consistently show that WS and BCN improve micro-batch training significantly. Moreover, using WS and BCN with micro-batch training is even able to match or outperform the performances of BN with large-batch training.

1 INTRODUCTION

Micro-batch training is necessary for memory-constrained vision tasks, but BN works well only with sufficiently large batches and GN/LN have difficulty matching large-batch BN. The paper proposes WS and BCN to bring BN’s smoothing and protection from elimination singularities to micro-batch training.

1 INTRODUCTION

  • BN works well only with sufficiently large batches, while micro-batches of 1–2 images are often necessary for memory-constrained tasks such as object detection and semantic segmentation.GN and LN were designed for micro-batch training but have difficulty matching large-batch BN performance.Source: §1 · p. 1
  • BN’s zero-mean, unit-variance activations keep networks far from elimination singularities, where neurons become eliminated and waste computation while reducing effective model complexity.Approaching these singularities can harm training speed and final performance.Source: §1 · p. 1
  • WS standardizes convolutional weights, while BCN combines batch and channel normalization using estimated activation statistics; both are designed to work with large or micro-batches.The goal is to bring BN’s smoothing effects and ability to avoid elimination singularities into micro-batch training.Source: §1 · pp. 1–2
  • GN+WS with batch size 1 matches or outperforms large-batch BN on ImageNet, while GN+WS improves performance on micro-batch COCO and BCN further improves results.The authors also report WS and BCN improvements across image classification, detection, instance segmentation, video recognition, and semantic segmentation.Source: §1 · p. 2

2 RELATED WORK

Prior normalization methods operate on activations or weights, and the paper frames normalization through smoothness, singularities, robustness, and generalization. Its BCN combines batch and group normalization as a composite function based on analysis rather than automated search.

2 RELATED WORK

  • Activation-based methods normalize along different dimensions: BN across the batch, LN and GN across channels, and IN per sample.GN is described as a middle point between LN and IN.Source: §2 · p. 2
  • Weight-based alternatives include WN, which separates weight length and direction, and CWN, which also centers weights; WS instead standardizes weights to zero mean and unit variance without a learnable length.When combined with GN, WS narrows the performance gap between BN and GN.Source: §2 · p. 2
  • Normalization has been studied through smoothness and elimination singularities as well as robustness, generalization, regularization, gradient explosion, and learning-rate tuning.The paper also relates WS to converting constrained optimization into unconstrained optimization.Source: §2 · p. 2
  • Unlike methods that search for a combination of normalizations, BCN uses BN and GN together as a composite function informed by analysis and theoretical understanding.The cited alternatives include SN, SSN, and DN.Source: §2 · p. 2

3 LIPSCHITZ SMOOTHNESS AND ELIMINATION SINGULARITIES

The section motivates normalization through smoother optimization and distance from elimination singularities. It focuses on neurons that become constantly inactive, noting that a weight-zero criterion does not fit real-world networks well.

3.1 Lipschitz Smoothness

  • Training smooth functions with gradient descent can support faster learning, while nonsmooth loss landscapes can make optimization unstable and sensitive to learning rate and initialization.Smoother, more reliable gradients can permit larger steps and accelerate training.Source: §3.1 · p. 3

3.2 Elimination singularities

  • Elimination singularities are training-trajectory points where neurons become constantly deactivated; approaching such degeneracies can slow learning and affect performance.The paper focuses on this type of singularity among several forms associated with non-identifiability.Source: §3.2 · p. 3
  • A weight-based definition requiring zero incoming channel weights is unsuitable for real-world training: a trained ResNet-50 has a minimum-to-average channel-weight norm ratio of 0.55 despite weight decay.For ReLU layers, neurons can instead become constantly inactive when their maximum post-normalization value is below zero, leaving zero gradients that make them hard to revive.Source: §3.2 · p. 3

4 WEIGHT STANDARDIZATION

WS standardizes convolutional weights to address smoothness with respect to the parameters optimized by the optimizer. It differentiably controls weight moments per output channel, and differs from WN/CWN by removing learnable weight length and using standard deviation.

4 WEIGHT STANDARDIZATION

  • WS is motivated by extending BN’s smoothing effects from activations to convolutional weights, which are directly optimized; the paper argues the two smoothing effects are additive.The section states that BN considers Lipschitz constants with respect to activations rather than weights.Source: §4 · p. 3

4.1 Weight Standardization

  • WS reparameterizes convolutional weights as a function of trainable weights and applies the resulting standardized weights in convolution.It controls the first and second moments of each output channel’s weights differentiably, without an affine transformation, so gradients are normalized during back-propagation.Source: §4.1 · p. 3

4.2 Comparing WS with WN and CWN

  • Unlike WN’s learnable weight length and CWN’s centering operation, WS removes the learnable length and divides by the weights’ standard deviation.The paper reports that WS outperforms WN and CWN on large-scale tasks.Source: §4.2 · p. 4

5 THE SMOOTHING EFFECTS OF WS

WS normalizes gradients, lowering the Lipschitz constants of the loss and gradients; this makes training changes more bounded and supports longer optimizer steps.

5.1 WS normalizes gradients

  • Eq. 14 subtracts a weighted average from back-propagated gradients and scales them, while Eq. 15 zero-centers the resulting gradients.The paper describes these as gradient-normalization operations in WS.Source: §5.1 · p. 4

5.2 WS smooths landscape

  • WS makes loss and gradient changes more bounded by lowering their Lipschitz constants, supporting longer optimizer steps.The authors argue this can accelerate training.Source: §5.2 · p. 4

5.2.1 Effects of WS on the Lipschitz constant of the loss

  • With a following BN or GN layer, the 1/σ² scaling cancels; Eq. 14’s remaining gradient-norm reduction lowers the loss Lipschitz constant.The analysis assumes the convolutional layer is followed by a normalization layer.Source: §5.2.1 · pp. 4–5

5.2.2 Effects of WS on the Lipschitz constant of gradients

  • Eq. 12’s effect is limited by weight and gradient distributions, although it slightly improves training and is retained for low computational cost.In the ResNet-50/ImageNet study, the major improvements came from Eq. 11.Source: §5.2.2 · p. 5
  • Eq. 11 reduces the Hessian Frobenius norm, making convolution-weight gradients more predictable.The paper connects this predictability to a smoother, easier-to-optimize loss.Source: §5.2.2 · pp. 5–6

5.3 Connections to constrained optimization

  • The paper relates WS to constrained optimization but uses stochastic gradient descent with standardization rather than projected gradient descent.The authors describe the added complexity as negligible relative to deep-network training.Source: §5.3 · p. 6

6 WS’S EFFECTS ON ELIMINATION SINGULARITIES

This section examines how normalization relates to elimination singularities and shows that Weight Standardization (WS) reduces statistical differences between channels, helping keep models farther from those singularities.

6.1.1 Batch- and channel-based normalizations

  • Unlike BN, channel-based LN and GN normalize groups without batch information; GN uses several groups, while LN uses one group per layer.The authors describe multiple GN groups as a mechanism for ensuring each group has at least one neuron not suppressed by its peers.Source: §6.1.1 · p. 6, §6.1.4 · p. 8

6.1.2 BN avoids elimination singularities

  • BN normalizes each channel to zero mean and unit variance, preventing constant deactivation or underrepresentation from small activation scales.This holds regardless of the weights and input distribution.Source: §6.1.2 · p. 6

6.1.3 Statistical distance and its affects on performance

  • As normalized means and variances move farther from BN’s statistics, performance decreases and extreme settings can fail during training.The authors use distance from BN as a measure of distance to singularities.Source: §6.1.3 · p. 7

6.1.4 Statistics in Channel-based Normalization

  • With GN or LN, larger StatDiff indicates greater within-group channel differences and can lead to underrepresented channels; WS brings StatDiff closer to zero.The section links WS’s smaller statistical differences to channels being more frequently activated and at more similar activation scales.Source: §6.1.4 · pp. 7–8, Figure 6 · p. 7, Figure 7 · p. 8

6.2 WS helps avoiding elimination singularities

  • WS constrains convolutional weights to pass statistical similarities from input channels to output channels throughout training.The authors compare this reparameterization to Xavier or Kaiming initialization, which they describe as pursuing a similar objective.Source: §6.2 · p. 8

7 BATCH-CHANNEL NORMALIZATION

Batch-Channel Normalization (BCN) combines batch information with channel-based normalization. It is designed to use batch statistics in micro-batch settings while retaining the properties of channel normalization.

7 BATCH-CHANNEL NORMALIZATION

  • BCN adds batch information and constraints to channel-based normalization to address elimination singularities.The section presents BCN as building on WS’s role in keeping models away from those singularities.Source: §7 · p. 8

7.1 Definition

  • BCN normalizes features using batch and channel information, reshapes channels into groups, then reshapes the normalized output back to the original feature-map shape.The stated input and output shapes are both B×C×H×W.Source: §7.1 · p. 8

7.2.1 Large-batch training

  • For large batches, BCN estimates the needed batch statistics using a Batch Normalization layer.Only the estimated channel mean and standard deviation require information from more than one sample.Source: §7.2.1 · p. 9, §7.2 · p. 9

7.2.2 Micro-batch training

  • For micro-batches, BCN continually estimates channel means and standard deviations and normalizes features using those estimates, which are updated outside the loss gradients.The estimates are initialized to 0 and 1, respectively.Source: §7.2.2 · p. 9
  • Unlike the related estimate-based approach, BCN follows estimate-based normalization with channel normalization, which the authors say stabilizes it.They describe the two operations as complementary: channel normalization makes estimate-based normalization possible, while estimated batch statistics help channel normalization keep models away from singularities.Source: §7.2.2 · p. 9

7.3 Is Batch-Channel Normalization Redundant?

  • The authors argue BCN is not redundant: it brings batch knowledge without large batches, adds nonlinearity, and normalizes test samples individually.They associate individual test-time normalization with adapting to sample statistics and greater robustness to statistical changes in unseen data.Source: §7.3 · p. 9

8 EXPERIMENTAL RESULTS

The experiments evaluate WS and BCN on ImageNet. WS improves micro-batch results, while adding batch information through BCN improves accuracy and training speed over channel-based normalization baselines.

8.1.1 Weight Standardization

  • With batch size 1 per GPU, GN+WS achieves performance comparable to BN trained with large batches on ImageNet.GN showed the best results among the compared normalization methods that can train with one image per GPU.Source: §8.1.1 · p. 10
  • On ImageNet, WS achieves lower top-1 error rates than WN and CWN when the ResNet-50 backbone uses GN and batch size 1 per GPU.The comparison is reported in Table 3.Source: §8.1.1 · p. 10, Table 3 · p. 10
  • In the WS ablation, Eq. 11 is the major component bringing performance improvements.The authors say this agrees with the training dynamics and their Lipschitz analysis.Source: §8.1.1 · p. 10, Table 4 · p. 10
  • With batch size 1 per GPU, WS makes training ResNeXt-50 and ResNeXt-101 easier than GN alone in the reported setting.The GN group count was set to 32 without tuning its hyperparameters for ResNeXt.Source: §8.1.1 · p. 10, Table 5 · p. 10

8.1.3 Experiment settings

  • The ImageNet comparisons use SGD, matched parameter-update counts across batch sizes, and different batch and training-duration settings for ResNet-50 and ResNet-101.The specified training durations are 90 epochs for ResNet-50 and 100 epochs for ResNet-101.Source: §8.1.3 · pp. 10–11

8.1.2 Batch-Channel Normalization

  • Adding batch information to GN+WS improves accuracy and training speed, with BCN+WS achieving better results than BN baselines.The ResNet-50 training-error curves show a large drop at each epoch; the authors link this to models being farther from elimination singularities.Source: §8.1.2 · p. 10, Table 6 · p. 11, Figure 8 · p. 11

8.2 Image Classification on CIFAR

On CIFAR, the reported comparisons show BCN outperforming normalization baselines in both large- and micro-batch settings. Comparisons also include recent methods on ResNet-18.

8.2 Image Classification on CIFAR

  • On C10, BCN is reported better than BN in large-batch training and better than GN (with or without WS), which is designed for micro-batch training.Micro-batch training uses 1 sample per batch; large-batch training uses 128 samples.Source: §8.2 · p. 11, Table 7 · p. 11
  • On CIFAR-10, BCN also outperforms the reported SN and DN comparisons on ResNet-18 across large- and micro-batch settings.The table caption notes that the BN, SN, and DN micro-batch comparisons use two images per batch, while BCN uses one.Source: Table 8 · p. 11, §8.2 · p. 11

8.3 Object Detection and Instance Segmentation

On COCO, the authors report that adding WS to GN or using BCN trains object detection and instance segmentation models better, with larger improvements as network complexity increases. The section evaluates these methods in a memory-constrained setting using Mask R-CNN and Faster R-CNN.

8.3 Object Detection and Instance Segmentation

  • COCO detection and segmentation use 1–2 images per GPU because high image resolution constrains memory.ResNet-50 training uses two images per GPU; ResNet-101 uses one because it cannot fit in 12GB GPU memory.Source: §8.3 · pp. 11–12
  • WS added to GN or BCN improves COCO detection and instance segmentation, with larger gains as network complexity increases.The comparison covers bounding-box AP and mask AP for Mask R-CNN, and AP for Faster R-CNN.Source: §8.3 · p. 12, Table 9 · p. 12, Table 10 · p. 12

8.4 Semantic Segmentation on PASCAL VOC

On PASCAL VOC, changing normalization to BCN increases mIoU by about 2%; on Something-SomethingV1, WS improves TSM accuracy with both GN and BN. BCN improves on GN for video recognition but does not surpass BN.

8.4 Semantic Segmentation on PASCAL VOC

  • About 2% higher mIoU on PASCAL VOC results from changing normalization to BCN.The comparison uses DeepLabV3 with ResNet-101 and follows prior hyperparameters; the authors leave tuning for BCN or WS to future work.Source: §8.4 · p. 12, Table 11 · p. 12
  • PASCAL VOC evaluation uses a 10,582-image training set and tests without multi-scale or horizontal-flipping augmentation.The model is trained for 30K iterations with batch size 16 and image crop size 513.Source: §8.4 · p. 12

8.5 Video Recognition on Something-Something

  • The video comparison uses a reimplemented TSM-BN baseline with top-1/5 accuracy of 44.30/74.53.The reimplementation starts from ImageNet-pretrained models rather than Kinetics-pretrained models.Source: §8.5 · p. 13
  • WS increases TSM top-1 accuracy by about 2% with both GN and BN on Something-SomethingV1.GN+WS roughly catches up with BN, while BCN improves GN but does not surpass BN; WS also boosts BN accuracy to roughly match the reported 24-frame TSM ensemble.Source: Table 12 · p. 12, §8.5 · p. 13

9 CONCLUSION

The conclusion presents WS and BCN as methods for bringing two BN benefits to micro-batch training: smoother optimization and greater distance from elimination singularities. Across several vision tasks, the reported experiments show gains from WS and BCN, including cases where WS+GN matches or outperforms large-batch BN.

9 CONCLUSION

  • WS and BCN target micro-batch training by bringing in BN’s smoothing effect and ability to avoid harmful elimination singularities.WS standardizes convolutional weights, while BCN uses estimated batch statistics of convolutional activations.Source: §9 · p. 13
  • WS reduces the Lipschitz constants of the loss and gradients, smoothing the loss landscape.The authors also report that WS alleviates elimination-singularity risk, while BCN can push models farther from those singularities by incorporating estimated batch statistics.Source: §9 · p. 13
  • Across classification, detection, instance segmentation, video recognition, and semantic segmentation, WS and BCN improve micro-batch training.The conclusion also reports that replacing GN with BCN leads to further improvement.Source: §9 · p. 13
  • WS+GN with batch size 1 can match or outperform BN with large batch sizes.The conclusion reports this comparison alongside the broader micro-batch results.Source: §9 · p. 13

Related papers

Loading 1903.10520v2…