PaperlayerView exact revision on arXiv

Source-linked AI summary

Switchable Whitening for Deep Representation Learning

Xingang Pan, Xiaohang Zhan, Jianping Shi, Xiaoou Tang, Ping Luo

arXiv:1904.09739v4cs.CV

TL;DR

Normalization methods either standardize features or whiten them, but existing approaches were typically designed and applied separately for specific tasks. The paper proposes Switchable Whitening, which learns to combine and select these operations, achieving consistent improvements across classification, segmentation, domain adaptation, and style transfer benchmarks.

  • Problem

    Existing whitening techniques were applied separately to different tasks and layers, while standardization leaves feature correlations intact, complicating model design and limiting shared benefits.

  • Method

    Switchable Whitening unifies whitening and standardization methods and learns importance weights to select their contributions in different CNN layers and tasks.

  • Results

    SW consistently outperforms counterparts across classification, segmentation, domain adaptation, and image style transfer, including improvements of 1.51%, 3.2%, 4.1%, and 3.0% over BN-based baselines.

  • Takeaways & Limitations

    SW adapts normalization choices to task and dataset characteristics while providing a tool for analyzing whitening and standardization techniques.

Abstract

from arXiv · show

Normalization methods are essential components in convolutional neural networks (CNNs). They either standardize or whiten data using statistics estimated in predefined sets of pixels. Unlike existing works that design normalization techniques for specific tasks, we propose Switchable Whitening (SW), which provides a general form unifying different whitening methods as well as standardization methods. SW learns to switch among these operations in an end-to-end manner. It has several advantages. First, SW adaptively selects appropriate whitening or standardization statistics for different tasks (see Fig.1), making it well suited for a wide range of tasks without manual design. Second, by integrating benefits of different normalizers, SW shows consistent improvements over its counterparts in various challenging benchmarks. Third, SW serves as a useful tool for understanding the characteristics of whitening and standardization techniques. We show that SW outperforms other alternatives on image classification (CIFAR-10/100, ImageNet), semantic segmentation (ADE20K, Cityscapes), domain adaptation (GTA5, Cityscapes), and image style transfer (COCO). For example, without bells and whistles, we achieve state-of-the-art performance with 45.33% mIoU on the ADE20K dataset. Code is available at https://github.com/XingangPan/Switchable-Whitening.

1. Introduction

Existing normalization methods either standardize features without decorrelation or whiten them, but prior techniques were applied separately across tasks and layers. Switchable Whitening unifies these operations, learns their task-dependent mixture, and improves performance across diverse vision benchmarks.

  • Standardization centers and scales features but leaves correlations between activations, whereas whitening also decorrelates them.
  • Existing whitening techniques were applied separately to different tasks and often assigned to different CNN layers, complicating model design and limiting shared benefits.
  • Switchable Whitening integrates BW, IW, BN, IN, and LN, learning importance weights that control each technique’s contribution end-to-end.
  • Semantic segmentation favors BW and BN, while domain adaptation selects more IW and IN; image classification and style transfer rely more heavily on IW.
  • 1.51%, 3.2%, 4.1%, and 3.0% improvements over BN-based baselines were reported for ImageNet, ADE20K, Cityscapes, and GTA5→Cityscapes, respectively.
  • SW is presented both as a CNN building block that outperforms counterparts across multiple benchmarks and as a tool for analyzing normalization characteristics.

2. Related Work

Related normalization methods mainly standardize activations using statistics from batches, instances, or layers, while whitening methods additionally decorrelate activations. Switchable Whitening generalizes these approaches, with Switchable Normalization as a special case.

  • BN, IN, and LN standardize activations using statistics computed over mini-batches, individual channels, and all channels in a layer, respectively.
  • Standardization methods center and scale activations but leave correlations between activations, which can lead to suboptimal optimization efficiency.
  • Whitening methods use covariance information to decorrelate activations and can improve optimization efficiency compared with standardization alone.
  • Switchable Whitening provides a general form integrating whitening and standardization techniques, with Switchable Normalization as a special case.

3. Switchable Whitening (SW)

Switchable Whitening unifies whitening and standardization by combining statistics from multiple normalizers and learning their importance during training. It can adapt normalization behavior across tasks while retaining practical CNN integration and computational efficiency.

  • General form: Batch Whitening whitens an entire mini-batch, whereas Instance Whitening computes statistics per sample and whitens each sample separately.Both transformations add ϵI to the covariance matrix, with φ(X)φ(X)^T = I for BW and φ(Xn)φ(Xn)^T = I for IW.
  • General form: Standardization appears as a special case of the general transformation when covariance off-diagonal elements are set to zero.The resulting diagonal covariance operation divides each channel by its standard deviation without decorrelating channels.
  • Formulation of SW: SW combines whitening and standardization statistics, using learned importance ratios to switch among alternative normalizers.The formulation can include BW and IW alone or BW, IW, BN, IN, and LN together; weights are generated through softmax control parameters.
  • Formulation of SW: SW is more general than Switchable Normalization because it can use the full covariance matrix rather than only its diagonal.This enables decorrelated features and can be extended to other normalization methods, although such extensions are outside this work’s scope.
  • Training and implementation: SW parameters and CNN parameters are optimized jointly by backpropagation, with running batch statistics used during inference and instance statistics computed per sample.The implementation adds scale and shift parameters γ and β after the whitening or normalization transform.
  • Acceleration: Newton’s iteration offers a faster alternative to GPU singular value decomposition for computing the inverse square root of the unified covariance.The authors set T = 5 and report similar performance to the SVD version.
  • Analysis and discussion: Instance Whitening can provide stronger appearance invariance than Instance Normalization because it standardizes and whitens each sample to the identity covariance.Switching toward standardization reduces decorrelation, allowing the whitening extent to be adjusted; group whitening reduces computation by C/G times.

4. Experiments

Experiments evaluate SW across classification, segmentation, domain adaptation, and style transfer, showing task-dependent normalizer selection and consistent performance gains. SW also reduces domain discrepancy and can improve advanced segmentation models while iterative whitening lowers computational cost.

  • SW is evaluated on image classification, semantic segmentation, domain adaptation, and image style transfer benchmarks.
  • 4.1. Classification: 1.51% and 1.09% reductions in ImageNet top1 and top5 error are achieved by ResNet50 with only 7 SWb layers.This performance is comparable with the original ResNet152, which has 5.94% top5 error.
  • 4.1. Classification: Combining normalization methods surpasses individual normalizers, while instance whitening improves SWa over BW by introducing style invariance.SWa and SWb perform comparably well, indicating that full whitening generally performs well when whitening is included.
  • 4.2. Semantic Segmentation: 3.2% and 4.1% mIoU gains are obtained on ADE20K and Cityscapes by replacing part of ResNet50’s BN layers with SW.SW also outperforms SN and BW on these segmentation benchmarks.
  • 4.2. Semantic Segmentation: 45.33% mIoU on ADE20K is achieved with PSPNet101 using some SW layers, exceeding other advanced semantic segmentation methods.Iterative whitening significantly reduces GPU running time while maintaining performance comparable to the SVD version.
  • 4.3. Domain Adaptation: SWa improves GTA5-to-Cityscapes mIoU by 3.0% over BN and 1.6% over SN, while reducing MMD for shallow and deep features.The analysis attributes easier cross-domain generalization to instance whitening reducing domain discrepancy in CNN features.
  • Task-dependent selection: SW selects different normalizers by task: BW and BN dominate segmentation, IW is selected more in classification, and IW dominates style transfer.Under domain adaptation, SW selects more IW and IN; in style transfer, SW performs comparably well with IW while BN produces poor stylization.

5. Conclusion

The paper proposes Switchable Whitening, which unifies whitening and standardization techniques and learns to select appropriate normalizers across CNN layers and tasks. Experiments report consistent improvements across classification, segmentation, domain adaptation, and image style transfer.

  • Switchable Whitening integrates multiple whitening and standardization techniques in a general form.
  • SW adapts to different vision tasks by learning to select appropriate normalizers in different CNN layers.
  • SW achieves consistent improvements over previous normalization methods across classification, segmentation, domain adaptation, and image style transfer.
  • The analysis highlights the importance of leveraging different whitening methods in CNNs.

Appendix

The appendix contains supplementary material on SW back-propagation, network configurations, instance segmentation results, and style transfer visualizations.

  • The appendix derives the back-propagation procedure for Switchable Whitening.
  • It discusses network configurations used in the experiments.
  • It reports results for instance segmentation and provides additional style transfer visualizations.

A. Back-propagation of SW

The appendix presents SW's backward pass, computing gradients with respect to inputs and importance weights through derivatives of integrated covariance and mean. It derives these terms from the forward pass of ZCA whitening.

  • Algorithm 2 computes gradients with respect to SW inputs and importance weights during each iteration.
  • The backward pass first computes derivatives with respect to integrated covariance and mean for each minibatch sample.
  • The appendix derives the covariance and mean gradients using the forward pass of ZCA whitening.
  • The derivation uses intermediate variables and the chain rule to express the backward computations.
  • The Frobenius inner product is used in the derivation, while K is a zero-diagonal matrix and ⊙ denotes element-wise multiplication.
  • The derivative of the integrated mean is calculated as part of the backward-pass derivation.

B. Discussion for Network Configurations

The appendix describes where SW and competing normalization layers are inserted in experimental CNN architectures, with supplementary style-transfer and instance-segmentation materials identified.

  • B. Discussion for Network Configurations: For CIFAR-10/100, SW or counterpart normalization layers are inserted after selected convolution layers in ResNet residual modules.In ResNet20, the considered normalization layers are the 1st, 4th, 8th, 12th, and 16th layers.
  • B. Discussion for Network Configurations: The appendix includes a visualization of style transfer using different normalization layers.
  • B. Discussion for Network Configurations: Table 7 reports Mask R-CNN results using ResNet50 and FPN with a 2× learning-rate schedule.

C. Instance Segmentation

On instance segmentation, SW replaces selected ResNet50 backbone normalization layers while other components use synchronized batch normalization, outperforming SyncBN and GN.

  • SW significantly outperforms SyncBN and GN on instance segmentation.The evaluation uses Mask-RCNN with the COCO dataset; seven ResNet50 backbone normalization layers are replaced by SW.

D. Style Transfer Results

In image style transfer, SW produces comparably well-stylized images to IW and adapts better than BN to the task.

  • SW produces comparably well-stylized images to IW in image style transfer.Visualization examples compare stylizing networks using different normalization techniques, with BN performing worse than the other methods.
Loading 1904.09739v4…