PaperlayerView exact revision on arXiv

Source-linked AI summary

Differentiable Learning-to-Normalize via Switchable Normalization

Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, Jingyu Li

arXiv:1806.10779v5cs.CVcs.LG

TL;DR

Existing normalization practices often use one normalizer across a network even though different layers and tasks may favor different choices. SN learns layer-specific combinations of channel-, layer-, and minibatch-wise statistics, and it performs better or comparably across batch settings and challenging vision tasks. The approach is presented as an adaptable alternative to handcrafted normalization choices.

  • Problem

    Using one normalizer throughout a network can be suboptimal, while choosing different normalizers for different tasks makes model design cumbersome.

  • Method

    SN combines IN, LN, and BN statistics from channel, layer, and minibatch scopes, learning their importance weights for different normalization layers.

  • Results

    SN performs better or comparably to BN and GN across batch settings and improves normalization techniques on multiple challenging benchmarks and tasks.

  • Takeaways & Limitations

    SN provides an adaptable alternative in which each normalization layer can learn its own operation rather than relying on handcrafted uniform choices.

Abstract

from arXiv · show

We address a learning-to-normalize problem by proposing Switchable Normalization (SN), which learns to select different normalizers for different normalization layers of a deep neural network. SN employs three distinct scopes to compute statistics (means and variances) including a channel, a layer, and a minibatch. SN switches between them by learning their importance weights in an end-to-end manner. It has several good properties. First, it adapts to various network architectures and tasks (see Fig.1). Second, it is robust to a wide range of batch sizes, maintaining high performance even when small minibatch is presented (e.g. 2 images/GPU). Third, SN does not have sensitive hyper-parameter, unlike group normalization that searches the number of groups as a hyper-parameter. Without bells and whistles, SN outperforms its counterparts on various challenging benchmarks, such as ImageNet, COCO, CityScapes, ADE20K, and Kinetics. Analyses of SN are also presented. We hope SN will help ease the usage and understand the normalization techniques in deep learning. The code of SN has been made available in https://github.com/switchablenorms/.

1 INTRODUCTION

Existing normalization practices often use one normalizer throughout a network, although different layers and tasks may favor different choices. Switchable Normalization learns layer-specific combinations of normalizers, remains robust across batch settings, and improves performance across challenging benchmarks.

  • Motivation: Existing practices commonly use the same normalizer in every layer, although different normalizers suit different tasks.This makes model design cumbersome and can yield suboptimal performance.
  • Method: SN learns importance ratios to combine channel-wise, layer-wise, and minibatch-wise statistics from IN, LN, and BN.The ratios select normalizers end-to-end for different normalization layers.
  • Adaptation: SN adapts its normalizer selection across networks and tasks, favoring different combinations for classification, detection heads, style transfer, and neural architecture search.For example, it selects IN for artistic style transfer and gives larger LN weights to detection box and mask heads.
  • Results: 76.9% top-1 accuracy: SN surpasses BN by 0.5% and GN by 1.0% for ResNet50 on ImageNet with batch setting (8, 32).Across batch settings, SN performs better or comparably to both baselines.
  • Results: SN is reported to improve normalization techniques across ImageNet, COCO, Cityscapes, ADE20K, artistic stylization, neural architecture search, and Kinetics.The contribution includes applicability to CNNs and RNNs/LSTMs and analyses of normalization behavior.

2 SWITCHABLE NORMALIZATION (SN)

Switchable Normalization (SN) combines channel-, layer-, and minibatch-wise statistics, learning separate importance weights to select among IN, LN, and BN within normalization layers. Its formulation preserves comparable computational complexity while supporting joint end-to-end optimization and extensions such as sparse SN.

  • Formulation of SN: IN computes statistics over (H, W) per sample and channel, LN over (C, H, W) per sample, and BN over (N, H, W) per channel.These scopes produce 2CN, 2N, and 2C statistical values, respectively.
  • Formulation of SN: SN combines IN, LN, and BN statistics estimated channel-wise, layer-wise, and minibatch-wise, respectively, using learned importance weights.The weights for means and variances are learned through softmax-controlled parameters and back-propagation.
  • Formulation of SN: SN reduces redundant statistic computation by reusing dependencies among IN, LN, and BN, achieving O(NCHW) complexity comparable to previous methods.The paper states that LN and BN means and variances can be computed based on IN.
  • Formulation of SN: SN uses six scalar importance weights: three for weighted means and three for weighted variances, each set normalized across the selected statistics.The weights are shared across all channels.
  • Variants and inference: 77.0% top-1 accuracy is achieved by sparse SN on ImageNet with ResNet50, comparable to 76.9% for SN without sparsity.Sparse SN applies argmax to select one normalizer per normalization layer during fine-tuning.

3 RELATIONSHIPS TO PREVIOUS WORK

The paper compares SN with established normalizers in parameterization, hyper-parameters, statistics, computation, and breadth of application. SN combines richer statistics with comparable complexity and is demonstrated across diverse networks, tasks, and datasets.

  • Comparisons of normalization methods: SN has comparable parameter and computation counts to prior normalization methods while combining statistics from IN, LN, and BN.Its broader statistics are presented as an advantage without requiring the covariance computations used by BKN.
  • Comparisons of normalization methods: SN adds six importance weights to the usual 2C scale and shift parameters, whereas BKN uses C2 transformation parameters and WN uses C scale parameters without biases.The comparison concerns learned parameter types and counts.
  • Comparisons of normalization methods: GN and BRN require searching architecture- or network-dependent hyper-parameters, while many other methods use fixed, relatively insensitive p and ϵ values.WN is described as hyper-parameter-free because it normalizes in parameter space rather than feature space.
  • Comparisons of normalization methods: SN combines 2CN + 2N + 2C statistics from IN, LN, and BN, while BKN estimates C + C2 statistics and has the highest computation because it estimates a covariance matrix.The comparison distinguishes variance-vector statistics from covariance-matrix estimation.
  • Applications: SN is demonstrated across various networks, tasks, and datasets, including image recognition, detection, scene parsing, stylization, architecture search, and video recognition.The paper presents this breadth as wider application than existing normalizers.

4 EXPERIMENTS

Experiments evaluate SN across image classification, detection, segmentation, and video recognition, showing robust performance across batch sizes and tasks. Ablations indicate that SN adapts its learned normalizer weights to task and batch conditions.

  • Main results: SN generally matches or exceeds BN and GN across ImageNet, COCO, Cityscapes, ADE20K, and Kinetics benchmarks.The experiments cover classification, detection, instance segmentation, semantic parsing, and video recognition.
  • Image classification: At (8, 1), SN uses IN and LN without BN, while BN fails to converge; SN and GN remain reasonably effective.With minibatch size one, IN and BN are equivalent during training, so SN excludes BN.
  • Ablation study: Importance-weight histograms show SN favors BN with sufficiently large minibatches and LN with small minibatches, while remaining similar across training protocols.The weights are averaged across SN layers and adapt across tasks and batch settings.
  • Object detection and instance segmentation: 39.3 AP for Faster R-CNN with SN exceeds GN's 38.2 AP by 1.1 AP when finetuning the SN backbone.Using SN in the head with a frozen BN backbone also improves over the corresponding GN configuration from 37.2 to 38.0 AP.
  • Object detection and instance segmentation: SN improves Mask R-CNN over GN by 0.2 box AP and 0.3 mask AP with comparable pretrained backbones, and reaches 41.0 and 36.5 AP with an SN backbone.The corresponding GN results with the SN backbone are 40.2 box AP and 35.7 mask AP.
  • Semantic image parsing: SN achieves the best reported mIoU on ADE20K and Cityscapes, exceeding SyncBN by up to 2.1 points across the stated testing scales.On ADE20K, SN reports 38.7 versus 36.4 and 39.2 versus 37.7 for SyncBN; on Cityscapes, the margins over SyncBN are 1.5 and 2.1.
  • Video recognition: On Kinetics, SN exceeds BN and GN at batch size 4 by 1.2% and 0.5% in top-1 accuracy, respectively.SN already surpasses both baselines at batch size 8, which further improves its results.

5 DISCUSSIONS AND FUTURE WORK

SN learns different normalization operations in different layers and has been demonstrated across multiple computer-vision tasks. The analyses connect SN with balancing learning and generalization and with understanding normalization approaches.

  • SN learns different operations in different normalization layers of a deep network.
  • SN has been demonstrated in recognition, detection, segmentation, image stylization, and neural architecture search.
  • Analyses suggest that SN balances learning and generalization when training deep networks.
  • The work positions SN analysis as a way to understand normalization approaches and motivate extensions such as sparse SN and switchable whitening.

APPENDICES

SN using batch average converges faster and more stably than BN and SN using moving average, while both SN settings reach the same final accuracy.

  • 76.9% top-1 accuracy is reached by both SN with moving average and SN with batch average when training ResNet50 on ImageNet.
  • SN with batch average converges faster and more stably than BN and SN with moving average.
  • 76.9% for SN exceeds BN’s 76.4% under the reported ImageNet ResNet50 setting.
  • Across batch settings, SN with batch average performs better than SN with moving average.

B PROOF OF REMARK 1

The proof analyzes IN, LN, and BN through a common formulation and relates BN’s behavior to population normalization and adaptive gamma regularization.

  • For a normalized input, IN can be expressed with filter normalization and a learned scale and bias.
  • IN, LN, and BN share a common normalization formulation while differing in the pixels used to estimate their statistics.
  • BN can be reformulated as population normalization together with adaptive gamma regularization.
  • The coefficient ζ(h) is data-dependent, so BN’s adaptive gamma regularization depends on the training data.
  • The adaptive regularization imposes the constraint γ ≤ v on BN, whereas weight normalization has no regularization on v.

C.1 EXPERIMENTAL SETTING

The experiments evaluate normalization methods across batch sizes using standardized ImageNet training, and report that SN converges faster while retaining higher or comparable accuracy than BN and GN. The importance-weight analysis shows that SN can select statistics differently across parameters and layers.

  • Experimental Setting: ImageNet models use 1.2M training images, 50K validation images, SGD, and batch sizes represented as (number of GPUs, samples per GPU).
  • Experimental Setting: Models train for 100 epochs with learning-rate drops after epochs 30, 60, and 90, using data augmentation and center-crop top-1 accuracy.
  • Learning Curves: SN enables faster convergence while maintaining higher or comparable accuracies to BN and GN across the plotted settings.
  • Importance Weights: Importance weights for means and variances can differ notably, with SN reducing noisy BN variance statistics to stabilize training.
  • Importance Weights: Adjacent SN layers may choose different importance weights, showing layer-specific operation selection within a network.
  • Importance Weights: Deeper layers prefer LN and IN more than BN, indicating that uniform BN placement can impede performance.

D COCO DATASET

The COCO experiments evaluate SN across Faster R-CNN settings, including pretrained fine-tuning and training from scratch. SN supports competitive detection performance while enabling normalization-layer fine-tuning and avoiding ImageNet pretraining.

  • Experimental setup: SN is implemented in Faster R-CNN, Faster R-CNN+FPN, and Mask R-CNN+FPN using ResNet50 backbones across PyTorch and Caffe2-Detectron.The experiments use three detection settings and compare normalization methods within existing detection frameworks.
  • Results: Figure 8 compares average-precision learning curves for pretrained fine-tuning and training from scratch.The figure organizes results into pretrained and scratch-training conditions.
  • Experimental setup: The experiments train on COCO 2017 and report bounding-box and mask average precisions, including AP, AP.5, AP.75, and object-size metrics.Models are evaluated on the 2017 validation set using standard COCO metrics.
  • Results: 33.0 AP is achieved by SN when fine-tuning Faster R-CNN normalization layers, outperforming the compared trainable-normalization alternatives in this setting.The frozen-BN baseline reaches 29.6 AP, while fine-tuned BN reaches 19.3 AP.

E SEMANTIC IMAGE PARSING

Semantic parsing experiments test SN under small per-GPU minibatches on ADE20K and Cityscapes. An ablation shows that training data can change the normalizers selected by SN even when architecture and task remain fixed.

  • Experimental setup: DeepLab with a ResNet50 backbone is evaluated on ADE20K and Cityscapes using small minibatches of 2 and 1 samples per GPU, respectively.The experiments replace normalization layers while keeping the remaining settings fixed.
  • ADE20K: ADE20K uses SN (8, 2) pretrained on ImageNet, while SyncBN aggregates statistics across 8 GPUs for an effective batch size of 16.Images are resized to 450 × 450 and evaluated with multi-scale testing.
  • Ablation study: Figure 9 plots SN importance weights across 53 ResNet50 layers for ADE20K and Cityscapes.The plotted weights for the two datasets correspond to SN (8, 2).
  • Cityscapes: Cityscapes fine-tunes pretrained ResNet50 models with batch size 8 and uses 713 × 713 random crops.SN is fine-tuned from the (8, 2) model and evaluated at multiple inference scales.
  • Ablation study: Different training data lead SN to choose different normalizers even when the models and tasks are the same.This conclusion is drawn from comparing importance weights on ADE20K and Cityscapes.

F ARTISTIC IMAGE STYLIZATION

The artistic stylization experiments compare SN with IN and BN using VGG16 on COCO. SN converges faster than both alternatives while automatically selecting IN and producing comparable stylization results.

  • Task and setup: Artistic stylization jointly minimizes feature-reconstruction and style-reconstruction losses, and prior work found IN better than BN for this task.The losses respectively penalize content deviation and differences in style such as color, texture, and exact boundaries.
  • Task and setup: SN, IN, and BN are compared with VGG16 on COCO using 256×256 training images, batch setting (1, 4), and 512×512 test images.Models are trained for 40,000 iterations without weight decay or dropout.
  • Results: SN converges faster than both IN and BN on the style and feature reconstruction losses.Figure 10(a) compares the loss curves during training.
  • Results: SN automatically selects IN for image stylization while achieving comparable stylization results to IN.Figure 11 visualizes content, style, IN outputs, and SN outputs.

G NEURAL ARCHITECTURE SEARCH

The ENAS experiments apply SN within an LSTM controller for convolutional-cell architecture search. SN combines LN and GN in this setting and achieves higher CIFAR-10 validation accuracy than either alternative.

  • ENAS setup: ENAS trains an LSTM controller with REINFORCE to sample convolutional-cell architectures and evaluates them by CIFAR-10 validation accuracy.A child CNN is trained by back-propagation with SGD, while higher validation accuracy indicates a better sampled architecture.
  • ENAS setup: SN is compared with LN and GN inside the LSTM controller because BN is not applicable to LSTMs and IN is equivalent to LN in fully connected layers.SN therefore combines LN and GN for this experiment.
  • Results: SN obtains better CIFAR-10 validation accuracy than both LN and GN in the architecture-search experiment.Figure 10(b) reports the validation-accuracy comparison.
  • SN computation: SN's output layer applies affine scaling and shifting after normalization, using γ and β on the normalized activation.The cited back-propagation description represents the SN output as γ times the normalized activation plus β.
Loading 1806.10779v5…