PaperlayerView exact revision on arXiv

Source-linked AI summary

Slimmable Neural Networks

Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, Thomas Huang

arXiv:1812.08928v1cs.CVcs.AI

TL;DR

Mobile and embedded applications need neural networks that adapt to varying runtime and resource budgets, but conventional width choices require separate models. The paper trains one shared network with switchable batch normalization, achieving similar or often better performance than individually trained models across classification and multiple vision tasks.

  • Problem

    Existing lightweight networks are not re-configurable across devices or changing latency, computation, and energy budgets without separately trained and deployed width variants.

  • Method

    Slimmable neural networks share parameters across widths and use independent batch normalization for each switch during joint training.

  • Results

    Slimmable models achieve similar or often better performance than individually trained models across ImageNet classification, object detection, instance segmentation, and person keypoint detection.

  • Takeaways & Limitations

    A single deployed model can adjust active channels on the fly to trade accuracy against runtime efficiency without downloading or offloading different models.

Abstract

from arXiv · show

We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

1 INTRODUCTION

Slimmable neural networks address runtime variability by letting one shared model adjust its width to trade accuracy against latency and resource budgets. Switchable batch normalization enables robust training across widths, with results matching or often exceeding individually trained models across classification and vision tasks.

  • Motivation: Mobile and autonomous applications need short response times, but devices have substantially different runtime characteristics.The paper cites over 24,000 unique Android devices in 2015.
  • Motivation: Existing width multipliers require separately training, benchmarking, deploying, and maintaining models for different device and budget conditions.Available computation and energy can also vary on the same device.
  • Approach: Slimmable networks share parameters across model variants while adjusting active channels at runtime for adaptive accuracy-latency trade-offs.A switch denotes a model variant, and 0.25× scales widths in all layers to one quarter of the full model.
  • Approach: Switchable batch normalization privately normalizes statistics for each width, addressing feature mean and variance discrepancies across switches.The method is motivated by naive multi-width training achieving around 0.1% testing accuracy on 1000-class ImageNet.
  • Results: Slimmable MobileNet, MobileNet v2, ShuffleNet, and ResNet-50 models achieve similar or often better performance than individually trained counterparts across classification, detection, segmentation, and keypoint tasks.The experiments also include an eight-switch MobileNet v1 without accuracy drop.

2 RELATED WORK

Related work reduces neural-network cost through pruning, adaptive computation, and conditional normalization. Slimmable networks fit this landscape by combining runtime width adjustment with shared-network training.

  • Model Pruning and Distilling: Model-pruning methods encourage sparsity in connections, channels, filters, or network depth to reduce computation.Unstructured sparsity often requires specialized software or hardware accelerators for speedups.
  • Adaptive Computation Graph: Adaptive computation methods use controllers, gates, or early exits to construct input-dependent computation graphs.Dynamic conditions can complicate layer fusion and memory optimization on mobile devices.
  • Conditional Normalization: Conditional normalization applies input-dependent scales and biases across networks and is used in tasks including style transfer and image recognition.The paper places batch and layer normalization within this broader feature-wise transformation approach.

3 SLIMMABLE NEURAL NETWORKS

The paper develops slimmable-network training through shared weights and switch-specific batch normalization. Naive and incremental alternatives expose why independent normalization and joint adaptation are needed.

  • 3.1 NAIVE TRAINING OR INCREMENTAL TRAINING: Naive shared training across widths is stable but reaches only around 0.1% top-1 accuracy on 1000-class ImageNet.The authors attribute the failure mainly to inconsistent feature statistics across width configurations.
  • 3.1 NAIVE TRAINING OR INCREMENTAL TRAINING: Incremental training raises MobileNet v2 accuracy from 60.3% for 0.35× to 61.0% for 0.5×, versus 65.4% for an individually trained 0.5× model.Adding connections between old and new parameters prevents their joint adaptation.
  • 3.2 SWITCHABLE BATCH NORMALIZATION: Switchable Batch Normalization assigns independent batch-normalization parameters to each switch and independently normalizes feature statistics during testing.This resolves feature aggregation inconsistency while allowing all switches to be jointly trained and all shared weights to update together.
  • 3.2 SWITCHABLE BATCH NORMALIZATION: S-BN adds negligible parameter overhead because batch-normalization layers are typically less than 1% of model size.Batch normalization can be re-fused into convolution layers after switching configurations.
  • 3.3 TRAINING SLIMMABLE NEURAL NETWORKS: Training optimizes the un-weighted sum of losses from all switches using a predefined switchable width list.The implementation cycles through widths, activates each width’s normalization parameters, computes its loss, and accumulates gradients.

4 EXPERIMENTS

Experiments evaluate slimmable networks on ImageNet and downstream COCO tasks across architectures, widths, and switch counts. The results show stable training with S-BN, performance comparable to individually trained models, and scalability to more runtime choices.

  • ImageNet classification: ImageNet experiments cover MobileNet v1, MobileNet v2, ShuffleNet, and ResNet-50 across multiple width configurations.Training settings follow the corresponding architecture papers for a fair comparison.
  • ImageNet classification: S-BN keeps testing error stable and preserves the ranking among switches throughout training, unlike naive shared training with high testing error.The naive approach is training-stable but performs poorly on validation data.
  • ImageNet classification: Slimmable models achieve similar performance to individually trained models at matching widths, with MobileNet v1 0.25× improving by 3.3%.The authors conjecture that joint training may provide implicit model distillation from larger to smaller switches.
  • ImageNet classification: The approach applies across residual and nonresidual architectures and supports convolutions, depthwise-separable convolutions, group convolutions, pooling, fully connected layers, residual connections, and concatenations.This broadens the training scheme beyond a single network family or layer type.
  • More switches in slimmable networks: An 8-switch S-MobileNet v1 retains similar performance to 4-switch and individually trained models, demonstrating scalability as the number of runtime choices increases.The evaluated 8-switch configuration spans widths from 0.25× to 1.0×.
  • Object detection, instance segmentation and keypoints detection: Slimmable networks are also evaluated for COCO bounding-box detection, instance segmentation, and keypoint detection using ResNet-50 backbones.The downstream experiments use established detection frameworks and compare individually trained with slimmable models.

5 VISUALIZATION AND DISCUSSION

The visualizations examine how shared channels and switch-specific batch-normalization parameters behave across widths. They show related but width-dependent feature semantics, with greater BN diversity emerging in deeper layers.

  • Visualization of top-activated images: Top-activated images for channel 3 9 are compared across different S-MobileNet v1 switches, with red outlines marking misclassified images.Rows correspond to different switches.
  • Visualization of top-activated images: As width increases, the same channel shifts from emphasizing white to emphasizing yellow while retaining a related color-recognition role.The quarter-sized model focuses more on white, whereas the full model focuses more on yellow.
  • Values of switchable batch normalization: Switchable BN parameters are close across switches in shallow layers but become diverse in deep layers.The reported BN quantities include mean, variance, scale, and bias.
  • Values of switchable batch normalization: The increasing BN discrepancy across depth indicates that the same channel develops slight semantic variations across switches.This connects switch-specific normalization with width-dependent learned features.

6 CONCLUSION

The paper introduces slimmable networks for runtime accuracy-efficiency trade-offs and uses switchable batch normalization to train them robustly. Across classification and several vision tasks, slimmable models match or exceed individually trained models.

  • Conclusion: Slimmable networks permit instant and adaptive accuracy-efficiency trade-offs at runtime.Switchable batch normalization facilitates robust training of the shared multi-width network.
  • Conclusion: Compared with individually trained models at the same widths, slimmable networks achieve similar or better performance on classification, object detection, instance segmentation, and keypoint detection.The conclusion states this pattern across the evaluated tasks.

A TRAINING ON IMAGENET

ImageNet experiments use training settings tailored to MobileNet v1, MobileNet v2, ShuffleNet, and ResNet-50, with reproduced full-width baselines reported for comparison.

  • A TRAINING ON IMAGENET: MobileNet v1 and v2 are trained for 480 epochs, while ShuffleNet uses 250 epochs and ResNet-50 uses 100 epochs.The settings also specify model-specific batch sizes and learning-rate schedules.
  • A TRAINING ON IMAGENET: The reproduced MobileNet v1 1.0×, MobileNet v2 1.0×, and ResNet-50 1.0× achieve top-1 accuracy within ±0.5% of reference results.
  • A TRAINING ON IMAGENET: ShuffleNet 2.0× has a reproduced top-1 error rate of 28.2%, 1.9% worse than the result reported by Zhang et al. (2017).The authors suggest inconsistency between mini-batch size and the number of training GPUs as a likely explanation.

B TRAINING ON COCO

COCO experiments use a PyTorch-style ResNet-50 backbone and MMDetection settings aligned with that implementation to compare individual and slimmable models.

  • B TRAINING ON COCO: The COCO experiments use PyTorch-style ResNet-50 models because the pretrained width variants for ImageNet classification use that style.This backbone differs slightly from the Caffe-style ResNet-50 used in Detectron: downsampling stride is placed in 3 × 3 rather than 1 × 1 convolutions.
  • B TRAINING ON COCO: Experiments are conducted in MMDetection with RCNN R50 FPN 1× hyper-parameter settings for fine-tuning individual and slimmable ResNet-50 models.The same settings are used for object detection and instance segmentation.
  • B TRAINING ON COCO: The reproduced ResNet-50 1.0× results are consistent with official MMDetection models.

C ABLATION STUDY OF CONDITIONAL PARAMETERS IN BN

The ablation examines conditional Batch Normalization parameters by comparing private versus shared scale and bias across slimmable MobileNet v1 variants.

  • C ABLATION STUDY OF CONDITIONAL PARAMETERS IN BN: Switchable Batch Normalization introduces private γ, β, µ, and σ2 parameters for each sub-network.
  • C ABLATION STUDY OF CONDITIONAL PARAMETERS IN BN: Batch Normalization maps input y to output y′ using learnable scale γ and bias β together with moving-average testing statistics µ and σ2.
  • C ABLATION STUDY OF CONDITIONAL PARAMETERS IN BN: After training, private scale and bias can be merged into γ′ and β′, preserving an affine form for the normalized output.The merged parameters are defined as γ′ = γ / √(σ2+ϵ) and β′ = β − γ′µ.
  • C ABLATION STUDY OF CONDITIONAL PARAMETERS IN BN: Table 6 compares top-1 ImageNet error rates for two S-MobileNet v1 models using private or shared scale and bias.
Loading 1812.08928v1…