PaperlayerView exact revision on arXiv

Source-linked AI summary

Universally Slimmable Networks and Improved Training Techniques

Jiahui Yu, Thomas Huang

arXiv:1903.05134v2cs.CVcs.AI

TL;DR

The paper addresses whether one network can run at arbitrary widths while handling batch normalization and efficient training. It proposes US-Nets with the sandwich rule and inplace distillation, reports similar or better performance than individually trained models across representative tasks, and extends the approach to nonuniform widths.

  • Problem

    Existing slimmable networks select widths from a predefined set, motivating the question of whether one network can run at arbitrary width while addressing batch-normalization and training challenges.

  • Method

    The paper trains universally slimmable networks using the sandwich rule and inplace distillation, with an approach applicable to networks with or without batch normalization.

  • Results

    US-Nets perform similarly or better than individually trained models across image classification, image super-resolution, and deep reinforcement learning, with ablations demonstrating the proposed methods' effectiveness.

  • Takeaways & Limitations

    A single network can support arbitrary or nonuniform widths, enabling runtime width adjustment and related FLOPs-Accuracy spectrum comparisons within the paper's demonstrated scope.

  • Takeaways & Limitations

    The theoretical analysis is based on a single hidden layer, leaving analysis of deep networks with nonlinear activation for future research.

Abstract

from arXiv · show

Slimmable networks are a family of neural networks that can instantly adjust the runtime width. The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. In this work, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. We further propose two improved training techniques for US-Nets, named the sandwich rule and inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

1. Introduction

The paper introduces universally slimmable networks that execute at arbitrary widths and presents training techniques addressing efficiency, batch normalization, and accuracy. Experiments span classification, super-resolution, and reinforcement learning, with extensions to nonuniform widths and FLOPs-Accuracy analysis.

  • Motivation and contribution: US-Nets address training challenges involving batch normalization, efficient optimization, and improving performance beyond individually trained networks.Batch normalization uses mini-batch statistics during training but moving averages during inference, creating inconsistencies across widths.
  • Training techniques: The sandwich rule trains the smallest and largest widths plus randomly sampled intermediate widths in each iteration.The method uses performance bounds at the smallest and largest widths to implicitly optimize intermediate widths.
  • Training techniques: Inplace distillation transfers knowledge from the full-network width to subnetworks within each training iteration.It adapts the two-step knowledge-distillation idea to jointly trained widths in a single US-Net.
  • Motivation and contribution: A single US-Net can execute at arbitrary width using a simple, general training approach.This extends earlier slimmable networks, whose widths were selected from a predefined set.
  • Experiments: The proposed methods are evaluated on image classification, image super-resolution, and deep reinforcement learning, with extensive ablation studies.The experiments examine width lower bound k0, width divisor d, sampled-width count n, and BN post-statistics subset size s.
  • Extensions: The approach also supports nonuniform US-Nets, where each layer independently adjusts its width ratio, and motivates FLOPs-Accuracy spectrum comparisons.These extensions are presented as applications beyond globally uniform width scaling.

2. Related Work

Prior work introduced slimmable networks for runtime width switching and addressed batch-normalization inconsistencies with switchable statistics. Knowledge distillation provides a related strategy for transferring information from larger or pretrained models to thinner ones.

  • Slimmable Networks: Slimmable networks use one neural network at different predefined widths to adapt accuracy-efficiency trade-offs at runtime.Their central training obstacle is inconsistent feature statistics across subnetworks sharing batch-normalization layers.
  • Slimmable Networks: Switchable batch normalization assigns independent normalization statistics to different subnetworks.This addresses discrepancies caused by aggregating different numbers of channels.
  • Knowledge Distilling: Knowledge distillation transfers learned information from a pretrained network to a new network through predicted features, soft targets, or both.The approach has been applied to computer vision, compression, reinforcement learning, and sequence learning.

3. Universally Slimmable Networks

US-Nets reinterpret feature aggregation so one network can execute across arbitrary widths, while addressing batch-normalization statistics that otherwise differ across sub-networks.

  • Rethinking Feature Aggregation: Feature aggregation is a weighted sum in which each input channel can act as a residual component for the output neuron.This view motivates adding channels progressively rather than treating each width as an unrelated network.
  • Rethinking Feature Aggregation: A wider network should have no worse performance than its slim counterpart because it can learn zero-valued connections for unused channels.The bounded residual-error argument supports execution at widths between predefined discrete choices.
  • Universally Slimmable Networks: US-Nets are designed to run at any width from a lower bound such as ⌈0.25n⌉ to n, regardless of the normalization layer used conceptually.Batch normalization still requires special treatment because training and inference use inconsistent statistics.
  • Post-Statistics of Batch Normalization: Batch normalization fails with shared statistics because different channel counts produce different feature means and variances.Switchable batch normalization stores separate statistics for each sub-network, but this becomes impractical for US-Nets.
  • Post-Statistics of Batch Normalization: US-Nets instead compute batch-normalization statistics for all widths after training, using moving averages or exact averages.Exact averages perform slightly better, while a randomly sampled subset such as 1k images can already estimate statistics accurately.

4. Improved Training Techniques

The proposed training procedure combines the sandwich rule with inplace distillation to train US-Nets efficiently across widths and transfer knowledge from the full network to smaller sub-networks.

  • Sandwich Rule: The sandwich rule trains the smallest width, largest width, and n−2 randomly sampled widths in each iteration.The smallest and largest widths represent the performance lower and upper bounds, while random widths provide intermediate coverage.
  • Inplace Distillation: Inplace distillation transfers knowledge from the full-network predictions to sub-networks during the same training iteration.The largest-width prediction becomes a detached soft target for sub-network losses.
  • Inplace Distillation: Unlike two-step distillation, inplace distillation is single-shot and adds no computation or memory cost.The technique is reported as applicable to image classification, image super-resolution, and deep reinforcement learning.
  • Training Universally Slimmable Networks: The overall US-Net algorithm uses the full network with ground-truth loss, then sampled sub-networks with detached full-network predictions, before one optimizer update.The procedure retains the same GPU memory cost and batch size as training individual networks.

5. Experiments

Experiments evaluate US-Nets across ImageNet classification, image super-resolution, and deep reinforcement learning, alongside ablations of their training techniques and design choices. US-Nets generally match or outperform individual models, while super-resolution shows a small performance gap under individual-model hyperparameters.

  • ImageNet Classification: ImageNet experiments evaluate non-residual MobileNet v1 and residual MobileNet v2 on 1000-class classification, comparing I-Net, S-Net, and US-Net under matched width configurations and FLOPs.I-Net denotes individually trained models, S-Net denotes 4-switch slimmable networks, and US-Net denotes universally slimmable networks.
  • ImageNet Classification: US-MobileNet v1 spans widths from [.25 : .025 : 1.0]× and US-MobileNet v2 spans [.35 : .025 : 1.0]×, with better average classification accuracy than individual and 4-switch slimmable baselines.The FLOPs-Accuracy spectrum is reported in Figure 3.
  • Image Super-Resolution: US-WDSR performs slightly worse than individually trained models, with only 0.01 lower PSNR in bicubic ×2 super-resolution.The experiment uses DIV2K and a WDSR network without batch normalization; the authors note that individual-model hyperparameters may not be optimal for slimmable models.
  • Deep Reinforcement Learning: US-Net consistently outperforms four individually trained networks on Atari BreakoutNoFrameskip-v4 using actor-critic PPO.Figure 5 reports mean episode reward with unsmoothed curves; the network has no batch normalization layer.
  • Ablation Study: The sandwich rule improves average performance while preserving good accuracy at the smallest and largest widths.The ablation compares random width sampling with variants that always include the minimum width, maximum width, or both.
  • Ablation Study: Inplace distillation significantly improves overall performance at no cost, and four sampled widths per iteration performs better than three while matching five.The authors use n = 4 by default; BN post-statistics can also be estimated accurately from a small randomly sampled subset.

6. Discussion

The discussion extends US-Nets to nonuniform per-layer widths and examines whether deep networks are naturally slimmable. It also considers output averaging as an optional conditional-parameter alternative.

  • Nonuniform Universally Slimmable Networks: Nonuniform US-Nets let each layer independently adjust its width ratio, supporting network-slimming applications.
  • Naturally Slimmable?: Deep neural networks are not naturally slimmable: naively trained models fail at different widths even after batch-normalization calibration.
  • US-Nets share batch-normalization scale and bias by default, while output averaging offers an option that mimics conditional parameters.

A. Discussion

The discussion examines nonuniform slimming, whether networks are naturally slimmable, and output averaging as an optional technique. Experiments show that trained US-Nets support broader width evaluation than fixed-width or limited-switch alternatives, while slimming different stages has different effects.

  • Nonuniform Universally Slimmable Networks: Nonuniform US-Nets let each layer independently adjust its width ratio, supporting network-slimming analyses beyond uniform scaling.The demonstration trains a nonuniform US-MobileNet v1 and applies an additional 0.6 ratio to individual resolution stages.
  • Nonuniform Universally Slimmable Networks: Slimming MobileNet v1 stage 5 is not a good choice according to the FLOPs-Accuracy spectrum.The spectrum compares models that apply different slimming strategies, including layer-specific width ratios.
  • Naturally Slimmable?: Naively trained and previously slimmable models achieve very low accuracy at arbitrary widths, even after batch-normalization statistics are calibrated.This result motivates the question of whether deep neural networks are naturally slimmable.
  • Naturally Slimmable?: An individually trained MobileNet v1 works well only at 1.0×, while a 4-switch model works well only at its four trained widths and fails at others.The individually trained model particularly fails below 200 MFLOPs; the 4-switch model uses [0.25, 0.5, 0.75, 1.0]×.
  • Improved Training and Implementation Techniques: Output averaging by input-channel count provides a slight performance gain and can be viewed as feature ensemble within each layer.The method is optional in US-Nets, should not be applied to depthwise convolutions, and may incur a switch cost when batch normalization is present.
Loading 1903.05134v2…