PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Slimmable Network

Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, Xiaojun Chang

arXiv:2103.13258v1cs.CV

TL;DR

Dynamic sparse filters reduce theoretical computation but often fail to accelerate real hardware because indexing, copying, and masking are costly. DS-Net dynamically slices contiguous filters with a double-headed gate and trains the supernet and gate in separate stages. On ImageNet, it achieves 2-4× computation reduction and 1.62× real-world acceleration with minimal accuracy drops, while outperforming static and dynamic compression methods.

  • Problem

    Dynamic sparse patterns can reduce theoretical computation but fail to provide real-world acceleration because hardware must perform indexing, weight-copying, or zero-masking.

  • Method

    DS-Net dynamically slices statically contiguous filters using a double-headed gate, then separately trains the slimmable supernet and gate with IEB and SGS.

  • Results

    2-4× computation reduction and 1.62× real-world acceleration are achieved over ResNet-50 and MobileNet with minimal accuracy drops on ImageNet.

  • Takeaways & Limitations

    DS-Net consistently outperforms static counterparts and state-of-the-art static and dynamic compression methods across ImageNet, CIFAR-10, and VOC evaluations.

Abstract

from arXiv · show

Current dynamic networks and dynamic pruning methods have shown their promising capability in reducing theoretical computation complexity. However, dynamic sparse patterns on convolutional filters fail to achieve actual acceleration in real-world implementation, due to the extra burden of indexing, weight-copying, or zero-masking. Here, we explore a dynamic network slimming regime, named Dynamic Slimmable Network (DS-Net), which aims to achieve good hardware-efficiency via dynamically adjusting filter numbers of networks at test time with respect to different inputs, while keeping filters stored statically and contiguously in hardware to prevent the extra burden. Our DS-Net is empowered with the ability of dynamic inference by the proposed double-headed dynamic gate that comprises an attention head and a slimming head to predictively adjust network width with negligible extra computation cost. To ensure generality of each candidate architecture and the fairness of gate, we propose a disentangled two-stage training scheme inspired by one-shot NAS. In the first stage, a novel training technique for weight-sharing networks named In-place Ensemble Bootstrapping is proposed to improve the supernet training efficacy. In the second stage, Sandwich Gate Sparsification is proposed to assist the gate training by identifying easy and hard samples in an online way. Extensive experiments demonstrate our DS-Net consistently outperforms its static counterparts as well as state-of-the-art static and dynamic model compression methods by a large margin (up to 5.9%). Typically, DS-Net achieves 2-4x computation reduction and 1.62x real-world acceleration over ResNet-50 and MobileNet with minimal accuracy drops on ImageNet. Code release: https://github.com/changlin31/DS-Net .

1. Introduction

Dynamic inference can reduce input-dependent computation, but dynamic sparse filters have not delivered practical acceleration because hardware must handle irregular indexing, copying, or masking. DS-Net addresses this gap by dynamically slicing contiguous filters and training a predictive gate with a disentangled two-stage scheme.

  • Motivation: Dynamic inference routes inputs through different architectures according to their difficulty, potentially improving the accuracy–complexity trade-off over static NAS or pruning solutions.The proposed motivation contrasts adaptive routing with architectures optimized over the whole dataset.
  • Problem: Dynamic-width pruning has not achieved real-world acceleration because varying sparse patterns require zero masking, inefficient indexing, or repeated weight copying.These hardware costs create a gap between theoretical computation reduction and practical acceleration.
  • DS-Net: DS-Net dynamically adjusts filter numbers at test time while keeping filters statically and contiguously stored through dynamic slicing.The design targets hardware efficiency without the extra burden associated with dynamic sparsity.
  • DS-Net: A double-headed dynamic gate uses attention and slimming heads to predict network width with negligible extra computation cost.The gate adaptively controls the width of each supernet stage through a predicted slimming ratio.
  • Training: DS-Net separates supernet and gate optimization into two stages using In-place Ensemble Bootstrapping and Sandwich Gate Sparsification.The first technique improves weight-sharing supernet training, while the second identifies easy and hard samples online for gate training.

2. Related works

Anytime neural networks execute sub-networks under different budgets, while dynamic neural networks adapt their architecture to input data. Related work spans variable depth, variable width, and one-shot methods that separately train supernet parameters and routing agents.

  • Anytime neural networks: Anytime neural networks are single networks that execute sub-networks under different budget constraints for adaptive deployment.They have been studied through variable-depth and variable-width architectures.
  • Architecture dimensions: Variable-depth networks benefit from nested depth structures and residual or dense connections, while variable-width networks include slimmable networks.Slimmable networks use switchable batch normalization and in-place distillation to improve performance across widths.
  • Dynamic inference: Dynamic neural networks change their architectures according to input data to reduce average inference cost across inputs with different difficulty levels.This distinguishes dynamic inference from fixed architectures selected for an entire dataset.
  • One-shot methods: One-shot methods disentangle supernet-parameter training from architecture-routing-agent optimization, freezing supernet weights before optimizing the agent.This separation provides the training precedent for DS-Net’s two-stage optimization.

3. Dynamic Slimmable Network

DS-Net combines a slimmable supernet, hardware-efficient dynamic channel slicing, and a double-headed gate to select input-dependent widths. A disentangled two-stage training process uses ensemble bootstrapping for the supernet and sparsification to prevent gate collapse.

  • Dynamic Slimmable Network: DS-Net learns a slimmable supernet and dynamic gating mechanism that routes each input through sub-networks with different widths.The supernet performs the main task, while gates select widths at each stage.
  • Sandwich Gate Sparsification: Sandwich Gate Sparsification trains the gate with classification and complexity objectives after the supernet is trained and its weights are fixed.SGS addresses the tendency of the gate to collapse into a static policy during multi-objective optimization.
  • Supernet and Dynamic Channel Slicing: Dynamic channel slicing selects a contiguous prefix of filters, replacing hardware-incompatible sparse indexing with dense matrix multiplication.The slice operation keeps filters static and contiguous, supporting practical acceleration.
  • In-place Ensemble Bootstrapping: In-place Ensemble Bootstrapping trains slimmer sub-networks using soft targets from an exponential-moving-average target network and multiple width-specific teachers.The EMA target network supplies stable representations, while widest, slimmest, and random-width sub-networks form the teacher ensemble.
  • Double-headed Design: The double-headed gate combines a soft channel-attention head with a hard channel-slimming head to adjust features and predict discrete widths.The slimming head maps encoded features to candidate ratios, while the attention head is activated during supernet training.

4. Experiments

Experiments evaluate DS-Net on ImageNet, CIFAR-10, and VOC, including architecture configurations, transfer learning, ablations, and dynamic gate behavior. DS-Net reduces computation while maintaining or improving accuracy and shows input-dependent routing.

  • Experimental setup: DS-ResNet and DS-MBNet use representative heavy and lightweight architectures, with gates inserted according to each network’s residual or depthwise structure.DS-ResNet controls projection convolutions in residual blocks, whereas DS-MBNet uses one slimming gate after its fifth depthwise separable block.
  • ImageNet results: DS-Net consistently outperforms recent static pruning, dynamic inference, and NAS methods across ImageNet complexity settings.Table 2 compares pruning, dynamic inference, architecture search, and DS-Net methods on ImageNet.
  • ImageNet results: 2-4× computation reduction is achieved over ResNet-50 and MobileNetV1 with accuracy changes ranging from 0% to -1.5% for ResNet and +0.9% to -0.8% for MobileNet.Real-world DS-MBNet acceleration reaches 1.17× and 1.62×, compared with ideal channel-scaled MobileNetV1 acceleration of 1.31× and 1.91×.
  • ImageNet results: DS-MBNet surpasses AutoSlim and US-Net by 2.2% and 5.9%, while DS-MBNet-M exceeds dynamic pruning method CG-Net by 2.5%.The comparisons cover static counterparts and dynamic inference methods with varying depth, width, or input resolution.
  • Transfer learning: Gate transfer improves DS-ResNet performance on CIFAR-10 by 0.4%, demonstrating transferability of the dynamic gate.Table 3 compares transfer-learning performance with and without gate transfer.
  • Object detection: On VOC object detection, DS-MBNet-M and DS-MBNet-L improve mAP by 0.9 and 1.8 while reducing computation by 1.59× and 1.34×, respectively.The models use FSSD with DS-MBNet as the backbone.
  • Ablation study: Sandwich Gate Sparsification produces a more efficient complexity-accuracy trade-off by encouraging easy and hard samples to use slim and wide sub-networks, respectively.The ablation compares target, complexity, and SGS losses on ImageNet.

5. Conclusion

The paper concludes that DS-Net dynamically adjusts filter numbers for input-dependent inference while retaining hardware-efficient execution. Across ImageNet evaluations, it reduces computation and real-world latency with minimal accuracy loss and also transfers across tasks.

  • Conclusion: DS-Net dynamically adjusts filter numbers at test time while keeping the network hardware-efficient.Its two-stage training uses In-place Ensemble Bootstrapping and Sandwich Gate Sparsification.
  • Conclusion: DS-Net also generalizes to CIFAR-10 classification and VOC object detection tasks.The conclusion reports transfer beyond ImageNet.

A. Implementation Details

Stage II trains the slimming gate with a joint objective combining classification, complexity, and SGS losses. Complexity is measured by on-the-fly multiply-adds and normalized against the supernet to favor faster target architectures.

  • Losses in Stage II: On-the-fly multiply-adds serve as the model-complexity metric for the complexity penalty.This provides the constraint used to optimize efficiency during Stage II.
  • Losses in Stage II: The normalization factor T is set to the supernet's total MAdds, stabilizing the complexity constraint.The loss pushes routing toward faster architectures with target MAdds and separates easy and hard instances by architecture.
  • Routing-space configuration: Different target MAdds are reached by adjusting the routing space during gate training.For DS-MBNet-S, ρ is set to [0.35 : 0.05 : 0.5] to avoid routing to heavier sub-networks.

B. Experiments on EfficientNet

The EfficientNet experiment applies dynamic slimming to DS-EffNet-B0 with one gate controlling the final eight blocks. DS-EffNet improves over the reproduced EfficientNet-B0 baseline on ImageNet.

  • Model configuration: DS-EffNet uses one slimming gate after the eighth inverted residual block to control the remaining eight blocks.The first eight blocks use a fixed slimming ratio of 0.5, while the last eight use dynamic ρ values from 0.75 to 1.75.
  • Training configuration: The DS-EffNet supernet contains 20 paths in total and uses a training configuration analogous to DS-ResNet and DS-MBNet.The reported setup uses a 512 total batch size, 0.2 learning rate, cosine decay, and 150 epochs without extra DropPath or RMSProp enhancement.
  • Results: 0.7% and 0.8% improvements over EfficientNet-B0 are reported for DS-EffNet on ImageNet.The result is presented as evidence of efficacy with inverted bottleneck blocks and Squeeze-and-Excitation modules.

C. Additional Ablations

Additional ablations show that the slimming gate improves performance over similarly sized supernet sub-networks, while a larger distillation temperature substantially harms the slimmest sub-network.

  • Slimming gate: 0.8% and 1.2% gains are reported for DS-MBNet-S and DS-ResNet-S after adding the slimming gate.The comparison is against similarly sized sub-networks in their respective supernets.
  • Distillation temperature: 10.2% lower performance is observed for the slimmest DS-MBNet sub-network when distillation temperature increases from τ = 1 to τ = 4.The temperature comparison is conducted over 40 epochs.
Loading 2103.13258v1…