Source-linked AI summary
SkipNet: Learning Dynamic Routing in Convolutional Networks
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, Joseph E. Gonzalez
TL;DR
Deep convolutional networks improve accuracy but impose substantial computation and latency, even though many inputs may not require their full depth. SkipNet learns input-dependent layer skipping with a hybrid supervised and reinforcement-learning procedure for discrete gates. Across four benchmark datasets, it reduces computation while preserving accuracy and outperforms prior dynamic and static approaches.
Problem
Very deep convolutional networks increase prediction cost and latency, while the appropriate depth may vary across input images.
Method
SkipNet uses gating units in modified residual networks and trains their discrete skipping policies with supervised pre-training followed by REINFORCE refinement.
Results
SkipNet reduces computation by 30% on ImageNet, 50% on CIFAR-10, 37% on CIFAR-100, and 86% on SVHN while preserving accuracy, and outperforms prior methods on CIFAR-10 and ImageNet.
Takeaways & Limitations
Input-dependent layer skipping can substantially reduce inference computation while retaining the original model’s accuracy across four benchmark datasets.
Takeaways & Limitations
The gating formulation requires each residual transformation and its input to have the same dimensions.
Abstract
from arXiv · showhide
While deeper convolutional networks are needed to achieve maximum accuracy in visual perception tasks, for many inputs shallower networks are sufficient. We exploit this observation by learning to skip convolutional layers on a per-input basis. We introduce SkipNet, a modified residual network, that uses a gating network to selectively skip convolutional blocks based on the activations of the previous layer. We formulate the dynamic skipping problem in the context of sequential decision making and propose a hybrid learning algorithm that combines supervised learning and reinforcement learning to address the challenges of non-differentiable skipping decisions. We show SkipNet reduces computation by 30-90% while preserving the accuracy of the original model on four benchmark datasets and outperforms the state-of-the-art dynamic networks and static compression methods. We also qualitatively evaluate the gating policy to reveal a relationship between image scale and saliency and the number of layers skipped.
1 Introduction
SkipNet addresses the cost of very deep convolutional networks by learning input-dependent skipping policies. Its hybrid training method handles discrete gating decisions while preserving accuracy and reducing inference computation.
- Motivation: Deeper networks improve accuracy but increase prediction cost, while many images may not require the additional depth.The optimal network depth can depend on the input image.
- Approach: SkipNet uses gating units to dynamically select which convolutional layers to skip for each input.The gating policy uses previous-layer activations to decide whether to bypass subsequent layers.
- Learning challenge: Discrete skipping decisions are non-differentiable, preventing direct optimization with standard gradient-based methods.This makes learning an efficient policy while retaining accuracy challenging.
- Learned policy: More layers are executed for challenging images than for easy images under the learned per-input policy.This behavior is illustrated in Figure 1.
- Learning method: SkipNet combines supervised learning with reinforcement learning to train hard skipping policies.Soft relaxation and cross-entropy pre-training initialize the gates, after which REINFORCE refines the policy without relaxation.
- Results: 50% on CIFAR-10, 37% on CIFAR-100, 86% on SVHN, and 30% on ImageNet are the reported computation reductions while preserving accuracy.SkipNet also consistently outperforms prior methods on CIFAR-10 and ImageNet benchmarks.
2 Related Work
Prior work accelerates convolutional networks through static compression, early termination, cascades, and gating mechanisms. SkipNet differs by conditionally bypassing individual layers while retaining feature reuse and allowing combinatorial layer selection.
- Static compression: Static compression methods reduce model cost after training but do not dynamically adjust complexity to the input.These methods include weight sparsification, filter pruning, vector quantization, and distillation.
- Early termination: Early-termination methods save computation by halting processing, whereas SkipNet conditionally bypasses individual layers instead of exiting early.The comparison is made against approaches that terminate computation within convolutional networks.
- Cascaded models: Cascaded models use smaller classifiers for easier images but can fail to reuse features across classifiers and require substantial storage overhead.SkipNet instead selects layers within one network in a combinatorial search space.
- Gating mechanisms: SkipNet’s gating modules regulate groups of layers and are related to gating designs used in recurrent and convolutional networks.The cited prior gates are described as soft gates.
3 SkipNet Model Design
SkipNet selectively executes or bypasses layers using binary gating decisions based on previous activations. Its design combines lightweight gating networks with a two-stage hybrid learning procedure for optimizing discrete skipping policies.
- Dynamic Layer Skipping: SkipNet gating networks map previous-layer activations to binary decisions that execute or bypass subsequent layers or groups of layers.The gated output requires the transformed input and bypassed input to have matching dimensions, as in residual architectures.
- Gating Network Design: Gating designs trade expressiveness against computation cost, spanning feed-forward convolutional gates and recurrent gates with parameter sharing.The designs include unique gates for residual blocks and a unified recurrent gate shared across blocks.
- Gating Network Design: 19% and 12.5% are the approximate computation costs of FFGate-I and FFGate-II relative to residual blocks.FFGate-I uses two 3x3 convolutions, whereas FFGate-II uses one strided convolution.
- Gating Network Design: 0.04% is the approximate computation cost of RNNGate relative to residual blocks.RNNGate uses a one-layer LSTM with input and hidden unit size 10, sharing parameters and reusing computation across stages.
- Gating Network Design: Recurrent gates dominate feed-forward gates in prediction accuracy and computation cost in the later experiments.The authors conjecture that recurrent gates better capture cross-layer dependencies.
- Hybrid Learning: SkipNet frames its sequence of discrete gate decisions as policy optimization, rewarding skipped layer cost while trading prediction loss against computation reduction.The gate action executes a layer when gi = 1 and skips it when gi = 0; α controls the loss–reward trade-off.
- Hybrid Learning: The hybrid learning algorithm first uses supervised pre-training with relaxed backpropagation, then applies REINFORCE to refine network and gate parameters.This two-stage procedure addresses non-differentiable decisions and can improve prediction accuracy while further reducing prediction cost.
4 Experiments
Experiments evaluate SkipNets built from ResNets across four image-classification benchmarks, testing computation savings, accuracy preservation, comparisons, and learned skipping behavior. The hybrid procedure reduces inference cost while maintaining accuracy and performs competitively against dynamic and static alternatives.
- Experimental setup: SkipNets are evaluated on CIFAR-10, CIFAR-100, SVHN, and ImageNet using ResNet-based architectures with hard gates between residual blocks.The experiments include gate designs, supervised pre-training, and hybrid reinforcement-learning refinement.
- Computation and accuracy: 50% and 37% computation reductions are achieved on CIFAR-10 and CIFAR-100, while SVHN reaches 86% and ImageNet reaches 30%, preserving full-network accuracy.The reported reductions use recurrent gates and the hybrid reinforcement-learning procedure with supervised pre-training.
- Computation and accuracy: Adjusting α controls the trade-off between computation and classification accuracy, allowing different operating points for resource or accuracy requirements.On ImageNet, the experiments plot accuracy against the average number of skipped layers for α values from 0.0 to 4.0.
- Comparisons: SkipNet outperforms the stochastic-depth variant SDV across different network depths and skipping ratios on CIFAR-10 and CIFAR-100.SDV randomly skips blocks during both training and inference, whereas SkipNet learns the skipping policy.
- Learned policies: The learned policy skips more layers for easier classes and smaller input scales, while executing fewer blocks at smaller scales on ImageNet and SVHN.Hard classes such as dogs receive less aggressive skipping than easier classes such as trucks.
- Learned policies: SkipNet skips less frequently in the middle ResNet group than in the first and third groups, suggesting that the middle group is more critical to features.This block-level pattern is visualized on CIFAR-10, with related dataset visualizations reported in supplementary material.
- Training analysis: Pure reinforcement learning fails on SkipNet-38, with roughly 10% accuracy, while hybrid training with supervised pre-training achieves higher accuracy than supervised-only refinement.The result supports supervised pre-training as an effective initialization for learning the skipping policy.
5 Conclusion
SkipNet dynamically skips redundant layers per input while preserving prediction accuracy. The paper frames dynamic execution as sequential decision-making and uses hybrid supervised–reinforcement learning; evaluations show substantial computation savings and favorable comparisons with prior methods.
- SkipNet dynamically skips redundant layers on a per-input basis without sacrificing prediction accuracy.
- The dynamic execution problem is formulated as sequential decision-making, with a hybrid algorithm combining supervised and reinforcement learning.
- Across four benchmark datasets, SkipNets substantially reduce computation while preserving the original accuracy.
- Compared with state-of-the-art dynamic models and static compression techniques, SkipNets obtain better accuracy with lower computation.
- Dynamic architectures may improve computational efficiency by specializing and reusing individual components.
Appendix
The supplementary material adds benchmark results and visualizations to examine computation–accuracy trade-offs and block-skipping patterns.
- Additional CIFAR-10, CIFAR-100, and SVHN results demonstrate how SkipNet addresses varying computation and accuracy requirements.
- Extra visualizations show the skipping ratios of different SkipNet blocks and reveal their skipping patterns.
6 Trade-off Computational Cost and Accuracy
SkipNet’s hybrid objective balances prediction error against skipping reward, while gate designs trade computational overhead against routing capability. Small α values can preserve accuracy with large computation reductions, whereas larger values rapidly reduce both computation and accuracy.
- The hybrid objective combines prediction error and skipping reward, with α tuning their relative balance.
- Gate designs: For networks deeper than 100 layers, experiments use FFGate-II, while shallower networks use FFGate-I.
- Gate designs: FFGate-I uses two 3x3 convolutional layers at roughly 19% of residual-block computation, while FFGate-II uses one stride-2 convolution at about 12.5%.
- Gate designs: RNNGate uses a one-layer LSTM with input and hidden size 10 and only 0.04% of residual-block computation.
- Trade-off: For small α≤1.0, a large computation decrease accompanies a negligible accuracy decrease.
- Trade-off: For larger α>1.0, both computation and accuracy decrease rapidly, making α useful for meeting different requirements.
- Trade-off: RNNGate is preferable to FFGate because it can skip a similar number of layers with lower computational overhead.
7 Skip Ratio of Different Blocks
SkipNet’s block-level routing is nonuniform: it skips more blocks in some groups and fewer in groups judged more critical for feature extraction. These patterns vary across datasets and network configurations while preserving the original models’ accuracy.
- ImageNet: On ImageNet, SkipNet skips fewer blocks in Groups 1, 2, and 4 and more in Group 3, indicating greater criticality in Groups 1, 2, and 4.
- The models augmented with gates can match the accuracy of the original models.
- CIFAR datasets: On CIFAR-100, SkipNet skips intensively across groups but skips less in Group 2, indicating that Group 2 is relatively more critical.
- CIFAR datasets: SkipNet-101 on CIFAR-100 reduces more computation than on ImageNet while preserving full-network accuracy.
- The visualized skipping patterns show that SkipNet identifies critical blocks for each input and skips unnecessary blocks to reduce computation.