Source-linked AI summary
BlockDrop: Dynamic Inference Paths in Residual Networks
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S. Davis, Kristen Grauman, Rogerio Feris
TL;DR
Deep neural networks deliver strong recognition but can be too computationally expensive for fast inference. BlockDrop learns an input-conditioned policy to drop residual blocks while preserving accuracy, achieving faster ImageNet inference and strong CIFAR results. The learned paths also reflect image difficulty and visual patterns.
Problem
Deep neural networks' increasing depth and complexity make inference costly for delay-sensitive and real-time computer-vision applications.
Method
BlockDrop trains a policy network with reinforcement learning to select residual blocks to drop or keep in a pretrained ResNet, using rewards for block usage and prediction accuracy.
Results
20% speedup on ImageNet is achieved with ResNet-101 while maintaining the same 76.4% top-1 accuracy, alongside strong CIFAR results and image-correlated policies.
Takeaways & Limitations
Instance-specific inference paths improve the efficiency-accuracy trade-off and encode semantic information about image content and difficulty.
Takeaways & Limitations
Policy-gradient training is highly sensitive to initialization, and the block-configuration search space grows exponentially with the number of blocks.
Abstract
from arXiv · showhide
Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications. We introduce BlockDrop, an approach that learns to dynamically choose which layers of a deep network to execute during inference so as to best reduce total computation without degrading prediction accuracy. Exploiting the robustness of Residual Networks (ResNets) to layer dropping, our framework selects on-the-fly which residual blocks to evaluate for a given novel image. In particular, given a pretrained ResNet, we train a policy network in an associative reinforcement learning setting for the dual reward of utilizing a minimal number of blocks while preserving recognition accuracy. We conduct extensive experiments on CIFAR and ImageNet. The results provide strong quantitative and qualitative evidence that these learned policies not only accelerate inference but also encode meaningful visual information. Built upon a ResNet-101 model, our method achieves a speedup of 20\% on average, going as high as 36\% for some images, while maintaining the same 76.4\% top-1 accuracy on ImageNet.
1. Introduction
BlockDrop addresses the inference cost of deep vision networks by dynamically selecting residual blocks per image. It combines learned block-dropping policies with accuracy preservation, improving efficiency while revealing image-specific computational and visual patterns.
- Fast inference is especially important for delay-sensitive and real-time applications, but higher accuracy has brought increasingly deep and complex architectures.
- Fixed model-compression schemes extract the same features for every image, unlike adaptive perception that allocates more scrutiny to difficult or occluded inputs.
- ResNet skip-connections support block dropping by providing direct paths between residual blocks and making the architecture behave like ensembles of shallower networks.
- BlockDrop learns image-specific strategies that preserve prediction accuracy while minimizing residual-block usage during inference.
- 20% speedup on ImageNet is achieved with ResNet-101 while maintaining the same 76.4% top-1 accuracy.
- BlockDrop uses 33% and 55% of ResNet-110 blocks while achieving 93.6% and 73.7% accuracy on CIFAR-10 and CIFAR-100, respectively.
2. Related Work
Related work includes static model-compression methods, conditional computation, early prediction, and prior studies of layer dropping in residual networks. BlockDrop differs by dynamically selecting residual blocks for each input rather than fixing layers at test time or restricting execution to early contiguous blocks.
- Layer Dropping in Residual Networks: Prior residual-layer dropping methods use dropping during training as regularization, while test-time layers remain fixed.
- Model Compression: Model-compression approaches reduce storage or computation through techniques including distillation, factorization, pruning, quantization, structured matrices, binarization, and hashing.
- Conditional Computation: Conditional-computation methods selectively activate network modules per example, often using input-dependent gating functions learned with reinforcement learning.
- BlockDrop Framework: BlockDrop's framework produces block keep/drop decisions, evaluates only active blocks, and trains the policy with rewards for both block usage and prediction accuracy.
- Early Prediction: Early-prediction models exit when a confidence criterion is satisfied, whereas related spatial ACT methods allow only early contiguous blocks within predefined ResNet segments.
3. Approach
BlockDrop treats image-specific block selection in ResNets as a reinforcement-learning search problem, seeking correct predictions with minimal computation. A policy network selects residual blocks, while curriculum learning and joint finetuning support scalable training and adaptation.
- The method seeks a minimum-block configuration for each test image without decreasing classification accuracy.
- ResNet skip-connections allow residual blocks to be dropped while preserving information through direct paths and the network’s ensemble-like structure.
- BlockDrop’s policy network outputs binary keep/drop decisions for all residual blocks, and only retained blocks execute during the forward pass.
- The policy is trained as a single-step contextual decision process using Bernoulli actions and policy gradients with Monte Carlo estimates and a self-critical baseline.
- The reward combines prediction correctness with block usage, penalizing incorrect predictions while tuning the efficiency–accuracy trade-off through γ.
- Curriculum learning gradually optimizes decisions for more blocks, after which the ResNet and policy network are jointly finetuned to address train–test mismatch.
4. Experiment
BlockDrop is evaluated on CIFAR-10, CIFAR-100, and ImageNet using dynamic policies, heuristic baselines, FLOPs, accuracy, efficiency overhead, and qualitative policy analyses. Across these experiments, learned instance-specific block selection reduces computation while preserving or improving recognition and reflects image difficulty and visual patterns.
- Experimental Setup: BlockDrop is evaluated on CIFAR-10, CIFAR-100, and ImageNet, reporting classification accuracy and ImageNet top-1 accuracy.The experiments use CIFAR datasets with 60,000 images each and ImageNet with 1.2M training images and 50,000 validation images.
- Experimental Setup: Policy-network experiments use shallow ResNets for control, with 3 blocks for CIFAR and 4 blocks for ImageNet.The ImageNet policy network receives images downsampled to 112×112.
- Quantitative Results: Image-specific block usage varies substantially: nearly 15% of CIFAR-10 images use fewer than 10 ResNet-110 blocks, with some using only 3.Static policies cannot capture this variance, whereas learned policies outperform heuristic dropping strategies.
- Quantitative Results: Joint finetuning improves accuracy while using fewer or nearly the same number of blocks, including gains of 2.7% and 18.2% on CIFAR-10.With ResNet-110, the method also outperforms the pretrained model by 0.4% and 1.5% using 31% and 55.9% of original blocks on CIFAR-10 and CIFAR-100.
- Quantitative Results: 0.4% higher accuracy with 65% fewer FLOPs is obtained than ResNet-110 on CIFAR, while matching ACT and SACT precision above 93.0% with 50% of their FLOPs.BlockDrop’s per-image FLOPs variance is three times larger than SACT’s, enabling speedups as high as 85% for some correctly classified samples.
- Quantitative Results: 20% speedup is achieved on ImageNet with ResNet-101 at the same 76.4% top-1 accuracy as the full model.A separate setting reaches 76.8% versus 76.4% with 6% speedup; the 20% result uses 1.25×10^10 versus 1.56×10^10 FLOPs.
- Qualitative Results: Learned policies group semantically similar images into distinct paths, while fewer blocks correspond to easier centered objects and more blocks to occlusion or clutter.Orange policies separate piles, close-ups, and slices; ImageNet samples using the most blocks contain several objects, occlusion, or cluttered backgrounds.
5. Conclusion
BlockDrop selectively evaluates residual blocks based on each input, targeting faster inference while retaining prediction accuracy. Experiments on CIFAR and ImageNet show improved efficiency–accuracy trade-offs and policies that encode semantic image information.
- 5. Conclusion: BlockDrop selects residual blocks to evaluate in a learned, input-conditioned manner for faster inference.The policy predicts blocks to drop in a pretrained ResNet, while the ResNet is jointly finetuned for block-dropping behavior.
- 5. Conclusion: Experiments on CIFAR and ImageNet show considerable efficiency–accuracy gains over existing methods.
- 5. Conclusion: The learned policies encode semantic information in images.
Supplemental Materials
Supplemental experiments describe a sequential BlockDrop variant and report implementation settings and benchmark comparisons. Results show substantial speedups on CIFAR-10 and about 20% average speedup on ImageNet.
- Details of BlockDrop-seq (Ours-seq): The sequential BlockDrop variant conditions each block's drop-or-keep decision on the previous block's activations.Unlike one-shot BlockDrop, the sequential model predicts one action at a time.
- Implementation Details: CIFAR training uses 5000 curriculum-learning epochs and 1600 joint-finetuning epochs.The stated batch sizes are 2048 for curriculum learning and 256 for joint finetuning.
- Implementation Details: ImageNet training uses 45 curriculum-learning epochs followed by 10 joint-finetuning epochs.The policy-training batch size is 2048, while joint finetuning uses 320.
- Detailed Results on CIFAR-10 and ImageNet: 37% to 76% speedups are observed on CIFAR-10 with little to no performance degradation.
- Detailed Results on CIFAR-10 and ImageNet: Around 20% average speedup is achieved on ImageNet with BlockDrop built upon ResNet-101 at γ = 5.