Resolution Adaptive Networks for Efficient Inference

Le Yang; Yizeng Han; Xi Chen; Shiji Song; Jifeng Dai; Gao Huang

Source-linked AI summary

Resolution Adaptive Networks for Efficient Inference

Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, Gao Huang

arXiv:2003.07326v5Mar 16, 2020cs.CV

TL;DR

Deep CNN inference is expensive, and existing adaptive methods mainly exploit architectural redundancy rather than spatial redundancy in inputs. RANet routes samples through progressively higher-resolution subnetworks, letting confident easy cases exit early while preserving finer features for hard cases. Experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate effectiveness in anytime and budgeted batch classification settings.

Problem
Deep CNNs have high inference costs, while existing adaptive inference methods mainly exploit redundancy in network depth or width rather than spatial redundancy in input samples.
Method
RANet uses lightweight subnetworks at progressively higher input resolutions, exiting samples with high confidence early and processing uncertain samples with finer features.
Results
Experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate RANet’s effectiveness in anytime prediction and budgeted batch classification.
Takeaways & Limitations
RANet reduces unnecessary high-resolution convolutions while retaining high-resolution processing for non-typical samples that coarse representations cannot reliably classify.

Abstract

from arXiv · show

Adaptive inference is an effective mechanism to achieve a dynamic tradeoff between accuracy and computational cost in deep networks. Existing works mainly exploit architecture redundancy in network depth or width. In this paper, we focus on spatial redundancy of input samples and propose a novel Resolution Adaptive Network (RANet), which is inspired by the intuition that low-resolution representations are sufficient for classifying "easy" inputs containing large objects with prototypical features, while only some "hard" samples need spatially detailed information. In RANet, the input images are first routed to a lightweight sub-network that efficiently extracts low-resolution representations, and those samples with high prediction confidence will exit early from the network without being further processed. Meanwhile, high-resolution paths in the network maintain the capability to recognize the "hard" samples. Therefore, RANet can effectively reduce the spatial redundancy involved in inferring high-resolution inputs. Empirically, we demonstrate the effectiveness of the proposed RANet on the CIFAR-10, CIFAR-100 and ImageNet datasets in both the anytime prediction setting and the budgeted batch classification setting.

1. Introduction

Adaptive inference targets the high computational cost of deep CNNs by allocating computation according to sample difficulty. RANet instead exploits spatial redundancy, using low-resolution predictions for easy samples and finer-resolution subnetworks for hard samples.

Deep CNN inference remains costly despite hardware advances, motivating lightweight architectures, pruning, quantization, and adaptive inference methods.
Existing adaptive inference methods mainly reduce network depth or width because samples differ substantially in classification difficulty.Some samples can be classified by smaller models, whereas others require larger networks.
RANet exploits spatial redundancy by assuming low-resolution features suffice for easy samples while hard samples require high-resolution details.The motivation is also framed through coarse-to-fine processing and low- versus high-frequency information.
RANet uses subnetworks with different input resolutions, escalating to higher resolution when a previous subnetwork fails a prediction-confidence criterion.Coarse features are reused and fused into the current subnetwork as resolution increases.
The paper evaluates RANet on CIFAR-10, CIFAR-100, and ImageNet under anytime and budgeted batch classification settings.The introduction reports experiments demonstrating the method’s effectiveness in adaptive inference tasks.

2. Related work

Prior acceleration methods often process every sample with a fixed model, while adaptive networks allocate computation according to input complexity. RANet addresses the underexplored spatial redundancy of images by processing most inputs coarsely and reserving larger scales for difficult cases.

Static acceleration methods use lightweight models, pruning, quantization, or distillation while consistently processing all input samples with a whole network.
Adaptive networks instead allocate computational resources according to input complexity through mechanisms such as cascades, model mixtures, and layer skipping.
Most prior adaptive networks exploit architectural redundancy, whereas RANet exploits both structural redundancy and spatial redundancy in input samples.
Multi-scale feature maps combine coarse and fine features, but retaining high-resolution maps throughout a network can create resource-hungry models.High-resolution features remain necessary for some atypical hard samples and tasks such as pose estimation.
RANet processes some inputs at small scales and applies larger scales only when coarse representations cannot recognize them, improving computational efficiency without sacrificing accuracy.Unlike ADASCALE, it can be implemented for budgeted classification during adaptive inference.

3. Method

RANet performs adaptive inference from coarse to fine: samples are processed first with low-resolution features and receive deeper or higher-resolution computation only when confidence is insufficient. Its architecture combines multi-scale sub-networks, dense feature reuse, fusion blocks, and depth adaptation.

Adaptive Inference Setting: RANet uses K intermediate classifiers sharing parameters to allocate computation dynamically according to each sample's complexity.A sample exits at the first classifier satisfying the confidence criterion.
Adaptive Inference Setting: The highest softmax confidence determines early exiting, while threshold ϵ controls the accuracy–computation trade-off.The final prediction is taken from the first classifier whose largest softmax output exceeds the threshold.
Overall Architecture: RANet first processes inputs with the lowest-resolution sub-network, then propagates uncertain samples to higher-resolution sub-networks using fused intermediate features.The procedure repeats across sub-networks until confidence meets the criterion or the final classifier is reached.
Overall Architecture: Each sub-network downsamples features toward the lowest resolution, and classifiers attach only to the last few blocks operating at that resolution.Thus, prediction heads remain at the lowest-resolution feature maps even as higher-resolution paths support difficult samples.
Initial Layer: The Initial Layer generates H base features across S scales, using regular and strided convolutions to derive coarse features from higher-resolution features.The resulting base features form the inputs to H sub-networks.
Sub-networks with Different Scales: Sub-network 1 applies Dense Blocks to the lowest-resolution features, while later sub-networks use Fusion Blocks to combine their base features with features from the preceding sub-network.Fusion can preserve resolution or reduce feature scale through a strided convolution, with up-convolution aligning resolutions before concatenation.
Resolution and Depth Adaptation: RANet combines depth adaptation within each scale with resolution adaptation across sub-networks, producing a zigzag-shaped computation graph from coarse to fine.Unlike MSDNet's continuation across all scales after an uncertain prediction, RANet repeats depth adaptation in the next-resolution sub-network.

4. Experiments

Experiments across CIFAR-10, CIFAR-100, and ImageNet show that RANet improves accuracy-efficiency tradeoffs in anytime and budgeted batch classification, while adapting resolution to sample difficulty.

4.1. Anytime Prediction: RANet outperforms MSDNet, especially when computational budgets are low, in anytime prediction across the evaluated datasets.Figure 5 evaluates accuracy as a function of computational budget on CIFAR-10, CIFAR-100, and ImageNet.
4.1. Anytime Prediction: On CIFAR-10 and CIFAR-100, RANet exceeds MSDNet by over 1% and 2%−5%, respectively, from 0.1 × 10^8 to 0.5 × 10^8 FLOPs.RANet also reaches its highest accuracy with around 0.25 × 10^8 FLOPs.
4.1. Anytime Prediction: On ImageNet, RANet surpasses MSDNet by around 1%−7% from 0.5 × 10^9 to 1.5 × 10^9 FLOPs and uses around 27% fewer FLOPs at 74% accuracy.Both models achieve similar classification accuracy at the last classifier.
4.2. Budgeted Batch Classification: In budgeted batch classification, RANet consistently outperforms MSDNet and other baselines across all budgets on the two CIFAR datasets.With budgets above 0.2 × 10^8 FLOPs on CIFAR-10, RANet reaches 94.2% accuracy, 0.5% above MSDNet.
4.2. Budgeted Batch Classification: On ImageNet, RANet leads MSDNet by about 0.5%, 1%, and 1.2% at 0.75 × 10^9, 1 × 10^9, and 1.75 × 10^9 FLOPs, respectively.At equal accuracy, RANet reduces computational budgets by around 65% versus GoogLeNet, 56% versus ResNets, and 44% versus DenseNets.
4.3. Visualization and Discussion: Visualization shows early classifiers recognize prototypical easy samples, while later high-resolution classifiers recognize hard samples involving multiple objects, tiny objects, or incomplete characteristics.High-resolution feature maps are necessary for accurate classification in these cases.

5. Conclusion

RANet combines resolution adaptation with depth adaptation, routing confident samples through coarse-feature subnetworks and reserving finer features for unreliable predictions. Experiments on three image-classification benchmarks demonstrate effectiveness in both evaluated adaptive-inference settings.

5. Conclusion: RANet uses a multi-scale dense connection architecture in which lightweight coarse-feature subnetworks classify high-confidence samples early.Higher-resolution features are used for samples with unreliable predictions from previous subnetworks.
5. Conclusion: Resolution adaptation together with depth adaptation improves RANet’s computational efficiency.The conclusion attributes this efficiency to the network’s resolution adaptation mechanism and depth adaptation within each subnetwork.
5. Conclusion: Experiments on three image-classification benchmarks demonstrate RANet’s effectiveness in anytime prediction and budgeted batch classification.The evaluated benchmarks are CIFAR-10, CIFAR-100, and ImageNet.

1. Appendix A: Implementation Details

Appendix A specifies the multi-scale architectures, classifier placements, and evaluation settings used for RANet and MSDNet across CIFAR and ImageNet.

Baseline configurations: MSDNet uses three scales and six, eight, or ten classifiers on CIFAR, while ImageNet configurations use four scales and five classifiers.On ImageNet, classifier placement varies with the network-block step t ∈ {4, 6, 7}.
Architecture configurations: CIFAR RANets are tested with three-base-feature and four-base-feature configurations, including models with six or eight total classifiers.Model-C-2 and Model-C-3 each use eight classifiers, while Model-C-1 has three resolution-specific sub-networks.
Evaluation settings: Model-C-3 is evaluated for anytime classification, while all three CIFAR RANet models are evaluated for budgeted batch classification; ImageNet uses Model-I-2 for anytime evaluation and both models for batch evaluation.The corresponding evaluations are reported in the main paper’s Figures 5 and 6.
Architecture configurations: ImageNet RANets use four resolution-specific sub-networks with eight, six, four, and two Conv Blocks, and Model-I-1 has eight classifiers.Model-I-2 keeps the same architecture as Model-I-1 but increases the base feature channel counts.

2. Appendix B: Improved Techniques

Appendix B evaluates ISC and OFA on RANet and MSDNet for CIFAR-100 adaptive inference. Improved RANet generally outperforms improved MSDNet, while the techniques can reduce first-classifier performance.

Observed limitation: ISC and OFA do not work well on RANet’s first classifier.The first-classifier degradation is also identified as a reason the original RANet can outperform improved RANet in batch classification.
Budgeted batch classification: Improved RANet remains superior to improved MSDNet in budgeted batch classification, especially when the budget exceeds 0.3 × 10^8 FLOPs.The batch comparison evaluates RANet, Model-C-3, and MSDNet with eight classifiers.
Budgeted batch classification: Improved RANet accuracy is 1% higher than MSDNet and 0.5% higher than improved MSDNet in budgeted batch classification.These comparisons support the effectiveness of RANet with the improved training techniques.