Source-linked AI summary
Adaptive Neural Networks for Efficient Inference
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, Venkatesh Saligrama
TL;DR
Deep neural networks are accurate but increasingly expensive to evaluate, motivating adaptive computation for individual examples. The paper learns early-exit and network-selection policies over existing networks, achieving substantial ImageNet speedups with small or no accuracy loss. These policies can also support fog-computing deployments, although using the same data to train networks and adaptive systems introduces bias toward more complex networks.
Problem
DNN test-time cost rises rapidly with performance, limiting deployment despite easy examples often not requiring a massive network’s full complexity.
Method
The paper learns adaptive early-exit and network-selection policies over existing DNNs, optimizing evaluation time and error through layer-by-layer weighted binary classification.
Results
2.8x speed-up over pure Resnet50 is achieved at 1% top-5 accuracy loss, while 1.9x speed-up is achieved with no model-accuracy change on ImageNet.
Takeaways & Limitations
Adaptive computation can reduce DNN evaluation cost by routing examples through early exits or relatively efficient networks while reserving complex models for harder cases.
Takeaways & Limitations
Training adaptive systems on the networks’ training data biases routing toward more complex networks; validation or excluded data generally improves performance.
Abstract
from arXiv · showhide
We present an approach to adaptively utilize deep neural networks in order to reduce the evaluation time on new examples without loss of accuracy. Rather than attempting to redesign or approximate existing networks, we propose two schemes that adaptively utilize networks. We first pose an adaptive network evaluation scheme, where we learn a system to adaptively choose the components of a deep network to be evaluated for each example. By allowing examples correctly classified using early layers of the system to exit, we avoid the computational time associated with full evaluation of the network. We extend this to learn a network selection system that adaptively selects the network to be evaluated for each example. We show that computational time can be dramatically reduced by exploiting the fact that many examples can be correctly classified using relatively efficient networks and that complex, computationally costly networks are only necessary for a small fraction of examples. We pose a global objective for learning an adaptive early exit or network selection policy and solve it by reducing the policy learning problem to a layer-by-layer weighted binary classification problem. Empirically, these approaches yield dramatic reductions in computational cost, with up to a 2.8x speedup on state-of-the-art networks from the ImageNet image recognition challenge with minimal (<1%) loss of top5 accuracy.
1. Introduction
Deep neural networks deliver strong accuracy but incur rapidly increasing test-time costs. The paper adapts existing networks so easy examples use less computation while difficult examples receive more capable models.
- DNNs achieve state-of-the-art accuracy across applications, including visual recognition, speech recognition, and machine translation.
- Resnet152 gains 4.4% top-5 accuracy over GoogLeNet on ImageNet but is about 14X slower at test-time.
- Natural data mixes easy and difficult examples, so easy examples may not require a massive DNN’s full power and complexity.
- The paper proposes adaptive early exits that bypass layers and adaptive network selection across pretrained DNNs with different cost/accuracy trade-offs.
- 2.8x speed-up is achieved over pure Resnet50 at 1% top-5 accuracy loss, while 1.9x speed-up retains model accuracy.
- The approach generalizes to fog-computing topologies, allowing complex models to be offloaded from memory-constrained devices.
2. Related Work
Prior work reduces DNN cost by changing precision, architecture, or network contents, whereas this paper adaptively chooses computation without modifying network structure or training. Its framework supports layer- and network-level exits through multiple evaluation paths.
- Existing approaches reduce evaluation cost through lower precision, efficient architectures, compression, or sparsification of networks.
- The proposed approach is complementary because it does not modify network structure or training and can be combined with prior cost-reduction methods.
- Spatially adaptive networks selectively activate nodes, while cascade methods allow early exits based on confidence feedback.
- Figure 2 illustrates network selection among AlexNet, GoogLeNet, and ResNet, alongside an AlexNet-based early-exit topology.
- The framework minimizes a system risk combining computational costs and accuracy, while supporting directed acyclic graphs with multiple model-evaluation paths.
- The methods adaptively choose sensor-like measurements or features for each example, drawing on the sensor-selection problem.
3. Adaptive Early Exit Networks
The adaptive early-exit method learns when examples should leave a network after intermediate layers or continue deeper. It optimizes system-wide time and error through sequential weighted binary classification.
- The early-exit strategy lets easy examples bypass layers by diverting them to a simple classifier for immediate classification.
- For AlexNet, the system considers exits after each of the first four convolutional layers because convolutional layers take more than three times longer than fully connected layers.
- After each convolutional layer, a decision function uses that layer’s output to choose early exit with the intermediate label or continuation to the next layer.
- The global objective minimizes adaptive-network evaluation time while keeping error within a user-chosen bound above full-network error.
- Policy training is reduced to layer-by-layer importance-weighted binary classification, where each policy compares future expected accuracy against computational loss.
- Including constant decision functions adds no policy overhead and guarantees the adaptive technique can only decrease test-time cost.
- Policies are trained bottom-up, beginning with the deepest early-exit block and then fixing its outputs while training preceding exits.
- The same framework extends to adaptive network selection, learning functions that trade average evaluation time against induced error.
4. Network Selection
The network selection system adaptively routes each example through pre-trained networks with different costs and uses cost-sensitive policy learning to balance evaluation time against error.
- Network Selection: The system arranges pre-trained networks with different cost/accuracy trade-offs and adaptively selects which network evaluates each example.Decision functions can return an earlier network’s prediction or route the example to a more costly network.
- Network Selection: κ1 routes examples after N1 to N1, N2, or N3, while κ2 routes examples reaching N2 to either N2 or N3.
- Network Selection: Policy training is posed as an importance-weighted supervised learning problem with λ controlling the trade-off between evaluation cost and loss.
- Network Selection: At stage 4, the optimal decision selects N2 when the cost of evaluating N3 exceeds its λ-weighted expected loss advantage; otherwise it selects N3.
- Network Selection: The adaptive network-learning algorithm trains the deepest untrained policy first, updates route costs, and prunes models that receive no routed examples.
5. Experimental Section
Experiments on ImageNet show that adaptive network selection provides larger computational savings than early exits, while preserving accuracy closely. The policies use lightweight confidence-based decisions to route examples among pretrained networks and exploit differing example difficulty.
- Policy Form: Entropy features make a simple linear policy significantly outperform more complex policies that exclude entropy features.The adaptive system uses logistic-regression policies over network outputs and prediction-confidence features.
- Network Selection: The full tree policy learns network orderings that outperform simpler cascades across budget regions and match the soft oracle at high and low budgets.A direct AlexNet-to-ResNet50 policy performs worse in lower-budget regions, while the full tree adapts its choices.
- Network Selection: At low budgets, policy overhead limits gains: AlexNet’s approximately 0.23ms overhead makes its balance point approximately 0.65ms, near GoogLeNet’s 0.7ms runtime.The ratio between network evaluation times is a significant factor in system performance.
- Network Error Analysis: 50% of examples are correctly classified by all networks for top-1 error and 77% for top-5 error, indicating that many examples need no costly network.Conversely, 18% and 5% are incorrectly classified by all networks for top-1 and top-5 error, respectively.
6. Conclusion
The paper proposes adaptive early-exit and network-selection schemes that trade off model accuracy against evaluation time, achieving significant computational gains with negligible accuracy loss on ImageNet.
- The two adaptive schemes reduce deep-network evaluation time while preserving near-original accuracy on ImageNet.The policy-learning objective is reduced to layer-by-layer weighted binary classification, and the method approximates an oracle with access to per-instance errors.