Source-linked AI summary
Improved Techniques for Training Adaptive Deep Networks
Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang, Gao Huang
TL;DR
Adaptive inference can reduce test-time computation, but prior work has focused more on architectures and inference policies than on training multi-exit networks. This paper introduces Gradient Equilibrium, Inline Subnetwork Collaboration, and One-for-all Knowledge Distillation, and reports consistently improved efficiency across CIFAR-10, CIFAR-100, and ImageNet.
Problem
Training adaptive networks with multiple intermediate classifiers is underexplored, despite conflicts among classifiers and the need to improve their cooperation.
Method
The paper combines Gradient Equilibrium for gradient stabilization with Inline Subnetwork Collaboration and One-for-all Knowledge Distillation for cooperation among exits.
Results
The proposed techniques consistently improve the efficiency of state-of-the-art adaptive deep networks on CIFAR-10, CIFAR-100, and ImageNet.
Takeaways & Limitations
Training strategies can further improve adaptive networks beyond their architecture and inference design, including by stabilizing optimization and transferring knowledge among exits.
Takeaways & Limitations
The experiments exclude efficient architectures such as MobileNet, ShuffleNet, and NASNet, leaving their combination with adaptive training for future work.
Abstract
from arXiv · showhide
Adaptive inference is a promising technique to improve the computational efficiency of deep models at test time. In contrast to static models which use the same computation graph for all instances, adaptive networks can dynamically adjust their structure conditioned on each input. While existing research on adaptive inference mainly focuses on designing more advanced architectures, this paper investigates how to train such networks more effectively. Specifically, we consider a typical adaptive deep network with multiple intermediate classifiers. We present three techniques to improve its training efficacy from two aspects: 1) a Gradient Equilibrium algorithm to resolve the conflict of learning of different classifiers; 2) an Inline Subnetwork Collaboration approach and a One-for-all Knowledge Distillation algorithm to enhance the collaboration among classifiers. On multiple datasets (CIFAR-10, CIFAR-100 and ImageNet), we show that the proposed approach consistently leads to further improved efficiency on top of state-of-the-art adaptive deep networks.
1. Introduction
Adaptive inference reduces average computation by conditioning execution on each input, but training multi-exit networks introduces classifier conflicts that conventional strategies may not handle well. The paper proposes Gradient Equilibrium, Inline Subnetwork Collaboration, and One-for-all Knowledge Distillation to stabilize training and improve classifier cooperation.
- Motivation: Adaptive inference dynamically trades accuracy for speed by allocating less computation to easy samples and adjusting to changing computational budgets.It can also combine with efficient architectures, pruning, and quantization.
- Training challenge: Multi-exit adaptive networks are harder to train because classifiers can negatively affect one another through conflicting or accumulated gradients.The paper focuses on networks with intermediate classifiers that enable early exits for easy samples.
- Proposed techniques: Gradient Equilibrium rescales gradients during backward propagation to maintain a constant scale, reducing gradient variance and stabilizing training.The technique targets conflicts among classifiers while jointly optimizing them.
- Proposed techniques: Inline Subnetwork Collaboration transfers earlier classifiers’ prediction logits forward as priors for learning current and subsequent classifiers.This encourages information sharing among exits during training.
- Proposed techniques: One-for-all Knowledge Distillation uses the deepest, typically most accurate exit as a teacher for supervising earlier classifiers.Together with ISC, it strengthens collaboration among the network’s exits.
- Results: Experiments on CIFAR-10, CIFAR-100, and ImageNet consistently improve the efficiency of state-of-the-art adaptive deep networks.The result is reported across three image-classification datasets.
2. Related Work
Prior efficient-network research emphasizes architectures, pruning, quantization, and adaptive inference, while adaptive-inference studies primarily design topologies or inference policies. This paper instead develops model-agnostic training strategies for adaptive networks with multiple intermediate classifiers.
- Computationally Efficient Deep Networks: Efficient deep-network research has pursued specialized architectures, weight pruning, and weight quantization to reduce computational cost.Examples include depthwise separable, grouped, and learned group convolutions.
- Adaptive Inference: Adaptive inference saves computation on easy samples while retaining compatibility, flexibility, and tunable resource allocation.It differs from static efficiency methods by adapting resource use during inference.
- Adaptive Inference: Most prior adaptive-inference work focuses on network topology selection policies, including early exits, adaptive decisions, and gating functions.Representative approaches use ensembles, multi-scale networks, or ResNet-based gates.
- Position of this work: This paper makes an orthogonal, model-agnostic contribution by improving training for adaptive architectures with multiple intermediate classifiers.The strategy is intended to apply across several adaptive-inference architectures.
- Knowledge Distillation: Its One-for-all Knowledge Distillation strategy relates to prior distillation work but uses classifier outputs within the adaptive network to supervise other exits.The supplied passage establishes the connection to online and teacher-student distillation approaches.
3. Method
The method trains multi-exit adaptive networks with Gradient Equilibrium to stabilize shared optimization, plus ISC and OFA to promote collaboration among classifiers.
- Method overview: The proposed training strategy combines GE for gradient conflicts with ISC and OFA for collaboration among the multiple classifiers.These techniques target optimization and information-sharing problems in adaptive networks rather than changing the inference architecture.
- Adaptive inference model: The adaptive model is a CNN with k−1 intermediate classifiers and one final classifier, producing k predictions from shared-parameter transformations.At test time, a confidence-based decision function selects an exit dynamically for each input.
- 3.1. Gradient Equilibrium: Jointly optimizing exit losses creates gradient imbalance because shared blocks receive contributions from their own and subsequent classifiers.As the number of subnetworks increases, gradient variance can grow and destabilize training.
- 3.1. Gradient Equilibrium: Gradient Equilibrium re-normalizes backward gradients at shared nodes while preserving forward information flow, addressing instability from overlapping subnetworks.Its scaling operations bound gradient magnitudes and avoid the overly small gradients produced by simply averaging cumulative losses.
- 3.2. Forward Knowledge Transfer: ISC transfers each classifier’s logits to its subsequent classifier through adjacent knowledge-transfer paths, enabling forward collaboration without backpropagating along those paths.The transfer path can be a small learned network or a non-learning function; even identity transforms improve adaptive inference.
- 3.3. Backward Knowledge Transfer: OFA uses the deepest classifier as a teacher so that all intermediate exits receive distillation supervision alongside their classification loss.The deepest classifier is selected because it usually achieves the highest accuracy through its greater capacity.
4. Experiments
Experiments evaluate the proposed training strategy for adaptive MSDNets across CIFAR-10, CIFAR-100, and ImageNet, including component ablations and comparisons under computational budgets. The strategy consistently improves accuracy or efficiency over baseline adaptive networks, with gains also observed for individual training components.
- Results on CIFAR: At 1×10^8 FLOPS, the proposed training strategy improves MSDNet’s CIFAR top-1 accuracy by more than 0.5% over the baseline.The improvement grows with larger computational budgets.
- Results on CIFAR: For equal CIFAR-100 accuracy, the proposed approach requires half the computation of DenseNet and one-third the computation of ResNet.
- Results on ImageNet: On ImageNet, the method improves baseline MSDNet by around 0.5% top-1 accuracy at a 1×10^8 FLOPS budget and exceeds ResNet and DenseNet by more than 6%.The advantage over baseline MSDNet increases as more computation is allowed.
- Ablation Study: Gradient Equilibrium consistently raises validation accuracy and stabilizes training across MSDNet architectures with different depths and numbers of exits.The gains are also reported in the CIFAR-100 and ImageNet accuracy tables.
- Ablation Study: Inline Subnetwork Collaboration improves performance consistently, with exit 5 gaining more than 1.4% top-1 accuracy on CIFAR-100.The method also increases consistency between sample-confidence rankings at different exits.
- Ablation Study: One-for-all Knowledge Distillation improves adaptive-network accuracy, especially for lower layers supervised by the deepest classifier.These results support using the network’s own deepest prediction as supervision for intermediate exits.
5. Conclusion
The paper presents three training techniques for adaptive neural networks with multiple exits, targeting training stability and classifier collaboration. These techniques improve the efficiency of MSDNet across image-recognition datasets.
- Three techniques improve training for adaptive neural networks with multiple exits.
- Gradient Equilibrium stabilizes training and resolves conflicts among different classifiers’ learning objectives.
- Inline Subnetwork Collaboration and One-for-all Knowledge Distillation strengthen collaboration among classifiers.
- Across multiple image-recognition datasets, the techniques significantly improve the efficiency of MSDNet.