Why should we add early exits to neural networks?

Simone Scardapane; Michele Scarpiniti; Enzo Baccarelli; Aurelio Uncini

Source-linked AI summary

Why should we add early exits to neural networks?

Simone Scardapane, Michele Scarpiniti, Enzo Baccarelli, Aurelio Uncini

arXiv:2004.12814v2Apr 27, 2020cs.NEcs.LGstat.ML

TL;DR

Deep networks require full-stack execution, creating challenges for efficiency, optimization, and deployment. This paper unifies multi-exit architectures, their design, training, and inference, while reviewing applications and open challenges in distributed settings.

Problem
Full-stack neural networks can be difficult to parallelize and deploy, and may suffer from overfitting and vanishing gradients.
Method
The paper provides a unified overview of neural networks augmented with auxiliary classifiers placed at multiple intermediate points.
Results
Multi-exit networks support efficient inference, computational-cost regularization, and distributed computation across multi-tier platforms.
Takeaways & Limitations
The overview organizes fragmented approaches for designing, training, and deploying multi-exit networks in time-constrained and distributed scenarios.
Takeaways & Limitations
Open challenges remain in integrating multi-exit networks with distributed Fog and IoT environments, including asynchronous training, exit placement, inference, and communication costs.

Abstract

from arXiv · show

Deep neural networks are generally designed as a stack of differentiable layers, in which a prediction is obtained only after running the full stack. Recently, some contributions have proposed techniques to endow the networks with early exits, allowing to obtain predictions at intermediate points of the stack. These multi-output networks have a number of advantages, including: (i) significant reductions of the inference time, (ii) reduced tendency to overfitting and vanishing gradients, and (iii) capability of being distributed over multi-tier computation platforms. In addition, they connect to the wider themes of biological plausibility and layered cognitive reasoning. In this paper, we provide a comprehensive introduction to this family of neural networks, by describing in a unified fashion the way these architectures can be designed, trained, and actually deployed in time-constrained scenarios. We also describe in-depth their application scenarios in 5G and Fog computing environments, as long as some of the open research questions connected to them.

1 Introduction

Deep networks rely on sequential stacks of differentiable layers and gradient propagation from the final layer, but very large stacks remain difficult to implement efficiently.

Deep networks are typically sequential stacks of differentiable layers trained by propagating gradients inward from the final layer.
Modern techniques such as residual connections improve optimization, but the implementation of very large layer stacks still poses challenges.

1 INTRODUCTION

Deep networks face parallelization, deployment, overfitting, and gradient-flow challenges. Multi-exit networks address these concerns while organizing fragmented research into a unified overview of design, training, and deployment.

Very deep networks are hard to parallelize and deploy in resource-constrained or distributed settings, while overfitting and vanishing gradients can persist.
Smaller architectures can classify many patterns, including ImageNet features from a single convolutional layer exceeding 30% top-5 accuracy.
Multi-exit networks add auxiliary classifiers at separate points to address these problems and support layered training, efficient inference, and computational-cost regularization.
The overview unifies fragmented literature that uses names such as cascaded networks, IDK networks, and deeply-supervised models.
The paper covers multi-exit models, auxiliary-component placement, training strategies, inference, biological plausibility, and distributed computation.

2 RELATED WORKS

Related work connects multi-output networks to efficient inference, distributed Fog Computing, layerwise training, and other neural-network approaches that do not fit the paper’s model.

Fast neural-network execution is especially important in resource-constrained applications such as mobile and IoT systems.
Multi-output networks reduce processing time for most inputs through early exits rather than designing one efficient network for every input.
Multiple exits can support distributing model training and evaluation across multi-tier platforms, including Fog Computing environments.
Multi-output networks make neural networks stage-wise classifiers and enable layerwise training strategies.
Boosting classifiers, kernel methods, progressive growing, and clustering methods are related research lines but do not fit this model.

3 Description of the model

The model augments a layered backbone with auxiliary predictors at selected intermediate points, producing multiple predictions that can support training, efficient inference, and distributed computation.

3.1 Basic neural networks: A backbone is a cascade of L differentiable operators whose intermediate outputs h_i form embeddings and whose final output is the prediction.
3.1 Basic neural networks: A basic network must evaluate all L layers before predicting, which may be harmful through unnecessary overfitting and energy consumption.
3.2 Neural networks with early exits: Selected indices C identify intermediate points where auxiliary classifiers or regressors c_i transform embeddings h_i into early predictions y_i.
3.2 Neural networks with early exits: Running the full network yields a sequence of potentially increasingly refined and accurate predictions from the different exits.
3.2 Neural networks with early exits: Design requires choosing auxiliary models, exit number and placement, training strategy, and inference mechanism.
3.2 Neural networks with early exits: Training can be joint across exits, layer-wise with freezing, or performed on a pretrained backbone followed by separate auxiliary-classifier training.

4 PLACEMENT AND OPTIMIZED DESIGN OF THE EARLY EXITS

Early-exit placement balances computational savings against accuracy and design complexity. The paper describes percentile-based, optimization-based, greedy, and hyper-parameter-tuned strategies for selecting exit locations and auxiliary classifiers.

Auxiliary classifier design: Auxiliary classifiers are commonly simple linear models or small one- or two-hidden-layer networks, with dimensionality reduction needed for high-dimensional convolutional features.The classifier design depends on whether the preceding network component is fully connected or convolutional.
Placement of early exits: Exit locations can be placed at network-cost percentiles to target specific inference-computation budgets.The approach estimates each layer’s relative computational cost and places exits at selected percentiles of total cost.
Placement of early exits: The efficiency benefit of an early exit depends on its cumulative computation cost and the fraction of inputs that exit there.The paper denotes these quantities by γ_i and I_i when evaluating an exit’s efficiency.
Placement optimization: Optimal placement becomes computationally infeasible for large networks because the number of possible M-exit placements grows exponentially with depth.Greedy algorithms provide potentially sub-optimal alternatives for evaluating placements across the network.
Placement optimization: Greedy placement can retain exits using a tunable threshold, while broader hyper-parameter optimization can tune the architecture and exit locations jointly.The threshold is described as controlling final validation accuracy, and alternative procedures compare constant and adaptive auxiliary classifiers.

5 Training neural networks with early exits

Training early-exit networks commonly combines the final loss with auxiliary losses through joint training, although layer-wise alternatives also exist. The cited analysis reports faster convergence under stated assumptions, while the relationship between the local and standard optimization problems remains open.

Joint training: Joint training combines the final loss with weighted auxiliary losses from intermediate classifiers using stochastic gradient descent.Earlier auxiliary classifiers are often assigned lower weights; Inception, for example, uses α_i = 0.3.
Joint training: Joint training has been applied across tasks including face recognition, detection, segmentation, pose estimation, saliency detection, metric learning, attention prediction, and super resolution.The same approach is also used for recurrent networks performing sequence classification or regression.
Convergence properties: Under squared-hinge-loss and ℓ2-regularization assumptions, the cited analysis shows that joint training can improve convergence of early-exit networks.The proof is stated to extend to losses such as cross-entropy.
Convergence properties: The theorem compares SGD solutions obtained by optimizing the final loss alone with those obtained by optimizing the combined joint-training objective.The comparison uses a decaying step size η_t = 1/t and a λ-strong-convexity condition near the optimum.
Convergence properties: The reported relative convergence speed-up has order Θ(exp{ln(T)λ}).Here T denotes the number of SGD iterations under the theorem’s notation.
Open questions: The relationship between the inner local optimization problem and the standard optimization problem remains an open question.The authors also note that more efficient sampling of patterns across exits may improve the strategy but has received limited attention.
Joint-training variants: A variant adaptively merges intermediate predictions instead of merging auxiliary losses.This provides an alternative to the standard joint-training formulation.

5.3 Layer-wise training

Layer-wise training optimizes multi-exit networks incrementally, freezing previously trained layers and adding subsequent classifiers. This can reduce gradient-related regularization problems and has formal and empirical support for scalable training.

5.3 Layer-wise training: Auxiliary predictions can be combined through fixed pooling, confidence-weighted recursive aggregation, or an additional recurrent component.The confidence-weighted approach assigns each exit a binary classifier that estimates how much its prediction should be trusted.
5.3 Layer-wise training: Layer-wise training alternates freezing the previous layer and optimizing the next layer's weights, repeating this process through the architecture.The procedure begins by training the first auxiliary classifier, then replaces training inputs with learned embeddings before training each subsequent layer.
5.3 Layer-wise training: Layer-wise training should make vanishing and exploding gradients less prominent because each optimized layer is relatively small.The relevant layer size depends on the feature extractor and auxiliary classifier components.
5.3 Layer-wise training: Formal analyses bound final prediction error by O(L^2ϵ) when each layer-wise optimization incurs error at most ϵ.The analyses also report increasing linear separability with added layers and demonstrate scaling to ImageNet.
5.3 Layer-wise training: Separate training decouples the backbone from each auxiliary classifier and can also accommodate non-neural intermediate classifiers.The approach is depicted as the separate-training alternative in Figure 3.

6 INFERENCE IN MULTI-EXIT NETWORKS

Inference in multi-exit networks may discard auxiliary classifiers, aggregate their predictions, or stop computation at an input-specific exit. Hybrid training strategies such as freezeout progressively reduce selected layers' learning rates to zero.

6 INFERENCE IN MULTI-EXIT NETWORKS: Figure 4 illustrates progressively freezing layers during training, with each layer becoming fixed at its assigned freezing point.The example uses L = 3, T = 150, and η_i(0) = 0.1.
6 INFERENCE IN MULTI-EXIT NETWORKS: Freezeout anneals individual layers' learning rates to zero at selected training iterations, combining joint-training accuracy with partial layer-wise speed-up.The freezing points are chosen across the training budget, and the resulting schedule is illustrated for a three-layer network.
6 INFERENCE IN MULTI-EXIT NETWORKS: At inference, auxiliary classifiers may be discarded, combined into a joint prediction, or used to stop forward propagation at an input-specific depth.The final option processes each input only until its selected exit.

6 INFERENCE IN MULTI-EXIT NETWORKS

Early-exit inference selects exits using prediction confidence and can substantially reduce computation while avoiding later incorrect predictions. The architecture can explicitly optimize average inference cost alongside accuracy.

6 INFERENCE IN MULTI-EXIT NETWORKS: Early-exit schemes significantly improve inference time and can prevent overfitting caused by later predictions overturning an already correct intermediate prediction.This latter phenomenon is described as over-thinking.
6 INFERENCE IN MULTI-EXIT NETWORKS: Normalized prediction entropy is compared with an exit-specific threshold to decide whether classification should terminate early.The entropy is normalized between 0 and 1, but separate thresholds may be needed to preserve average accuracy.
6 INFERENCE IN MULTI-EXIT NETWORKS: A single threshold can be iteratively tuned toward a desired accuracy level using the gap between current and target accuracy.The cited strategy is also proved convergent.
6 INFERENCE IN MULTI-EXIT NETWORKS: Multiple exits enable per-input accuracy-efficiency optimization rather than selecting one efficient architecture for every input.The average computational cost can be defined using exit-processing costs and incorporated into model hyperparameter selection.
6 INFERENCE IN MULTI-EXIT NETWORKS: A differentiable soft approximation of cascaded computational cost can regularize the network toward an explicit accuracy-efficiency trade-off.For pretrained backbones, auxiliary classifiers can instead be trained separately under computational-time constraints.

7 Additional topics

Multi-exit networks connect layer-wise training with biological plausibility, distributed multi-tier computation, and the information bottleneck principle. These applications introduce practical opportunities alongside unresolved questions about accuracy, deployment, and theoretical assumptions.

7 Additional topics: A biologically motivated layer-wise variant uses random forward and backward matrices with local error terms, improving plausibility while performing below classical layer-wise training.The construction replaces matrix M used by an auxiliary classifier with a separate random matrix K during the backward pass.
7 Additional topics: Matching state-of-the-art classification while preserving biological plausibility remains an open research question.Proposed extensions include projected targets, parallel optimization buffers, and adaptive target computation.
7 Additional topics: In multi-tier platforms, auxiliary classifiers can be distributed across tiers, with communication costs included in exit design or model selection.This supports deployment across settings such as 5G, IoT, and fog computing, where edge decisions may be delegated to more powerful centers.
7 Additional topics: Layer-wise training naturally supports distributed adaptation because each tier can train its own model before sending new features onward.This is useful when tiers adapt to new data without sending all information to a centralized controller.
7 Additional topics: The information bottleneck principle motivates layer-wise training that maximizes information about targets while minimizing information about inputs.Its approximation replaces local loss terms and requires multiple auxiliary classifiers acting as discriminators between class-specific embeddings.
7 Additional topics: For deterministic networks with continuous inputs, the information-bottleneck term is infinite in practice and must be replaced by a noisy approximation.This is an explicit implementation caveat for the formal objective.

8 OPEN RESEARCH CHALLENGES

The paper identifies open challenges in establishing the formal properties, broader applicability, and practical deployment of multi-exit neural networks. It also frames these architectures as a step toward more complex, conditionally connected systems.

Formal analysis remains incomplete, including quantifying the relative generalization gap of individual components.
The usefulness of multi-output networks remains to be tested on graph convolutional and transformer architectures beyond their primary CNN applications.
Distributed deployment still requires jointly addressing architectural constraints, accuracy, communication costs, placement, and efficient inference.
Multi-exit architectures offer a natural route beyond purely feedforward systems toward multiple networks that conditionally connect, infer, or route information to one another.