Source-linked AI summary
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
Yoshitomo Matsubara, Marco Levorato, Francesco Restuccia
TL;DR
Mobile devices increasingly run complex DNNs, while local execution can exhaust resources and edge offloading can suffer delay from variable network and server conditions. This paper surveys split computing and early exiting, compares relevant approaches, and identifies research challenges; its central outcome is a comprehensive state-of-the-art comparison and research agenda.
Problem
Mobile DNN inference strains device computation and energy, while edge offloading can incur significant delay under erratic channel, network, and server conditions.
Method
The paper surveys and compares split-computing and early-exiting strategies, including their approaches, tasks, models, training strategies, and applications.
Results
The paper provides a comprehensive state-of-the-art comparison of SC and EE approaches and identifies research challenges for the field.
Takeaways & Limitations
SC and EE offer intermediate operating strategies that distribute computation or terminate inference early to balance accuracy, delay, and energy.
Takeaways & Limitations
SC and EE research remains primarily focused on image classification, while many application domains and broader structural analyses remain unexplored or incomplete.
Abstract
from arXiv · showhide
Mobile devices such as smartphones and autonomous vehicles increasingly rely on deep neural networks (DNNs) to execute complex inference tasks such as image classification and speech recognition, among others. However, continuously executing the entire DNN on mobile devices can quickly deplete their battery. Although task offloading to cloud/edge servers may decrease the mobile device's computational burden, erratic patterns in channel quality, network, and edge server load can lead to a significant delay in task execution. Recently, approaches based on split computing (SC) have been proposed, where the DNN is split into a head and a tail model, executed respectively on the mobile device and on the edge server. Ultimately, this may reduce bandwidth usage as well as energy consumption. Another approach, called early exiting (EE), trains models to embed multiple "exits" earlier in the architecture, each providing increasingly higher target accuracy. Therefore, the trade-off between accuracy and delay can be tuned according to the current conditions or application demands. In this paper, we provide a comprehensive survey of the state of the art in SC and EE strategies by presenting a comparison of the most relevant approaches. We conclude the paper by providing a set of compelling research challenges.
1 INTRODUCTION
Mobile DNN inference must balance device computation, wireless transmission, latency, energy, and accuracy. The paper surveys split computing and early exiting as intermediate strategies between local and edge execution, then identifies research challenges.
- Motivation: Mobile applications increasingly use complex DNNs, but many mobile devices cannot satisfy their computational and energy requirements.Lightweight models reduce burden but can sacrifice accuracy, including up to 6.4% loss on ImageNet for MnasNet and MobileNetV2 versus ResNet-152.
- Motivation: Complete edge offloading avoids mobile computation but remains vulnerable to fluctuating wireless conditions, mobility, interference, and limited-throughput networks.These factors can increase latency or impair execution in latency-bound applications.
- Approaches: Split computing divides a DNN into mobile-device head and edge-server tail models, while early exiting halts inference at confident intermediate classifiers.Both approaches provide alternatives between local and complete edge execution.
- Paper scope: The paper overviews and compares SC and EE approaches, tasks, and models, provides code repositories when available, and proposes research challenges.It also presents graphical overviews of local, edge, split-computing, and early-exit models.
- System objectives: Split, edge, local, and early-exit strategies seek operating points across accuracy, end-to-end delay, and energy consumption under device and channel constraints.The system is characterized by mobile and edge computing capacities, wireless-channel capacity, inference time, and output-quality degradation.
3 BACKGROUND OF DEEP LEARNING FOR MOBILE APPLICATIONS
Mobile DNN efficiency research reduces inference cost either by designing lightweight models or by compressing existing large models. These strategies face a central trade-off between computational efficiency and predictive accuracy.
- Approach categories: Approaches for reducing DNN complexity on resource-constrained mobile devices fall into lightweight-model design and model compression.The categorization frames the section's discussion of efficiency techniques.
- Lightweight Models: Lightweight model design reduces inference cost by constructing smaller networks, but lower complexity can trade off against model accuracy.MobileNet uses depth-wise and point-wise convolutions to reduce model size and computing load.
- Lightweight Models: Neural architecture search automates aspects of lightweight-model design, including search spaces and platform-aware model construction.NASNet illustrates scaling a search space developed with CIFAR-10 experiments to larger, higher-resolution datasets.
- Model Compression: Model pruning removes parameters and quantization represents them with fewer bits after training a large model.Both methods compress an already trained model rather than directly designing a lightweight model.
- Model Compression: Knowledge distillation trains a smaller student model using informative outputs from a trained teacher to improve the student's accuracy.The method transfers knowledge from a trained model into a predesigned small model.
4 SPLIT COMPUTING: A SURVEY
Split computing studies partition DNNs between mobile devices and edge servers, either without changing the architecture or by injecting bottlenecks. The survey compares partitioning, compression, training, and evaluation strategies, highlighting latency, energy, accuracy, and transferred-data trade-offs.
- 4.1 Split Computing without DNN Modification: SC approaches are categorized as either preserving the original DNN architecture or injecting bottlenecks into early layers.Without modification, the head and tail retain the original model’s corresponding layers and weights; bottleneck injection alters the architecture to reduce transmitted data.
- 4.1 Split Computing without DNN Modification: Original-architecture SC searches partitioning layers using mobile computation, communication delay, edge computation, energy, latency, or privacy costs.Neurosurgeon formalizes inference time as mobile processing, communication, and edge processing, while most studies derive performance profiles for candidate splits.
- 4.2 The Need for Bottleneck Injection: For most considered models, input- or output-layer splits were best for energy and total latency, favoring local computing or full edge computing over intermediate partitions.The surveyed experiments also note that relatively small models and inputs can advantage edge computing, while complex computer-vision tasks motivate more careful splitting.
- 4.3 Split Computing with Bottleneck Injection: Bottleneck-injected models use encoder, decoder, and tail components to reconstruct an intermediate representation before producing the final prediction.The encoder–decoder output may be a noisy version of the original intermediate activation, and accuracy degradation is measured relative to the original output.
- 4.3 Split Computing with Bottleneck Injection: Bottleneck injection and specialized training reduce transferred data, but the trade-off depends on task architecture, accuracy, memory, and mobile workload.Object detectors have narrower bottleneck-placement options; one distillation approach reduced transmitted data by 94% with a 1-point mAP loss.
- 4.4 SC with Bottlenecks: Training Methodologies: Head-network distillation consistently outperformed cross-entropy and knowledge-distillation methods on ImageNet in training cost and trained-model accuracy.Cross-entropy training caused larger accuracy loss than more advanced techniques on complex tasks such as ImageNet, while bottleneck-injected models improved their accuracy–transferred-data trade-off over autoencoder compression.
Knowledge distillation
Knowledge distillation trains a student DNN with signals from a larger pretrained teacher, complementing hard targets with softened teacher outputs. In bottleneck-injected models, teacher-guided training consistently outperforms training without teachers, although the approach remains unaware of introduced bottlenecks.
- Knowledge distillation: Knowledge distillation trains a student model using additional signals from a pretrained, often larger teacher model.Teacher outputs provide soft targets alongside the hard targets used in standard cross-entropy training.
- Knowledge distillation: The distillation loss balances task-specific hard-target loss with a temperature-softened Kullback-Leibler divergence between student and teacher outputs.The balancing factor α weights the hard-target and soft-target terms, while τ controls output softening.
- Knowledge distillation: On ImageNet, all considered bottleneck-injected student models trained with teachers consistently outperform those trained without teachers.The comparison uses original models without injected bottlenecks as teachers.
- Knowledge distillation: Knowledge distillation remains unaware of bottlenecks introduced into DNN architectures and may therefore result in significant accuracy loss.The passage identifies this limitation as similar to the limitation of cross-entropy training.
- Reconstruction-based training: Reconstruction-based training injects autoencoders into existing DNNs and minimizes reconstruction error to compress intermediate feature outputs.An encoder compresses an intermediate output into a bottleneck, and a decoder reconstructs the original representation.
- Reconstruction-based training: Yao et al. report high compression rates with small accuracy loss using encoder-decoder architectures, while adding squared errors between intermediate feature maps as extra loss terms.The additional feature-map losses are inspired by knowledge distillation.
Head network distillation
Head network distillation simplifies only the DNN head around an inserted bottleneck while retaining the teacher’s tail architecture and pretrained parameters. Experiments report lower training cost and higher accuracy than cross-entropy and knowledge-distillation alternatives.
- Training strategy: Head network distillation distills the DNN head containing a bottleneck from a pretrained model to reduce training cost and preserve accuracy.It was proposed to improve on end-to-end and encoder-decoder training, which respectively require harder targets or add model complexity.
- Architecture: The student’s tail is copied from the teacher with identical architecture and pretrained parameters, while only the head architecture is simplified.This preserves the pretrained tail and limits changes to the head portion.
- Training strategy: The student head is trained to mimic the behavior of the teacher head for an input x.The supplied passages introduce the teacher and student head functions in the distillation formulation.
- Results: On ImageNet, head network distillation consistently outperformed cross-entropy and knowledge-distillation training in training cost and trained-model accuracy.The comparison concerns bottleneck-introduced models.
- Extensions: GHND extends head network distillation to complex object detection by mimicking finer intermediate feature maps and improves bottleneck-injected R-CNN detection on COCO 2017 at high compression.The generalized method uses a weighted linear combination of multiple losses, with HND as a special case.
5 EARLY EXITING: A SURVEY
Early exiting adds intermediate classifiers so inference can stop when an exit reaches the desired confidence, avoiding full execution for easy samples. The survey organizes this work across computer vision, NLP, and training methodology, noting different model families across domains.
- Training methodology: Early-exit studies are categorized into joint and separate training methodologies for training the added early classifiers.The survey presents these methodologies in a dedicated training section.
- Core idea: Early exiting introduces exits into DNNs and terminates inference at the first exit whose prediction reaches the desired confidence.This avoids making the entire DNN smaller and can reduce computation for easy inputs.
- Core idea: An example exits at the second classifier when its class probability reaches 0.85, leaving subsequent layers unexecuted.All earlier exits still execute, so complexity increases with the depth reached before termination.
- Design constraint: Early-exit classifiers must remain simple because every exit before the selected one is executed and later branches would otherwise add complexity.The passage specifies that early classifiers should have fewer layers than the layers after their branches.
- Survey scope: The survey covers early exiting in both computer vision and NLP, with CNNs common in vision and Transformer models such as BERT common in NLP.It reports a balance between CV and NLP studies, unlike the stronger CV concentration in split-computing research.
Design approaches
Design approaches adapt early-exit inference to input difficulty, resource conditions, and communication decisions. The surveyed work also addresses reporting dimensions and the training trade-off between computational complexity and overall accuracy.
- Adaptive designs: Dual Dynamic Inference combines input-adaptive selection of sub-networks with resource-adaptive early exiting for cost-efficient, anytime classification.IADI determines which sub-networks execute, while RADI uses early exits according to available resources.
- Evaluation dimensions: The surveyed studies report model accuracy, complexity, transferred data size, energy consumption, latency, and training cost as key evaluation dimensions.These metric labels are defined alongside the early-exit study table.
- Adaptive designs: EE can determine whether an input is transferred to the edge and dynamically adjust the layers used by an auxiliary mobile neural model.These designs target efficient use of communication channels in edge-computing systems.
- Energy-aware designs: EE termination points can decompose a DNN inference pipeline into multiple stages for energy-efficient inference.The design places termination opportunities across the pipeline rather than requiring a single fixed execution depth.
- Training designs: Dynamic loss-weight adjustment addresses the burden of manually tuning early-exit loss weights to balance computational complexity and overall accuracy.The strategy is evaluated on the ResNet models considered by Wang et al.
Inference approaches
Inference approaches extend early exiting across scalable inputs, distributed device-cloud execution, quality enhancement, object filtering, and NLP models. The survey also describes transformer-specific training choices and emphasizes broad code availability in NLP studies.
- Computer vision: Multi-scale early exiting classifies easy samples with smaller neural models and increases input resolution when classification difficulty requires it.The approach adapts image scale to the difficulty of each sample.
- Distributed inference: Progressive inference combines device-cloud computation with early exiting for collaborative distributed inference.The system is designed as a synergistic device-cloud inference approach.
- Computer vision: A neural filter in a head-distilled Keypoint R-CNN identifies images without objects of interest and terminates execution before forwarding the bottleneck output.This connects object filtering with early termination in split-computing pipelines.
- Natural language processing: Early exits are widely applied to transformer-based NLP models, including BERT, RoBERTa, and ALBERT, whose large parameter counts motivate intermediate prediction points.BERT is reported with up to 355 million parameters.
- Natural language processing: Transformer sequence-to-sequence machine translation models can train decoder exits with aligned or mixed training methods.Aligned training optimizes all decoder classifiers simultaneously, while test-time token-level exit choices create additional hidden-state considerations.
- Natural language processing: NLP early-exit research includes sequential reranking, score-based exit decisions, and adversarial-robustness analyses for text-ranking models.These examples use intermediate classifiers in RoBERTa, BERT, and ALBERT architectures.
- Reproducibility: Most NLP early-exit studies provide source code, supported by the open-source Hugging Face Transformers framework and its pretrained models.The framework includes BERT, RoBERTa, and ALBERT models used in the surveyed studies.
- Training methodology: Early-exit studies use two main training classes: joint training and separate training.These approaches are illustrated in the survey’s training-methodology figure.
Joint training
Joint training optimizes all early classifiers and the final classifier simultaneously, typically by minimizing a weighted sum of their cross-entropy losses. An alternative uses knowledge distillation from the final classifier to train earlier exits.
- Joint training: Joint training minimizes a weighted sum of cross-entropy losses for all early classifiers and the final classifier.The correct label is shared across classifiers, and the base model's final classifier counts among the N classifiers.
- Knowledge distillation: Knowledge-distillation approaches treat the final classifier as a teacher and the early classifiers as students.This assumes the last classifier achieves the highest accuracy and that earlier classifiers can learn from it.
Separate training
Separate training uses a two-stage paradigm: first train the base model, then introduce and train early classifiers while keeping the pretrained model parameters fixed.
- Separate training: Separate training first trains the base model, then trains newly introduced early classifiers with the pretrained model parameters fixed.The paper describes this as a two-stage training paradigm and gives BERT fine-tuning as an example of the first stage.
6 SPLIT COMPUTING AND EARLY EXITING: RESEARCH CHALLENGES
The survey identifies practical evaluation as a major research challenge for split computing and early exiting. Reliable assessment must account for datasets, models, wireless conditions, and computing environments rather than relying mainly on small benchmarks.
- Evaluation of SC and EE in more practical settings: Many studies validate SC and EE techniques only on small datasets such as MNIST and CIFAR, raising concerns about input size and compression.The survey calls for practical and convincing evaluation settings because of the field's cross-disciplinary nature.
- Evaluation of SC and EE in more practical settings: Evaluation should consider accuracy, state-of-the-art models and datasets, and the wireless and computing environments.The survey notes that some early studies omit accuracy and that model and dataset choice can affect conclusions about bottlenecks.
- Evaluation of SC and EE in more practical settings: Using small models on simple tasks can produce misleading bottleneck conclusions, while complex ImageNet vision tasks make bottleneck injection challenging.MobileNetV2, ResNet-50, and VGG-16 are cited as examples of models that may be overparameterized for simple classification tasks.
Optimization of bottleneck design and placement in SC
Bottleneck design and placement in split computing must balance communication cost, mobile-side computation, and model accuracy. The survey identifies theoretical optimization of this trade-off as an open research direction.
- Optimization of bottleneck design and placement in SC: Important bottleneck metrics are data size or compression rate, mobile-side head complexity, and resulting model accuracy.A smaller bottleneck representation generally lowers communication cost between the mobile device and edge server.
- Optimization of bottleneck design and placement in SC: Bottleneck injection should not compromise model accuracy, but optimizing representation size, head complexity, and accuracy simultaneously is challenging.Existing studies empirically design bottlenecks and determine their placements rather than solving the trade-off theoretically.
- Optimization of bottleneck design and placement in SC: Theoretical discussion of bottleneck design and placement is identified as an important topic for future work.The survey presents this as a consequence of the difficulty of jointly optimizing the three metrics.
Dynamic control of exits in EE
EE systems must dynamically balance early-classifier confidence against inference efficiency and accuracy, while broader SC/EE design still lacks a strong information-theoretic foundation.
- Dynamic exit control: Manual confidence thresholds for multiple early exits make it challenging to balance inference efficiency against preserving the original model’s accuracy.The paper suggests optimizing which classifier terminates inference based on system characteristics rather than fixing thresholds empirically.
- Application scope: SC and EE remain concentrated on image classification, leaving health monitoring and other application domains comparatively unexplored.Health monitoring may involve high-rate sensor samples, high-complexity DNNs, and lower computing and bandwidth availability than computer-vision settings.
- Information-theoretic perspective: Information bottleneck theory offers a possible compression-based perspective on SC and EE because many state-of-the-art DNNs are over-parameterized.The perspective concerns extracting compressed representations that retain sufficient task-relevant information early in the network.
- Information-theoretic perspective: A strong connection between information-bottleneck formalism and SC/EE techniques remains elusive, despite early attempts and promising interpretive potential.The paper presents this connection as a foundation for future analysis of structural properties in neural-network transformations.