Source-linked AI summary
SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
Stefanos Laskaridis, Stylianos I. Venieris, Mario Almeida, Ilias Leontiadis, Nicholas D. Lane
TL;DR
Mobile CNN inference faces excessive model computation, heterogeneous device capabilities, and unreliable or costly cloud dependence. SPINN combines progressive inference with synergistic device-cloud execution, using a runtime scheduler to co-optimise early exits and CNN partitioning. The system supports performance goals in dynamic environments while maintaining robust operation, accuracy, and availability.
Problem
Mobile CNN inference must handle computationally demanding models, diverse devices, dynamic connectivity, and uncertain cloud availability.
Method
SPINN uses progressive inference and a runtime multi-objective scheduler that jointly tunes early-exit policies and device-cloud partitioning.
Results
SPINN delivers higher performance than state-of-the-art systems across diverse settings without sacrificing overall system accuracy and availability.
Takeaways & Limitations
SPINN supports complex performance goals in highly dynamic environments while guaranteeing robust end-system operation.
Takeaways & Limitations
The prototype scheduler does not explicitly optimise client energy or memory consumption.
Abstract
from arXiv · showhide
Despite the soaring use of convolutional neural networks (CNNs) in mobile applications, uniformly sustaining high-performance inference on mobile has been elusive due to the excessive computational demands of modern CNNs and the increasing diversity of deployed devices. A popular alternative comprises offloading CNN processing to powerful cloud-based servers. Nevertheless, by relying on the cloud to produce outputs, emerging mission-critical and high-mobility applications, such as drone obstacle avoidance or interactive applications, can suffer from the dynamic connectivity conditions and the uncertain availability of the cloud. In this paper, we propose SPINN, a distributed inference system that employs synergistic device-cloud computation together with a progressive inference method to deliver fast and robust CNN inference across diverse settings. The proposed system introduces a novel scheduler that co-optimises the early-exit policy and the CNN splitting at run time, in order to adapt to dynamic conditions and meet user-defined service-level requirements. Quantitative evaluation illustrates that SPINN outperforms its state-of-the-art collaborative inference counterparts by up to 2x in achieved throughput under varying network conditions, reduces the server cost by up to 6.8x and improves accuracy by 20.7% under latency constraints, while providing robust operation under uncertain connectivity conditions and significant energy savings compared to cloud-centric execution.
1 INTRODUCTION
Mobile CNN inference is difficult because modern models are computationally demanding, devices vary widely, and cloud execution depends on unstable networks and costly server resources. SPINN addresses these constraints by jointly adapting device-cloud partitioning and progressive early exits to application requirements.
- Modern CNNs impose excessive computational demands on resource-constrained mobile devices, while low-tier devices struggle to meet application performance needs.
- Cloud-centric inference adds powerful computation but can degrade sharply with poor connectivity and incurs significant hosting costs.
- Device-cloud collaborative inference partitions a CNN between client and server, but execution can fail under severe network deterioration.
- Existing collaborative methods cannot sufficiently satisfy applications requiring simultaneous latency, throughput, accuracy, and device or cloud cost targets.
- SPINN jointly tunes CNN split points and early-exit policies at runtime to sustain performance and meet application SLAs under fluctuating resources.
2 BACKGROUND AND RELATED WORK
Prior work explores CNN optimization, approximate inference, progressive networks, and device-cloud partitioning, but generally lacks coordinated adaptation for dynamic mobile environments. SPINN combines runtime split selection with early-exit control and local-result resilience.
- Prior CNN optimization spans compiler, runtime, custom-cloud, and accelerator designs, often targeting device-only or cloud-only execution.
- Approximate Inference: Approximate inference methods expose accuracy-complexity trade-offs through pruned, quantized, or otherwise varied model versions, but some cannot natively split models across device and cloud.
- Progressive Inference Networks: Progressive inference networks use intermediate exits so easier samples can leave early and avoid further computation.
- Progressive Inference Networks: Existing progressive-inference techniques have not fully exploited distributed execution and application-tailored early exiting for mobile performance.
- Device-Cloud Synergy for CNN Inference: Collaborative systems such as Neurosurgeon select a single split point, whereas SPINN jointly tunes offloading and early-exit policies for dynamic conditions and application requirements.
- Device-Cloud Synergy for CNN Inference: SPINN guarantees a local result, providing resilience to server disconnections.
3 PROPOSED SYSTEM
SPINN combines progressive inference with device-cloud execution and dynamically co-optimises CNN splitting and early exits to satisfy application-level requirements under changing conditions.
- System overview: SPINN distributes progressive early-exit models across device and server, keeping one exit on-device so a result remains available.The confidence threshold is tunable to adapt the accuracy-speed trade-off.
- Offline components: Offline preparation derives the progressive network, identifies split candidates, and profiles exit rates, classifier accuracy, and client-server performance.These measurements provide initial latency estimates for runtime decisions.
- Runtime adaptation: At runtime, the scheduler jointly selects the split point and early-exit policy using profiled timings, target SLAs, and current network and load conditions.The execution engine coordinates partitioned inference while online profiling supports adaptation.
- Progressive inference: Higher confidence thresholds increase accuracy, whereas lower thresholds allow more samples to exit earlier, making confidence a control for processing time.Different CNNs require network-specific threshold tuning because their classifiers become confident at different depths.
- Model splitting: Different split points trade off client, server, and transfer time, while connectivity and compute capabilities change which split is fastest.SPINN therefore cannot rely on a statically selected split point.
4 EVALUATION
SPINN’s evaluation examines its core components and compares it with device-only, cloud-only, and collaborative inference systems. Throughput is evaluated across varied model, device, dataset, and network-speed setups.
- The evaluation compares SPINN with standard device- and cloud-only implementations, state-of-the-art collaborative systems, and a non-progressive ablation.
- Figure 7 reports achieved throughput for varied model, device, and dataset setups as network speed changes.
4.1 Experimental Setup
Experiments use a powerful server and an Nvidia Jetson Xavier AGX client, with power profiles emulating devices of different compute capabilities. Server-side load is also emulated by scaling CNN computation times.
- Experiments use a powerful computer as the server and an Nvidia Jetson Xavier AGX as the client.
- Three Jetson power profiles—30W, 10W, and underclocked 10W—emulate end devices with different compute capabilities.
- Server load is emulated by linearly scaling CNN computation times on the server.
4.2 Performance Comparison
SPINN adapts early exits and device-cloud partitioning to network, server-load, device, and latency-SLA conditions. It achieves the highest reported throughput and improves server-time and accuracy outcomes under latency constraints.
- 4.2.1 Throughput Maximisation: SPINN achieves the highest throughput across all setups, with speedups of up to 83% over Neurosurgeon and 52% over Edgent.
- 4.2.1 Throughput Maximisation: SPINN selects device-only execution at low bandwidth, exploits compression in the mid-range, and uses optimised early exits at high bandwidth.
- 4.2.1 Throughput Maximisation: SPINN’s scheduler jointly adapts the split point and early-exit policy, while Edgent always transmits inputs and uses one intermediary exit for all samples.
- 4.2.2 Server Slowdown: As server load increases, performance deteriorates for remote-execution approaches, while device-only execution remains invariant to server load.
- 4.2.3 Case Study: Latency-driven SLAs at minimal server cost: 68.64% and 82.5% average server-time reductions over Edgent and Neurosurgeon are reported across latency-SLA targets.
- 4.2.3 Case Study: Latency-driven SLAs at minimal server cost: For the 10W device, SPINN and Edgent meet latency constraints as low as 20% for ResNet-56 and 30% for MobileNetV2.
- 4.2.3 Case Study: Latency-driven SLAs at minimal server cost: For deadlines above 50% of on-device latency, SPINN improves accuracy over Edgent by 17.3% on ResNet-56 and 20.7% on MobileNetV2.
- 4.2.3 Case Study: Latency-driven SLAs at minimal server cost: SPINN trades accuracy against stricter latency SLAs and increases attainable accuracy as constraints are relaxed.
4.3 Runtime Overhead and Efficiency
SPINN’s scheduler adapts split and early-exit choices to changing network conditions while keeping runtime overhead small. Under bandwidth variation, it falls back locally at low bandwidth and accelerates inference substantially when bandwidth is high.
- Runtime overhead: The scheduler executes in at most 14 ms, with an 11 ms geometric mean, across the examined CNNs and datasets.Its memory consumption is a few KB, below 1% of Jetson’s RAM, and costs are amortised across multiple inferences.
- Network adaptation: At bandwidth below 5 Mbps, SPINN falls back to device-only execution and lowers the confidence threshold to enable more early exits.For the selected thresholds, accuracy impact remains below 1%.
- Network adaptation: At high bandwidth, SPINN selects an earlier split point and achieves up to 7× more inferences per second than pure on-device execution.The scheduler adapts to running conditions without continuous invocation.
- Scheduler triggering: Using a three-value historical average and a 5% difference threshold, the scheduler is invoked for one-third of bandwidth changes.Small bandwidth changes therefore do not substantially alter the selected split and early-exit strategies.
4.4 Energy Consumption
SPINN’s energy profile depends on the device-cloud configuration and early-exit behavior. Its scheduler can achieve much faster processing than device-only execution while using energy comparable to, or substantially lower than, key baselines.
- Early-exit effects: Early exits reduce local-processing energy in device-only execution by allowing samples to leave the network before its final layers.The energy breakdown identifies local processing as dominant for device-only execution.
- Energy trade-offs: Across inference setups, energy consumption differs by 5×.Device-only execution has the lowest per-sample energy footprint but is also the slowest.
- Energy trade-offs: SPINN deployments use 4.2× less energy than full offloading while delivering faster end-to-end processing.This comparison concerns the scheduler-selected deployments versus full offloading.
- Energy trade-offs: SPINN’s energy consumption is 0.76−1.12× that of on-device execution while providing significantly faster processing.Different configurations vary in both energy and performance.
4.5 Constrained Availability Robustness
SPINN remains usable when network or server availability is uncertain by producing local early-exit results. Compared with non-progressive offloading, it degrades more gradually in accuracy and avoids sharply increasing retransmission latency.
- Accuracy robustness: SPINN preserves local results up to the split point when the server is unavailable, unlike the non-progressive baseline.The baseline has no usable local result and therefore misclassifies under failure conditions.
- Accuracy robustness: SPINN’s accuracy degrades gradually as failure probability increases, whereas the baseline quickly loses accuracy.The comparison is shown for multiple split points against a non-progressive baseline.
- Latency robustness: SPINN’s average latency decreases slightly as failure probability increases because more samples early-exit when the server is unavailable.The non-progressive baseline instead follows an increasing-latency trajectory.
- Overall robustness: Under uncertain availability, SPINN maintains sufficiently accurate and low-latency operation through local exits instead of catastrophic application unresponsiveness.The authors report gradual quality-of-service degradation as offloading failures become more likely.
5 DISCUSSION
SPINN replaces separate cloud and device model deployments with one progressively trained network whose execution is tuned at deployment time. The discussion also identifies energy, split-point, and model-placement boundaries for the prototype.
- Discussion: The conventional deployment process maintains a large accurate cloud model and a compact lower-accuracy device model, creating substantial deployment overheads.SPINN is presented as an alternative to this two-model paradigm.
- Discussion: SPINN requires one network design step and one training process for the backbone and its early exits.This contrasts with maintaining separate cloud and device models.
- Discussion: At deployment, SPINN adapts execution to multiple objectives, environmental conditions, and device and cloud load.The paper describes this as a dynamically customised deployment for mobile settings.
- Limitations and future work: The prototype scheduler does not explicitly optimise client energy or memory consumption.The authors suggest adding energy as an optimisation objective and loading only part of the model to reduce memory footprint.
- Limitations and future work: Candidate split points are restricted to ReLU outputs because their activations are highly compressible.Sub-layer or filter-level splitting would enlarge the scheduler’s search space and add overhead.
- Limitations and future work: The workflow assumes the model is available on both client and server sides.The authors identify incremental offloading and caching as possible responses to edge-resource locality challenges.
6 CONCLUSION
SPINN partitions CNN inference across device-server setups and jointly tunes early exits and partitioning at runtime to support complex goals in dynamic environments.
- SPINN jointly tunes early-exit policies and CNN partitioning through a runtime scheduler.The system uses multi-objective optimisation and a CNN-specific communication optimiser.