Adaptive AI: Energy Efficient Multi-exit TinyML on Intelligent Vision Systems at the Edge

Luca Crupi; Lorenzo Lamberti; Alessandro Giusti; Daniele Palossi

Source-linked AI summary

Adaptive AI: Energy Efficient Multi-exit TinyML on Intelligent Vision Systems at the Edge

Luca Crupi, Lorenzo Lamberti, Alessandro Giusti, Daniele Palossi

arXiv:2609.11939v1Jul 7, 2026cs.ARcs.CVcs.DC

TL;DR

Fixed-depth TinyML inference spends the same computation on inputs of differing complexity, motivating adaptive execution under MCU resource constraints. This paper deploys a confidence-gated, four-exit MobileNetV2 on GAP9 and reports lower computation, latency, and energy with about a 1% accuracy loss versus the full-depth baseline.

Problem
Early-exit inference is well established on high-end devices but remains marginal on resource-constrained MCUs because added exits create critical memory, compute, and energy overheads.
Method
The paper extends MobileNetV2 with four confidence-gated exits, compares parallel and sequential training, and deploys the quantized model on the GAP9 SoC.
Results
41% lower average computation, 29% lower inference time, and 24% lower energy were achieved versus single-exit MobileNetV2, with about a 1% accuracy loss from its 80.5% Top-1 accuracy.
Takeaways & Limitations
Multi-exit inference can provide adaptive TinyML execution on sub-100 mW MCUs while preserving near-baseline accuracy and improving computational efficiency.

Abstract

from arXiv · show

Traditional TinyML systems for edge devices achieve high accuracy by relying on fixed-depth models that require a constant number of multiply-accumulate (MAC) operations regardless of the input complexity. This approach wastes critical resources in battery-powered Internet-of-Things (IoT) devices and limits the real-time performance of edge cyber-physical systems. Multi-exit execution schemes mitigate these issues and are widely used on high-end devices such as GPUs, but are rarely exploited on edge IoT devices because they require substantial rethinking given their strict memory and computational constraints. We address these aspects by designing and deploying, on an ultra-low-power GWT GAP9 System-on-Chip (SoC), a novel multi-exit computational scheme, demonstrating it on a MobileNetV2 convolutional neural network (CNN) for the ImageNet-100 classification task. Our approach introduces multiple exits at different CNN depths, each with a confidence-based gating mechanism that dynamically and autonomously decides whether to continue or stop inference. Comparing our multi-exit strategy to the standard MobileNetV2 on a GAP9 SoC, we show a 41% reduction in the average computational cost (from 313 MMAC to 185 MMAC), a 29% lower inference time (from 49 to 35 ms), and an energy saving of 24% (from 2.1 to 1.6 mJ per frame). All these improvements come with a ~1% loss in accuracy compared to the full-depth MobileNetV2, which achieves 80.5%. Finally, comparing our adaptable multi-exit scheme with a third-party state-of-the-art adaptive CNN, also deployed on the GAP9, we achieve more than 2x its computational efficiency, increasing it from 8.1 to 17.2 MAC/cycle.

I. INTRODUCTION

Fixed-depth TinyML models waste resources because they execute the same computation for every input, while early-exit methods adapt computation to input complexity. This paper brings confidence-gated multi-exit inference to an ultra-low-power MCU using a four-exit MobileNetV2 on ImageNet-100.

Early exits let easy inputs stop at intermediate classifiers while harder inputs continue to deeper layers.The approach adapts computation using confidence-based gating at multiple network depths.
Early-exit methods remain marginal on resource-constrained MCUs because added classifier heads impose critical memory, computation, and energy overheads.These constraints are substantially more stringent than on GPU-class devices.
The system extends MobileNetV2 with four exits, analyzes parallel and sequential training, and deploys the quantized model on a GWT GAP9 SoC.Each exit adds approximately 25 k parameters and uses a confidence threshold to decide whether inference continues.
41% fewer average operations, 29% lower inference time, and 24% lower energy were achieved versus single-exit MobileNetV2 on GAP9, with only about 1% accuracy loss.Average operations fell from 313 to 185 MMAC, inference time from 49 to 35 ms, and energy from 2.1 to 1.6 mJ per frame.
Parallel training produced higher early-exit accuracy than sequential training while preserving approximately 80.5% final-exit accuracy.The study evaluates accuracy, operations, power, and energy using float16 and int8 deployments.

II. RELATED WORK

Prior multi-exit research spans GPU-class and embedded platforms, but constrained deployments must balance adaptive computation against memory, overhead, and training complexity. The reviewed methods therefore differ substantially in architecture, platform, and reported resource use.

Adaptive multi-exit methods have been studied from unconstrained GPUs to low-power embedded systems, with Table I comparing platforms and reported properties.N.A. denotes unavailable information, while N.A. (GPU) identifies an unspecified GPU-based platform.
GPU-oriented methods commonly trigger classification exits using softmax confidence or entropy, while regression methods use output stability or predictive uncertainty.Some approaches use exit-specific thresholds stored in lookup tables or derived at test time.
MSDNet reduces the accuracy gap of shallow classifiers through dense multi-scale connectivity but requires 16.8 M parameters, creating a severe constrained-device memory burden.Its dense connectivity increases memory demands relative to small embedded platforms.
Multi-exit training can suffer gradient conflicts because supervision from different exits competes during backpropagation.Deep Feature Surgery addresses this through exit-specific feature partitioning and feature referencing.
Embedded early-exit systems have demonstrated dynamic latency and energy reduction across low-power boards and tasks, including STM32 deployments and a two-exit Cortex-M0 system.The reviewed Cortex-M0 system used confidence thresholds and a 73.2 kB model for CIFAR-10.
The proposed system reports 17.2 MAC/cycle on GAP9, more than twice the 8.1 MAC/cycle reported for a prior adaptive CNN.The comparison attributes the gain to higher parallelization.

A. Baseline CNN

The baseline is a standard MobileNetV2 configured for 224×224 ImageNet-100 classification, with seven inverted-residual stages and a conventional final classification head. Training uses the ImageNet-100 subset and a separate test set.

MobileNetV2 uses 224×224 inputs, width multiplier 1.0, seven inverted-residual stages, and a final 1280-channel projection.The baseline classification head applies batch normalization, ReLU6, global average pooling, and a fully connected softmax layer.
The baseline architecture is the full MobileNetV2 model, corresponding to exit 4 in the paper’s multi-exit design.This establishes the reference model for later adaptive-exit comparisons.
ImageNet-100 contains 100 classes with 1300 images per class for training and validation and 50 test images per class.Images are resized to 224×224 pixels and normalized to the [0, 1] range.

B. Embedded platform

The GAP9 platform combines heterogeneous RISC-V processing, FPUs, and an int8 accelerator for parallel CNN execution. The proposed MobileNetV2 adds three intermediate heads, distributes four exits across computation budgets, and uses confidence thresholds to adapt inference depth.

B. Embedded platform: GAP9 combines a single-core Fabric Controller with a nine-core RISC-V cluster, four mixed-precision FPUs, and the NE16 int8 accelerator.The cluster delivers up to 15.6 GOp/s and NE16 up to 150 GOp/s.
Adaptive multi-exit CNN: The architecture adds three intermediate classification heads to MobileNetV2 so simple inputs can exit early and harder samples can use deeper features.The final exit remains the original MobileNetV2 output.
Adaptive multi-exit CNN: The four exits require approximately 35.9, 94.4, 175.0, and 313.0 MMAC, spanning 11%, 30%, 55%, and 100% of full-network computation.Exit placement is chosen to distribute accuracy–computation operating points while keeping head costs negligible.
Adaptive multi-exit CNN: Parallel training jointly minimizes all exit losses, whereas sequential training freezes previously trained components while optimizing exits one at a time.Parallel optimization is intended to encourage cross-exit knowledge transfer and consistent feature learning.
Adaptive multi-exit CNN: At test time, each exit compares its maximum softmax confidence with threshold τ and returns immediately when the threshold is met.Changing τ controls the trade-off between average inference cost and accuracy.

B. Deployment

The GAP9 deployment converts trained PyTorch models to ONNX and uses GAPflow to manage tiling and tensor movement across the memory hierarchy.

B. Deployment: GAPflow automatically designs tiling and tensor movement across GAP9’s L1, L2, and off-chip RAM after PyTorch-to-ONNX conversion.The deployment uses float16 and int8 models on RISC-V cores, plus an int8 variant using NE16.

A. Image classification performance

Multi-exit accuracy increases with deeper exits, while confidence-based inference dynamically trades accuracy for computational savings and can occasionally outperform always using the final exit.

A. Image classification performance: Parallel training raises early-exit accuracy to 34.6%, 54.8%, and 77.8%, while the final exit remains comparable at 80.3%.Sequential training reaches 21.4%, 39.8%, and 58.3% at the first three exits, with 80.5% at the final exit.
A. Image classification performance: Parallel training improves shallow exits because all exits jointly update shared backbone features, unlike sequential training with partially frozen layers.Sequential training avoids gradient conflicts but limits shallow-layer adaptation.
A. Image classification performance: The confidence threshold determines whether inference stops at the first exit, continues to the last exit, or distributes samples across intermediate exits.τ = 0 forces first-exit termination, whereas τ = 1 forces last-exit termination.
A. Image classification performance: At τ = 0.7, inputs span all four exits, with 22% reaching exit 3 and 51% reaching exit 4 while accuracy remains within 1% of always exiting at exit 4.Lower thresholds route more samples to early exits; higher thresholds route more samples deeper.
A. Image classification performance: At τ = 0.9, the multi-exit model reaches 80.7% accuracy, exceeding the single-exit baseline by 0.4%.The comparison counts samples corrected by multi-exit inference versus samples lost relative to always using exit 4.

B. Embedded system performance

On GAP9, deployment variants show that NE16 provides the strongest performance-energy trade-off, while dynamic early exiting reduces latency and energy across configurations.

B. Embedded system performance: NE16 reduces int8 latency to 22.2, 26.8, 37.5, and 49.3 ms across exits 1–4, with energy consumption between 0.9 mJ/frame and 2.1 mJ/frame.The int8 model retains a 2.5 MB memory footprint in this configuration.
B. Embedded system performance: With NE16 and τ = 0.5, 0.7, and 0.9, the multi-exit model achieves 76.1%, 79.5%, and 80.7% average accuracy with latency of 31.7, 35.0, and 38.6 ms.These measurements compare dynamic multi-exit inference with the exit 4 CNN.
B. Embedded system performance: Early exiting reduces float16 latency from 207.9 ms at exit 4 to 138.9 ms at τ = 0.7, while energy falls from 9.3 to 6.5 mJ/frame.The reported reductions are 33% for latency and 30% for energy.
B. Embedded system performance: Power measurements cover the GAP9 SoC and off-chip memories, excluding the camera, with waveforms evaluated for float16, int8 cluster, and int8 NE16 variants.The waveform scenario always produces predictions at exit 4, corresponding to τ = 1.0.
B. Embedded system performance: The float16 variant averages 44.83 mW, while int8 cluster averages 48.65 mW because quantization reduces idle periods despite similar peak power.Idle power remains 17.38 mW across configurations.

C. Discussion

The GAP9 evaluation shows higher computational efficiency than a prior tiny adaptive CNN and estimates multi-day operation under a specified battery-powered sensing workload.

C. Discussion: 17.2 MAC/cycle gives the multi-exit MobileNetV2 more than 2× the computational efficiency of the prior CNN’s 8.1 MAC/cycle final exit.The comparison uses full-network execution on the same GAP9 NE16 platform.
C. Discussion: The prior CNN’s maximum 0.4 MMAC workload limits hardware parallelism, whereas the deeper MobileNetV2 improves utilization of the cluster and NE16 accelerator.The paper attributes the prior system’s lower efficiency to data-movement and scheduling overhead dominating execution time.
C. Discussion: Under a 1000 mJ battery scenario with one image every 30 minutes for 12 daylight hours, the system averages 3.65 mJ per frame and can operate up to 11.4 days.The estimate includes camera and LoRaWAN transmission energy and does not rely on energy harvesting.

VI. CONCLUSIONS

The four-exit MobileNetV2 system brings confidence-based adaptive inference to the GAP9 SoC, with parallel training improving early-exit accuracy. It reduces latency and energy with negligible accuracy loss and more than doubles computational efficiency over a prior adaptive CNN.

Parallel training improves accuracy at exits 1, 2, and 3 by 13.2%, 15.0%, and 19.5%, respectively, while the deepest exit reaches approximately 80% accuracy.
The deployed four-exit MobileNetV2 achieves more than 2× the computational efficiency of a state-of-the-art adaptive CNN, increasing from 8.1 to 17.2 MAC/cycle.The paper attributes this improvement to the system’s higher level of parallelism.