Source-linked AI summary
Accuracy and Robustness of Model Cascades Under Data Perturbations
Pallavi Mitra, Jai Kushwaha, Felix Biessmann
TL;DR
Confidence-based model cascades can save energy, but their routing reliability under degraded inputs remains unclear. This paper evaluates cascades under static and sequential perturbations, identifying distinct failures in routing and model robustness.
Problem
Confidence-based routing reliability in model cascades remains largely unexplored under corrupted or perturbed inputs, despite established clean-data efficiency gains.
Method
The study evaluates a Gatekeeper cascade on static and sequential CIFAR corruption benchmarks to analyze accuracy, routing, and deferral behavior under distribution shift.
Results
Cascade failures arise from routing breakdown when the large model remains robust or model collapse when distribution shift degrades both models.
Takeaways & Limitations
Reliable evaluation of energy-efficient cascades requires examining confidence-based routing under data shifts, not only clean predictive accuracy.
Takeaways & Limitations
The analysis is limited to the evaluated corruption benchmarks and calls for validation on larger datasets, additional model families, and real deployment measurements.
Abstract
from arXiv · showhide
Prediction cascades significantly reduce energy consumption of Artificial Intelligence (AI) models while maintaining high predictive performance. The idea is that easy inputs are routed through a lightweight small model, and difficult uncertain cases are deferred to a larger model. While this design can improve computational efficiency on clean data, its effectiveness depends on the reliability of confidence-based routing. Input degradations, such as static corruptions and sequential perturbations, can shift model confidence and routing decisions. In this paper, we study confidence-based cascade frameworks for image classification and investigate how such degradations affect their confidence-based deferral behavior. We select a model cascade at the pareto-optimum of accuracy, routing quality, and energy consumption that achieves competitive predictive performance with an up to 10-fold decrease in CO$_2$ emissions. We study the behavior of that model cascade under input corruptions and analyze how the cascade's routing decisions change when the input distribution shifts. Our analysis identifies three failure modes. Static corruptions either (1) break the routing signal while the large model remains useful, or (2) degrade both models so deferral no longer recovers accuracy. Sequential perturbations reveal a third mode: predictions stabilize but deferral suppresses, yielding stable but unreliable predictions. These findings demonstrate that energy efficient model cascades require evaluation beyond clean accuracy, with explicit attention to routing reliability under distribution shift.
1 Introduction
Prediction cascades reduce inference effort by routing easy inputs to lightweight models and deferring uncertain cases to larger ones, but degradations can undermine confidence-based routing. This study characterizes how static and sequential perturbations affect accuracy, routing reliability, and deferral quality, distinguishing routing failures from fallback-model degradation.
- Motivation: Prediction cascades adaptively route easy inputs to lightweight models and defer uncertain cases to larger models, reducing computational effort.The Gatekeeper framework uses confidence-based deferral, with a small model attempting each prediction first.
- Problem: Input degradations can shift model predictions and confidence estimates, making cascade behavior under distribution shift a critical practical concern.Examples include compression artifacts, sensor noise, motion blur, and weather-related degradations.
- Contributions: The study quantifies how degradation severity, type, and temporal structure affect cascade accuracy, routing reliability, and deferral quality across datasets.It examines both static corruptions and sequences of data perturbations.
- Contributions: Cascade failure can result from unreliable confidence-based routing or collapse of the large fallback model under severe corruption.The analysis explicitly disentangles routing failure from fallback-model degradation.
- Contributions: Routing calibration is the bottleneck in lower-complexity settings, whereas base-model robustness dominates in higher-complexity settings.The relative importance of these failure modes depends on problem complexity.
2 Related Work
Prior work establishes cascades as efficient sequential model systems with confidence-based routing, but their reliability under distribution shift remains insufficiently studied. Existing robustness research shows that corruptions degrade accuracy and calibration, while cascades add routing reliability as a distinct failure point.
- Model cascades: Prediction cascades process inputs through increasingly capable models, terminating early when confidence is sufficient to reduce computational cost.Classical examples include Viola–Jones, while modern approaches use learned routing for deep models.
- Cascade metrics: Cascade behavior is characterized by cascade accuracy and deferral performance, with higher deferral performance indicating more effective routing.Deferral performance measures the normalized area between realized and ideal deferral curves.
- Research gap: Prior cascade research emphasizes clean, in-distribution efficiency, leaving confidence-based routing reliability under distribution shift largely unexplored.Whether cascade-induced calibration remains effective for corrupted or perturbed inputs is identified as an open question.
- Input perturbations: Standardized corruption benchmarks evaluate robustness to static and sequential degradations, including noise, blur, weather effects, and digital artifacts.These benchmarks show substantial performance drops for DNNs under corrupted inputs, including at moderate severity levels.
- Calibration under shift: Distribution shift can miscalibrate models trained on clean data, and post-hoc calibration methods may not transfer reliably to shifted distributions.Prior work therefore connects corrupted inputs with both accuracy degradation and unreliable uncertainty estimates.
- Cascade-specific risks: Cascade systems add routing reliability as a failure point because corrupted confidence signals can change small-model acceptance and large-model deferral behavior.In Gatekeeper cascades, confidence estimates determine whether inputs are accepted by MS or deferred to ML.
3 Experimental Setup
The experiments evaluate Gatekeeper cascade robustness on static and sequential CIFAR corruptions, using complementary benchmarks and inference-time confidence routing. They compare small-model, large-model, and cascade accuracy with deferral and prediction-stability measures, while estimating energy and CO2 emissions.
- Benchmarks: CIFAR-10-C/100-C test static robustness across 19 corruption types at five severity levels, grouped into Noise, Blur, Weather, and Digital.CIFAR-10-C/100-C applies single-step corruptions at fixed severity levels.
- Benchmarks: CIFAR-10-P/CIFAR-100-P evaluate sequential perturbations with consecutive frames and progressive degradation.Prediction stability is measured by mFP-Casc over sequence length T.
- Model and training: The cascade uses SmallCNN as MS and ResNet-18 as ML, with both models trained on clean data using standard augmentation.MS is trained for 50 epochs with Adam, while ML is trained for 200 epochs with SGD and cosine annealing; ML is then fixed while MS is fine-tuned using Gatekeeper training.
- Evaluation: Corruptions are applied at inference time before confidence-based routing, and evaluation reports MS, ML, and GK accuracy alongside deferral performance (sd).Energy consumption or CO2 emissions are estimated with the codecarbon library.
- Evaluation: Grouped-result bars average across corruption or perturbation types, with error bars showing variability across degradation types rather than independent training seeds.This distinguishes degradation-type variability from variability across training runs.
4 Result & Analysis
The cascade is energy-efficient on clean data but becomes vulnerable under distribution shift. Static corruptions cause routing failure or jointly degrade both models, while sequential perturbations suppress deferral and produce stable but less reliable predictions.
- Clean-data performance: On CIFAR-10, the α = 0.9 cascade matches competitive large-model accuracy while using only 60% of the energy.Table 1 reports clean-data accuracy, deferral performance sd, and estimated CO2 emissions for the Gatekeeper cascade and component models.
- Static corruptions: Static corruptions reduce cascade accuracy with severity, with Noise most damaging and Weather comparatively robust across datasets.On CIFAR-10-C, Noise reduces accuracy from 0.922 to ≈0.38–0.58, while Weather remains around 0.75–0.90; CIFAR-100-C degrades more sharply.
- Static corruptions: On CIFAR-10-C, routing fails despite the large model retaining an advantage, as Noise-1 accuracy is 0.84 versus 0.67 and sd falls from 0.686 to 0.35.Poor deferral causes the cascade to lose ≈19pp of recoverable accuracy.
- Static corruptions: On CIFAR-100-C, severe corruptions degrade both small- and large-model accuracy, eliminating the large model’s advantage and limiting cascade compensation.When the large model cannot correctly identify samples, deferral cannot recover accuracy.
- Sequential perturbations: Sequential perturbations lower accuracy less severely than static corruptions but suppress deferral, yielding reduced flip rates and stable yet less reliable predictions.Accuracy drops from 0.922 to 0.80–0.85 on CIFAR-10-P and from 0.744 to 0.51–0.59 on CIFAR-100-P; Weather reduces mFP-Casc by 67% and 76%, respectively.
5 Conclusion and Future Work
Model cascades can deliver competitive predictive performance with up to 10-fold lower energy consumption on clean data, but data degradation can undermine both accuracy and confidence-based routing. The paper therefore motivates shift-aware routing, uncertainty calibration, and broader deployment evaluation.
- Conclusion: Up to 10-fold reduction in energy consumption accompanies competitive predictive performance on clean data.This establishes model cascades as a per-sample tradeoff between predictive performance and energy consumption.
- Conclusion: Input degradation can cause routing breakdown when the large model remains accurate but corrupted confidence signals misroute samples.This is one identified corruption-induced failure mode affecting reliable efficient inference.
- Conclusion: Model collapse occurs when distribution shift degrades both cascade models, eliminating the accuracy recovery expected from deferral.The observed failure mode depends partly on label-set cardinality, with CIFAR-10’s lower number of classes preserving a routing-related behavior.
- Future Work: Future work will develop shift-aware Gatekeeper thresholds, corruption-aware confidence correction, and uncertainty calibration for degraded inputs.These mechanisms aim to prevent degraded samples from being incorrectly accepted by MS instead of using a fixed clean-data configuration.
- Future Work: Future evaluations will extend to larger-scale datasets, additional model families, and real deployment measurements of latency and energy.