Calibration-Aware Uncertainty Cascades for Efficient Heterogeneous Model Collaboration

Yilin Zhang; Han Jiang; Cai Xu; Ying Liu; Wei Zhao

Source-linked AI summary

Calibration-Aware Uncertainty Cascades for Efficient Heterogeneous Model Collaboration

Yilin Zhang, Han Jiang, Cai Xu, Ying Liu, Wei Zhao

arXiv:2609.11446v1Sep 10, 2026cs.AI

TL;DR

Existing routers and raw-confidence cascades struggle to provide transferable routing or reliable thresholds across heterogeneous model pools. CAUC independently calibrates model confidence and uses it for shared-threshold escalation and selective fusion, improving performance–cost trade-offs across language and image benchmarks while reducing strong-model use and GFLOPs. Its image limitation is weaker efficiency when the small model performs poorly on out-of-distribution data.

Problem
Routers are tied to tasks and model pools, while raw-confidence cascades lack consistent reliability semantics across heterogeneous models.
Method
CAUC independently calibrates each model and uses calibrated confidence for early acceptance, escalation, and selective output fusion; CAUC-RF recursively integrates earlier predictions.
Results
Across six LLM benchmarks, CAUC improves the accuracy–cost trade-off while avoiding about 47% of strong-model calls; across three image datasets, it maintains or improves performance while reducing measured GFLOPs by up to 57%.
Takeaways & Limitations
Calibrated confidence provides a shared decision interface for heterogeneous model collaboration without training a pool-specific router.
Takeaways & Limitations
On iWildCam OOD, CAUC exceeded both routers because single-model routing avoids the cascade’s double evaluation of deferred examples.

Abstract

from arXiv · show

Heterogeneous model collaboration seeks to exploit the complementary strengths of different models to balance predictive performance and inference cost. Existing approaches typically rely either on trained routers, which tie routing decisions to a fixed task and model pool, or on raw-confidence cascades, whose thresholds lack consistent reliability semantics across heterogeneous models. Consequently, these approaches adapt poorly to changing model pools and deployment budgets. We propose Calibration-Aware Uncertainty Cascades (CAUC), a simple post-hoc framework that independently calibrates each model's confidence and selects deployment policies using validation data. The resulting calibrated confidence scores establish a common reliability scale for accepting an early prediction, invoking a stronger model, or selectively combining model outputs. This unified decision criterion decouples deployment policies from any particular model pool or operating budget. We further show theoretically that calibration gives confidence thresholds an explicit selective-risk interpretation, whereas uncalibrated scores offer no comparable reliability guarantee. Extensive experiments demonstrate that, across six language benchmarks, CAUC achieves an average relative accuracy improvement of 1.9% over strong-model-only inference while avoiding approximately 47% of strong-model calls. On image classification benchmarks, it maintains or improves predictive performance while reducing measured GFLOPs by up to 57%.

Introduction

Heterogeneous deployments must balance predictive performance against computation, but existing routers and raw-confidence cascades are tied to specific model pools or unreliable thresholds. CAUC uses independently calibrated confidence to support shared-threshold escalation and selective fusion while adapting through post-hoc updates.

Introduction: The deployment objective is predictive performance under a computation or service budget rather than accuracy alone.Compact models are cheaper, while larger models generally provide stronger performance at higher latency and computation.
Introduction: Trained routers require routing supervision and are coupled to tasks, candidate model pools, and cost definitions.Model or budget changes may require new performance labels or retraining.
Introduction: Raw-confidence cascades can assign different empirical accuracies to the same score across models and discard earlier predictions after deferral.Separate thresholds complicate model replacement and cascade expansion.
Introduction: CAUC independently calibrates heterogeneous models and uses calibrated confidence for early acceptance, escalation, and selective fusion.The framework separates model-specific reliability estimation from deployment policy.
Introduction: CAUC's contributions include a common reliability scale, lightweight calibration and policy updates, selective-risk theory, and evaluation across language and image benchmarks.The experiments cover 6 LLM benchmarks and 3 image classification datasets.

Related Work

Prior work uses model routers or adaptive cascades to allocate inference, but both face transfer and thresholding challenges across tasks and heterogeneous models. CAUC retains the cascade paradigm while explicitly calibrating stopping statistics and selectively fusing deferred predictions.

Model Routing: Model-routing methods estimate query–model compatibility using benchmark outcomes, preference data, representations, or graph structure.These approaches exploit specialization within a model pool but depend on pool-specific compatibility signals.
Cascades and Adaptive Invocation: Cascades invoke models in increasing cost order and stop when an intermediate prediction is considered reliable.This design appears in adaptive image inference and language-model service systems.
Cascades and Adaptive Invocation: Cascade decision statistics and suitable thresholds can vary substantially across tasks and models.Confidence-based deferral is effective only when the score relates sufficiently to correctness and downstream recoverability.
Cascades and Adaptive Invocation: CAUC calibrates the stopping statistic explicitly and retains selective fusion for deferred, ambiguous examples.This differs from conventional cascades that replace early predictions after deferral.
Confidence Calibration: Post-hoc calibration aligns predictive confidence with empirical correctness without retraining the base model.Temperature scaling is used for image classifiers and adaptive temperature scaling for language-model option logits.

Calibration-Aware Uncertainty Cascade

CAUC calibrates each model’s confidence, uses a shared reliability threshold for cascading, and selectively fuses deferred outputs when complementarity supports retention. CAUC-RF extends this process by recursively accumulating calibrated evidence and selecting a shared budget-aware stopping threshold.

Problem Formulation: The cascade evaluates the small model first and seeks to reduce the fraction sent to the large model while maintaining or improving large-model performance.Expected cost depends on the fraction of inputs deferred to the large model.
Calibrated Confidence for Cascading: CAUC independently temperature-scales model outputs so confidence estimates share a common correctness interpretation.Calibration changes confidence without changing predicted labels because positive temperature scaling preserves logit ordering.
Calibrated Confidence for Cascading: CAUC accepts the small model when calibrated confidence reaches the large model’s observed calibration accuracy; otherwise, it invokes the large model.Replacing a model requires fitting its scalar temperature and recomputing calibration statistics rather than training a decision model.
Selective Collaboration after Deferral: CAUC enables fusion only when calibration-set complementarity is positive, meaning more confident small-model corrections outweigh harmful replacements.Otherwise, deferred inputs use the large-model prediction directly.
Selective Collaboration after Deferral: When fusion is enabled, standardized calibrated logits provide class evidence while calibrated confidences determine each model’s contribution.The method estimates logit means and standard deviations on the calibration set before combining deferred outputs.
Recursive Fusion Extension: CAUC-RF recursively accumulates calibrated outputs with learned pair temperatures and applies one shared threshold across nonfinal cascade stages.Its threshold is selected on a separate cascade-validation set to satisfy an explicit deployment budget, with the final stage always returning its recursive prediction.

Theoretical Analysis

CAUC gives calibrated confidence thresholds explicit accuracy and cost-sensitive interpretations, with calibration controlling the reliability of accepted predictions and the near-optimality gap of unified thresholding.

Calibration guarantees: Calibration turns a stopping threshold into an accuracy guarantee, whereas uncalibrated confidence can provide no reliability guarantee for accepted samples.The accepted-sample accuracy is lower-bounded by the threshold minus a calibration-error term scaled by the accepted fraction.
Calibration guarantees: CAUC’s analysis separates accepted-sample reliability, governed by calibration, from early-stopping coverage, governed by confidence discrimination.Calibration determines how accurate accepted samples are, while confidence discrimination determines how many inputs can stop early.
Unified-threshold optimality: For a calibrated small model, accepting when its confidence exceeds τ⋆ is optimal under the cost-sensitive fallback objective.The threshold is defined by the fallback loss and balances the small model’s conditional correctness against large-model error and added cost.
Unified-threshold optimality: The threshold rule’s excess expected loss is bounded by average calibration error, becoming zero under perfect calibration.The bound follows because disagreements between the threshold rule and oracle occur only when reported and true correctness probabilities fall on opposite sides of τ⋆.
Deployment implications: The threshold depends on fallback error and deployment cost rather than small-model architecture, enabling reuse after calibrated model replacement.This reuse assumes the fallback setting remains unchanged; CAUC-RF instead selects a budget-specific threshold using cascade-validation data.

Experiments

Across language and image benchmarks, CAUC improves or maintains predictive performance while reducing computation through calibrated cascading, selective fusion, and model-complementarity policies.

Language-model results: 70.43–70.44% accuracy at mean cost 21.73 versus 69.12% at cost 26.00 for large-only inference; CAUC-RF improves large-only accuracy on all six language tasks.CAUC approaches Asymmetric Duo’s 70.84% accuracy while using less than the large-model endpoint cost.
Language-model results: Both CAUC variants occupy the strongest high-accuracy region across MMLU and PIQA budget sweeps, with gains saturating at moderate cost while competing cascades plateau lower.Because Base and CAUC-RF share a routing schedule, their separation at equal costs reflects their prediction rules.
Image-model results: CAUC achieves the best predictive performance in all four image settings, reducing relative GFLOPs from 1.05 for Asymmetric Duo to 0.43–0.64.CAUC-RF leads on Caltech256 and ImageNet, while Base leads on both iWildCam settings; the variants have identical per-dataset costs.
Image-model results: CAUC is most compute-efficient when the small model is accurate, reaching 0.43 and 0.55 relative GFLOPs on Caltech256 and ImageNet while surpassing both routers.On iWildCam, weaker small-model accuracy raises costs to 0.56 and 0.64, and CAUC exceeds both routers on OOD.
Calibration: Calibration moves MnasNet-0.75 confidence near the reliability diagonal and aligns heterogeneous models to a common confidence–accuracy relation.Temperature scaling with T = 0.542 changes confidence without changing the predicted class, supporting a shared reliability threshold.
Complementarity-guided fusion: Across six LLM pairs, complementarity sign selects direct deferral or fusion, after which CAUC-RF improves accuracy by 0.05–0.56 points.The relationship predominantly aligns across Figure 5, but crossings of the zero-gain boundary show complementarity is an admission signal rather than a guarantee.
Longer cascades: Adding models to an ImageNet cascade reduces large-only cost by 63.0% for CAUC and 58.8% for CAUC-RF at the four-model cost minimum.CAUC-RF reaches 86.48% accuracy with five models, 0.26 points below large-only inference at less than half the cost, while further stages raise cost without improving accuracy.

Conclusion

CAUC uses independently calibrated confidence as a shared interface for heterogeneous model collaboration, supporting cascading and selective fusion without a task-specific router. The paper combines this framework with recursive fusion, theoretical reliability guarantees, and post-hoc deployment procedures.

Conclusion: CAUC uses independently calibrated confidence to support early acceptance, selective deferral, and confidence-weighted fusion without training a pool-specific router.CAUC-RF recursively integrates predictions from earlier models in longer cascades.
Conclusion: CAUC’s theoretical analysis links calibration error to accepted-sample reliability and bounds regret for unified-threshold decisions.Without calibration, identical confidence thresholds provide no reliability guarantee; calibration controls accepted-sample accuracy while confidence discrimination controls early stopping coverage.
Conclusion: Across six LLM benchmarks, CAUC improves the accuracy–cost trade-off while avoiding about 47% of strong-model calls.Across three image datasets, it maintains or improves predictive performance while reducing measured GFLOPs by up to 57%.
Conclusion: Fusion and multi-model experiments support model complementarity and show diminishing returns from adding excessive cascade stages.The conclusion frames these findings as evidence for selective fusion and restrained cascade expansion.
Conclusion: The algorithms fit model temperatures and stopping or fusion policies offline using calibration and validation data, then evaluate models from inexpensive to expensive at deployment.Final evaluation labels are not used when fitting parameters or stopping thresholds.

B Additional Experimental Details

The additional experiments specify the evaluated model families, dataset-splitting procedure, baselines, and a supplementary three-model comparison.

Experimental details: Language experiments use models from six listed families, while vision experiments cover seven families including ConvNeXt, EfficientNet, MnasNet, ResNet, ShuffleNetV2, Swin-V2, and ViT.The language families are DeepSeek-R1-Distill, Gemma 4, Ministral 3, Qwen3, Qwen3-VL, and Qwen3.5.
Experimental details: For datasets with labeled test partitions, validation data is split equally into calibration and cascade-validation sets while official test sets remain reserved for final evaluation.PIQA and MedMCQA lack test labels and use their original validation sets as test data.
Experimental details: Baselines distinguish Small-only and Large-only endpoints from learned routing methods, including RouterDC and Benchmark Routing.Small-only evaluates the inexpensive model on every example, whereas Large-only evaluates the stronger model on every example.

C Additional Language-Model Results

Additional language-model results show that CAUC remains competitive across a three-model chain and four further two-model tasks, achieving strong accuracy–cost frontiers at moderate-to-high costs.

Three-model setting: 72.77% accuracy at cost 25.84 outperforms 71.76% at cost 31.00 for Gemma-4-31B in the three-model setting.The stronger CAUC variant wins on five of six tasks, with gains from 0.75 points on MMLU to 2.92 points on SocialIQA, but is 0.33 points lower on PIQA.
Two-model setting: CAUC occupies the upper accuracy region at moderate-to-high costs across all four additional two-model tasks.Its advantage is clearest on MathQA and MedMCQA, while both variants remain stable near 60% accuracy on LogiQA.
Two-model setting: On SocialIQA, Margin Sampling is competitive at intermediate costs, but CAUC retains the highest endpoint.The appendix results therefore extend the broader accuracy–cost frontier beyond the selected main-paper tasks.

D Method-Specific Setup Time

Method-specific policy construction is substantially faster for CAUC than for learned routers, while the theoretical setup formalizes the risks used to compare threshold and fallback decisions.

Offline setup time: 12 seconds is the lowest setup time for Base, compared with 20 seconds for RF and 156–4,891 seconds for learned routers.Base and RF are respectively 13.0–407.6× and 7.8–244.6× faster than the learned-router group; Margin Sampling takes 16 seconds.
Risk definitions: The fallback is a fixed post-deferral policy, such as the large-model prediction or a predetermined fusion rule, with incremental cost weighted by λ.The score-aware oracle chooses between acceptance and fallback using only the small model’s confidence.
Risk definitions: CAUC’s threshold risk accepts the small model above a reference confidence and uses fallback loss below it.The comparison uses calibrated confidence, small-model loss, and a fallback policy whose loss may depend on confidence.

Performance Guarantee

The performance guarantee bounds the excess loss of a unified confidence threshold relative to an oracle that knows the conditional acceptance and fallback losses.

Performance guarantee: The unified-threshold performance gap is established by comparing the threshold policy with a score-aware oracle that selects the lower conditional loss at each confidence.The proof defines disagreement regions where the threshold and oracle choose different actions, with excess loss equal to the absolute true loss difference there.
Performance guarantee: The excess loss is nonnegative because the oracle minimizes conditional loss at every confidence value.Taking expectation over confidence yields the stated upper bound.

Relation to the Main-Paper Result

Under constant fallback loss, the appendix definitions reduce to the main-paper threshold result, and perfect calibration makes the threshold optimal among confidence-based stopping policies.

Relation to the main-paper result: Under constant fallback loss, ¯τ = 1 − Llarge = τ⋆ and ∆fb(¯L) = 0, so the appendix risks recover the main-paper quantities.The generalized proposition therefore reduces to the main-paper result in this special case.
Relation to the main-paper result: Under perfect calibration, the threshold rule is optimal among stopping policies based only on Ps.For direct large-model fallback with λ = 0, the optimal threshold is τ⋆ = 1 − rl = Al, estimated from the calibration set.
Scope: For fusion fallback, near-optimality requires both small calibration error and small variation in fallback quality.The guarantee concerns stopping only, not fusion-rule or recursive CAUC-RF optimality, and excludes finite-sample estimation error.