PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Confidence-Based Cascade Deferral Suffice?

Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, Sanjiv Kumar

arXiv:2307.02764v2cs.LGstat.ML

TL;DR

Cascades adapt inference cost by deciding when to invoke the next classifier, but the conditions where confidence-based deferral falls short are not fully understood. The paper characterizes an optimal rule and studies post-hoc alternatives, which improve on confidence-based deferral in identified settings. The findings point to the importance of identifying cases where the first model is wrong and the second is right.

  • Problem

    The paper asks when confidence-based deferral, which can work well despite ignoring downstream-model errors, is sub-optimal for cascades.

    Source: §1 · pp. 1–2

  • Method

    The paper characterizes the Bayes-optimal deferral rule and studies post-hoc estimators that approximate its form.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Post-hoc deferral can significantly improve on confidence-based deferral when downstream errors are non-uniform, labels are noisy, or train and test distributions shift.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The Bayes-optimal rule suggests that useful deferral depends on identifying inputs where the first model is wrong and the second is right.

    Source: §6 · p. 10

    Post-hoc Diff-01 can overfit and fail to generalise in the ImageNet-Dog specialist setting, even with a small feedforward network.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. One simple deferral rule employs the confidence of the current classifier, e.g., based on the maximum predicted softmax probability. Despite being oblivious to the structure of the cascade -- e.g., not modelling the errors of downstream models -- such confidence-based deferral often works remarkably well in practice. In this paper, we seek to better understand the conditions under which confidence-based deferral may fail, and when alternate deferral strategies can perform better. We first present a theoretical characterisation of the optimal deferral rule, which precisely characterises settings under which confidence-based deferral may suffer. We then study post-hoc deferral mechanisms, and demonstrate they can significantly improve upon confidence-based deferral in settings where (i) downstream models are specialists that only work well on a subset of inputs, (ii) samples are subject to label noise, and (iii) there is distribution shift between the train and test set.

1 Introduction

Cascades adapt inference cost by deciding whether to use the next, more complex model, and confidence-based deferral often works well despite ignoring downstream-model errors. This paper studies why it works and when it can fail, then evaluates post-hoc alternatives.

1 Introduction

  • For a two-model cascade, the theoretically optimal deferral rule relies on the confidence of both models.The paper characterizes this rule in Proposition 3.1.Source: §1 · p. 2
  • Confidence-based deferral is highly competitive in many regular classification tasks where model 2 consistently estimates the true posterior probability.Source: §1 · p. 2
  • Confidence-based deferral can be significantly sub-optimal with non-uniform downstream errors, label noise, or train-test distribution shift, while post-hoc rules improve on it in these settings.The identified cases include specialist downstream models and samples whose labels are noisy.Source: §1 · p. 2
  • The paper presents its study as the first to identify specific practical settings where confidence-based deferral can be sub-optimal.Its findings are intended to inform when deploying confidence-based cascades is appropriate.Source: §1 · p. 2

2 Background and Related Work

This section contrasts fixed-cost inference with adaptive prediction and introduces cascades and confidence-threshold deferral. The paper focuses on using fixed, pretrained base models while training only the deferral rule.

2 Background and Related Work

  • Adaptive prediction offers a way to lower inference cost for easy samples while reserving full cost for hard ones, unlike approaches that lower cost uniformly.Source: §2.1 · p. 2

2.1 Cascades and Deferral Rules

  • A cascade uses base models, typically of non-decreasing cost, and a deferral rule to select which model handles each input.The study fixes pretrained base models and trains only the deferral rule.Source: §2.1 · p. 2

2.2 Confidence-based Cascades

  • Confidence-based cascades threshold each model’s maximum predictive probability and use the first model whose confidence is sufficiently high.This post-hoc construction does not require retraining the models or deferral rule, though practical underperformance settings were unclear.Source: §2.2 · p. 3

3 Optimal Deferral Rules for Cascades

The section formulates cascade deferral as choosing between two fixed classifiers under an inference-cost constraint, then derives an oracle rule and practical approximations. The optimal choice depends on the difference in the classifiers’ probabilities of being correct, not only the first model’s confidence.

3 Optimal Deferral Rules for Cascades

  • The analysis treats a cascade as choosing between two pre-trained classifiers, with a deferral rule deciding which classifier predicts each input.The authors state that the analysis generalizes to more than two models.Source: §3 · p. 3, §3.1 · p. 3

3.1 Optimisation Objective

  • The objective minimizes errors on the predictions actually used while limiting the fraction of inputs sent to the more costly classifier.A Lagrangian formulation expresses the same trade-off as error risk plus a deferral-cost term; deferral and cost-risk curves assess cascade quality.Source: §3.1 · pp. 3–4

3.2 The Bayes-Optimal Deferral Rule

  • The Bayes-optimal rule defers to the more costly classifier when its probability of being correct exceeds the first classifier’s by more than the invocation cost.It depends on the true posterior, and disagreement with the rule incurs excess risk proportional to the models’ error-probability difference.Source: §3.2 · p. 4

3.3 Plug-in Estimators of the Bayes-Optimal Deferral Rule

  • Plug-in estimators approximate the oracle using labels or both models’ softmax confidences, with the latter relying on probabilities approximating the true posterior.The label-based oracle is impractical at test time, and the oracle estimators cannot provide adaptive computation because they invoke the second model on every input.Source: §3.3 · pp. 4–5

3.4 Relation to Existing Work

  • The two-model cascade risk is a special case of the learning-to-defer risk of Mozannar and Sontag, whose proposition assumes c = 0.The paper generalizes the proposition to cascades with more than two models.Source: §3.4 · p. 5

4 From Confidence-Based to Post-Hoc Deferral

Confidence-based deferral is optimal under an ordering condition on the two models’ confidence, but can fail when downstream error varies across inputs. The section examines post-hoc estimates of the optimal rule, their finite-sample error, and their relation to prior work on rejection and deferral.

4.1 When Does Confidence-Based Deferral Suffice?

  • Confidence-based deferral has the Bayes-optimal deferral curve when model 1 confidence and its relative confidence advantage over model 2 induce the same ordering of inputs.A constant downstream error probability is a sufficient, but not necessary, condition for this optimality.Source: §4.1 · p. 5
  • Confidence-based deferral may fail with specialist downstream models, label noise, or train–test distribution shift.For specialists, it may forward samples where model 2 performs worse; for label-noise cases, it may spend computation without improving chance-level accuracy.Source: §4.1 · p. 6

4.2 Post-Hoc Estimates of the Deferral Rule

  • Post-hoc rules estimate the optimal deferral rule using model 1 outputs, avoiding the need to query model 2 at inference time.The candidate estimators target different quantities, including whether model 1 is wrong while model 2 is correct and model 2’s correctness advantage.Source: §4.2 · p. 6
  • When model 2’s output z is not strongly predictable, a non-trivial predictor must achieve average squared error below Var(z)=E[(z−E[z])²]; otherwise estimation reduces to constant E[z], returning to constant-probability-error, confidence-based deferral.Source: §4.2 · p. 7

4.3 Finite-Sample Analysis for Post-Hoc Deferral Rules

  • The finite-sample excess-risk bound separates irreducible approximation error from estimation error.Approximation error measures the distance from the best scorer in the hypothesis class to the oracle scoring function; the remaining terms quantify estimation error.Source: §4.3 · p. 7

4.4 Relation to Existing Work

  • The deferral setting generalizes learning to reject, whose Bayes-optimal classifier is Chow’s rule, while joint training methods do not fit the paper’s pre-trained, black-box setting.The authors also distinguish their closer study of post-hoc method construction and its cases of limited benefit from prior variants.Source: §4.4 · p. 7

5 Experimental Illustration

Across the illustrated failure settings, post-hoc deferral offers better accuracy–cost trade-offs, while confidence-based deferral remains competitive when the downstream model is a generalist. The results also show that post-hoc estimators can overfit and fail to generalise.

5 Experimental Illustration

  • Post-hoc deferral offers better accuracy–cost trade-offs in the specialist, label-noise, and distribution-shift settings.Deferral rate implicitly measures overall compute cost.Source: §5 · p. 8, Figure 2 · p. 9

5.1 Confidence-Based versus Oracle Deferral

  • When the downstream model is a generalist, Relative Confidence offers little gain over Confidence; as it specializes in dogs, Relative Confidence gains accuracy by selectively deferring dog images.As non-dog training examples decrease, confidence-based deferral may send non-dog images to the dog specialist instead.Source: §5.1 · p. 8, Figure 1 · p. 8

5.2 Confidence-Based versus Post-Hoc Deferral

  • In the specialist setting, post-hoc deferral gains accuracy, especially at low deferral rates, by predicting when model 2 is likely to err.In the generalist setting, confidence-based deferral remains highly competitive.Source: §5.2 · p. 9, Figure 2 · p. 9
  • With label noise, post-hoc approaches achieve higher accuracy than confidence-based deferral across a large range of deferral rates.Confidence-based deferral tends to send noisy samples to a model that performs equally poorly, whereas post-hoc rules can give up on samples with extremely low model 1 confidence.Source: §5.2 · p. 9, Figure 2 · p. 9

5.3 On the Generalisation of Post-Hoc Estimators

  • In the ImageNet-Dog specialist setting with 8% non-dog training images, Diff-01 can overfit and fail to generalise, even with a small feedforward network.The reported model has two hidden layers of 64 and 16 units.Source: Figure 3 · p. 10, §5.3 · p. 10

5.2 Confidence-Based versus Post-Hoc Deferral

  • As distribution shift increases, post-hoc approaches, notably Diff-01, show clearer gains by deferring tail-class inputs to the better-performing ResNet 56.With no shift, post-hoc approaches show little to no gain because both models are sufficiently accurate.Source: Figure 2 · p. 9, §5.2 · p. 10

6 Conclusion and Future Work

The paper concludes that optimal deferral centers on identifying cases where the first model is wrong and the second is right, and that post-hoc rules can improve on confidence-based deferral. It identifies other distribution shifts, refined confidence measures, and non-image tasks as directions for future work.

6 Conclusion and Future Work

  • The Bayes-optimal deferral rule indicates that optimal deferral requires identifying when the first model is wrong and the second is right.The studied post-hoc estimators can improve on commonly used confidence-based deferral.Source: §6 · p. 10
  • The identified failure conditions are not exhaustive; future work could study other distribution shifts, conformal-prediction confidence measures, and analogous trends in natural language processing.The reported results focus on image classification.Source: §6 · p. 10

A.1 Proof of Proposition 3.1

The appendix passages give proof steps for risk and deferral-curve claims, describe a finite-sample argument for a learned deferral scorer, and report experimental compute requirements.

A.1 Proof of Proposition 3.1

  • The proof writes cascade risk as the probability of error when retaining model 1, plus the probability of error after deferring to model 2 and the deferral cost.Source: §A.1 · p. 17

A.2 Proof of Corollary 3.2

  • For fixed classifiers, the corollary proof starts from the Bayes-optimal rejector and expresses risk for an arbitrary rejector to obtain excess risk.Source: §A.2 · p. 18

A.3 Proof of Lemma 4.1

  • Lemma A.1 equates cascade accuracy with E[r(x)β(x)]; the curve argument compares confidence-based and Bayes-optimal rules under a shared ordering condition.Source: §A.3 · p. 18

A.4 Proof of Lemma 4.2

  • The learned scorer's squared-loss target is the difference in correctness between the two base classifiers, and the finite-sample argument separates approximation from estimation error.Source: §A.4 · p. 19

B Amount of Compute Used for Experiments

  • Post-hoc model training takes a few minutes for 20 epochs; base-classifier training time varies by dataset and architecture, with ImageNet models trained on eight TPUs.The listed ImageNet training times are 7 hours for MobileNet V2 and 4 hours for EfficientNet B0.Source: B Amount of Compute Used for Experiments · p. 20

C.1 Training of Models in a Cascade

The appendix specifies training details for the cascade models and the post-hoc deferral rules, including their input features and regularization.

C.1 Training of Models in a Cascade

  • The ImageNet-Dog specialist EfficientNet B0 uses learning rate 0.1, a cosine schedule, 90 epochs, and batch size 512.Source: §C.1 · p. 21

C.2 Training of Post-hoc Deferral Rules

  • The post-hoc scorer is an MLP over model 1's probability outputs, using entropy, the ten highest probabilities, and a one-hot encoding of the predicted class.Source: §C.2 · p. 21
  • The MLP is trained with Adam for 20 epochs on held-out data with L2 regularization weight 0.001; controlling model capacity helps mitigate overfitting.Source: §C.2 · p. 21

D Calibration Analysis

Calibration analysis finds that the first model’s confidence can misestimate when the first model is wrong and the second is right. Additional experiments show confidence-based deferral can underperform with label noise and specialist models, while remaining competitive when the second model is highly accurate.

D Calibration Analysis

  • The first model’s confidence systematically overestimates the chance that it is wrong while the second model is correct.Calibration plots estimate this event in 10 confidence buckets; the overestimate can lead to erroneous forwarding to model 2.Source: D Calibration Analysis · p. 22, Figure 4 · p. 22

E.1 Confidence-Based Deferral Underperforms Under Label Noise

  • On Mini-ImageNet with 60% label noise, Diff-01 significantly outperforms confidence-based deferral.The experiment uses two ResNet-10 models with widths 16 and 64; Figure 5 compares accuracy against deferral rate.Source: §E.1 · p. 22, Figure 5 · p. 23

E.2 When Model 2 is Highly Accurate

  • When the second model is highly accurate, confidence-based deferral is competitive.The experiments use an MLP and a CIFAR ResNet-56 on MNIST.Source: §E.2 · p. 22, Figure 6 · p. 23

E.3 Oracle Curves

  • With a dog-specialist second model, confidence-based deferral is sub-optimal, with a considerable gap to oracle curves.The specialist’s probability of being correct varies across inputs; oracle curves use the true label and serve as upper bounds for post-hoc rules that imitate them.Source: §E.3 · pp. 23–24, Figure 7 · p. 23

E.4 Confidence of a Dog-Specialist Model

In the ImageNet dog-specialist setting, the specialist can be confident on non-dog inputs even though those inputs are outside its training classes. This makes its maximum confidence an unreliable indicator for routing, whereas its probability for the true label is low on labeled non-dog images.

E.4 Confidence of a Dog-Specialist Model

  • The models’ confidence distributions are examined on 50,000 ImageNet test images, grouped by dog versus non-dog category.The figures show each model’s confidence distribution and a scatter plot comparing their confidences.Source: §E.4 · pp. 24–25, Figure 8 · p. 24, Figure 9 · p. 24
  • The dog-specialist EfficientNet B0 can be highly confident on non-dog images, so relative confidence may erroneously route them to the specialist.The comparison is with MobileNet V2 trained on all ImageNet classes; EfficientNet B0 was trained only on the dog synset.Source: §E.4 · p. 24
  • For labeled images outside the dog synset, the specialist’s probability of the true label is low.The dog synset contains 119 classes.Source: §E.4 · p. 24

F Extension to Multi-Model Cascades

In a three-model cascade with label noise, post-hoc deferral significantly improves over confidence-based deferral. For K classifiers, the extension learns sequential deferral rules and selects classifiers by balancing error and inference cost.

F Extension to Multi-Model Cascades

  • For K classifiers, a simple recipe trains K−1 deferral rules, each predicting whether to defer from classifier k to k+1.At inference time, the rules are invoked sequentially to determine where prediction terminates.Source: F Extension to Multi-Model Cascades · p. 25
  • In the three-model label-noise setting, post-hoc deferral significantly improves over confidence-based deferral.The latter wastefully defers on noisy samples that no model can correctly predict; relative inference cost is measured against always querying the large model.Source: Figure 10 · p. 25, F Extension to Multi-Model Cascades · p. 25
  • The optimal selector for K>2 chooses the classifier with the best balance between misclassification error and inference cost.The selector’s objective combines standard misclassification accuracy under the chosen classifier with an inference-cost penalty.Source: F Extension to Multi-Model Cascades · pp. 25–26

G Limitations

The paper identifies three settings where confidence-based deferral can be sub-optimal, while noting that these settings are not exhaustive.

G Limitations

  • The identified failure settings—specialist models, label noise, and distribution shift—are not exhaustive; finite-sample cascade behavior remains an open topic.The authors identify other conditions for poor performance and finite-sample behavior of confidence-based and post-hoc rules as future work.Source: G Limitations · p. 26

Related papers

Loading 2307.02764v2…