Calibrated Learning to Defer with One-vs-All Classifiers

1. Introduction

• L2D compares expected human and model performance, deferring when the human is more likely to act correctly and supporting division of labor.The introduction frames human involvement as a near-term response to consequential, high-stakes deployments.Source: §1 · p. 1
• Calibration matters for trust: an expert-correctness forecast of 70% should correspond to about 70 correct outcomes per 100 cases.The paper motivates calibration as a way to help prevent AI systems from misleading physicians.Source: §1 · p. 1
• The paper finds Mozannar & Sontag’s (2020) consistent multiclass surrogate is not calibrated for expert correctness because its softmax parameterization can yield estimates above one.It proposes an alternative loss based on one-vs-all classifiers.Source: §1 · p. 1
• Across hate speech detection, galaxy classification, and skin-lesion diagnosis, the one-vs-all model performs comparably to or better than other L2D frameworks and common baselines.The comparisons include Mozannar & Sontag (2020), Okati et al. (2021), and confidence thresholds.Source: §1 · p. 2

2. Background: Multiclass Learning To Defer

• Mozannar & Sontag (2020) proposed the only known consistent surrogate for multiclass L2D, whose minimizers also minimize the 0–1 L2D loss.The background describes this surrogate as convex in its score functions.Source: §2 · p. 2
• Training examples pair features with labels and expert predictions, with the expert potentially using information unavailable to the model.The background assumes the expert may therefore outperform the Bayes-optimal classifier.Source: §2 · pp. 2–3
• The L2D model learns a classifier h and rejector r; r(x)=0 uses the classifier, while r(x)=1 defers to a human or backup system.The rejector acts as a meta-classifier for deciding which inputs are appropriate for the classifier.Source: §2 · p. 2
• The rejector combines classifier and expert losses, selecting which prediction incurs loss; under 0–1 loss, an indicator marks prediction-label mismatch.The formulation also applies to general losses.Source: §2 · p. 2
• Mozannar & Sontag (2020) construct a consistent surrogate by augmenting the label space with a rejection option and applying a softmax-style formulation.The parameterization uses K class functions and one rejection function.Source: §2 · p. 2
• At test time, the classifier selects the highest-scoring class and the system defers when the rejection score is at least as high as every class score.Using α<1 encourages more division of labor, but makes the surrogate inconsistent.Source: §2 · p. 2

• The study concerns expert-correctness calibration, not classifier calibration; the authors do not know a general post-hoc calibration procedure for L2D under this softmax parameterization.They attribute the difficulty to the softmax range problem and interdependence of its g functions.Source: §3 · pp. 3–4
• If p_⊥(x) > 1/2 for any input, then p_m(x) > 1 and cannot estimate expert correctness.The Bayes solutions are fragile: valid estimates require p_⊥(x) ≤ 1/2, although its range is (0, 1).Source: §3 · p. 3
• 39.4% of CIFAR-10 test samples had p_m(x) > 1, so the softmax estimator sometimes produced invalid probabilities.Source: Figure 1 · p. 4
• 7.58% ECE for clamped softmax estimates versus 3.01% for OvA; clamping also created a false zero-risk mode.The resulting risk estimates were biased toward zero error.Source: §3 · p. 4, Figure 1 · p. 4
• Using p_⊥(x) itself as a proxy for expert correctness also failed, with ECEs above 30%.Source: §3 · p. 4

4.1. One-vs-All-Based Surrogate Loss

• The OvA surrogate combines the same classifier and rejector functions as the softmax formulation using binary surrogate losses.For logistic loss, the binary loss is log(1 + exp{−f(x)}); the classifier and rejector are computed as in the softmax case.Source: §4.1 · p. 4
• OvA directly estimates expert correctness within (0, 1), but estimates only the most likely class probability, so distribution calibration cannot be evaluated.The authors identify this as a trade-off for obtaining an appropriate estimator of expert correctness.Source: §4.1 · p. 5

4.2. Theoretical Analysis

• The consistency proof uses error-correcting output codes to reduce the multiclass problem to multiple binary problems.The coding matrix assigns one binary problem to each column.Source: §4.2 · p. 5
• Under a strictly proper binary composite loss with a continuous inverse link, Theorem 4.1 establishes calibration; with minimizability, the OvA loss is consistent for 0-1 L2D.The minimizer over all measurable functions agrees with the Bayes-optimal classifier and rejector.Source: §4.2 · pp. 5–6

5. Related Work

• Rejection learning dates at least to Chow (1957), with later confidence-based and classifier-rejector approaches to abstention.The cited classifier-rejector work established theoretical guarantees for binary classification, while extending theory to multiclass settings proved difficult.Source: §5 · p. 6
• Madras et al. (2018) introduced L2D to model interactions between a human decision maker and an autonomous system; later work explored mixture-of-experts and confidence-based deferral.Raghu et al. (2019) and Wilder et al. (2020) proposed approaches to deciding when to defer.Source: §5 · p. 6
• Mozannar & Sontag (2020) proposed the first consistent multiclass L2D loss, while this paper is the first to study confidence calibration within L2D systems.The related work also notes that they found earlier algorithms by Madras et al. (2018) inconsistent.Source: §5 · p. 6

6.3. Overall Accuracy

• OvA is at least competitive with the best-performing competitor across hate speech detection, galaxy classification, and skin lesion diagnosis, without sacrificing predictive performance.The comparison includes Mozannar & Sontag’s method and other state-of-the-art methods.Source: §6 · p. 6, Figure 4 · p. 8, §6.3 · p. 9

6. Experiments

• The experiments use logistic loss for OvA and set softmax’s re-weighting parameter α to 1, the value for which its surrogate is provably consistent.Results are averaged across six random seeds.Source: §6 · p. 6

6.1. Comparison to the Softmax Loss on CIFAR-10

• ECE was 7.58% for softmax and 3.01% for OvA in the CIFAR-10 simulation, an over 50% reduction for OvA.OvA also produces sharper risk-estimate modes nearer true error values and avoids softmax’s false mode at zero.Source: §6.1 · p. 7
• OvA had the best ECE in all but one tested case; the p⊥ proxy’s ECE reached 37.15% when both data and expert were useful and 39.06% when data were random and the expert useful.The exception for OvA was the case where both expert and data were random.Source: §6.1 · p. 7
• On CIFAR-10, OvA had a modest but consistent accuracy advantage over softmax as expert expertise varied, and outperformed it at most coverage levels.The reported advantage was significant at coverage levels 0.2–0.3, where the rejector must select instances for classification carefully.Source: §6.1 · pp. 7–8

6.2. Risk Assessment on HAM10000

• On HAM10000, the Wasserstein distance between predicted and true expert error was 8.02 ± 1.37 for OvA and 26.72 ± 1.77 for softmax.The experiments restrict the softmax estimate to (0, 1], while OvA substantially reduces the gap between predicted and true error.Source: Figure 3 · p. 7, §6.2 · p. 8

7. Conclusions

• The OvA loss is consistent and experimentally yields better calibration than the softmax-based surrogate, while the paper identifies calibration in non-surrogate L2D systems as future work.The softmax parameterization can produce degenerate estimates of expert correctness.Source: §7 · p. 9

A.1. A General Classification Problem and Surrogate Losses

• A surrogate loss replaces a target loss when directly minimizing the target risk is computationally difficult, as with 0–1 misclassification loss.The appendix describes calibration as a necessary condition for surrogate consistency.Source: §A.1 · p. 12

A.2. Calibration of Binary Surrogate Losses

• For convex binary surrogate losses, classification calibration is equivalent to differentiability in the second argument at zero with a negative derivative there.The stated result is attributed to Bartlett et al. (2006).Source: §A.2 · p. 13

A.3. Binary Proper Losses and Proper Composite Surrogate Losses

• Minimizing a strictly proper composite loss yields Fisher-consistent class-probability estimates and classification-calibrated predictions.The appendix gives logistic loss as an example.Source: §A.3 · p. 13

A.4. Code Based Surrogates for Multiclass Classification

• Code-based multiclass methods use a code matrix to decompose classification into binary problems and decode their outputs into the original label space.The appendix describes each binary classifier as trained on a code-defined subset of the original sample.Source: §A.4 · p. 14

B. One-vs-All surrogate Loss for L2D

• The OvA L2D surrogate is formed from binary surrogate-loss terms for the predicted class and the defer option, with the binary loss applied to signed scores.The appendix derives separate cases depending on whether the expert prediction matches the label.Source: B. One-vs-All surrogate Loss for L2D · p. 14

C. Proofs

• The derivation starts from the Bayes-optimal rejection output and rewrites it as a function of the probability that the expert is correct.The passage gives the expression p⊥(x) = P(m = y|x) / (1 + P(m = y|x)).Source: §C.1 · p. 15
• Rearranging the rejection-output expression yields an estimate of the expert-correctness probability.The derivation then turns to the Bayes-optimal class outputs.Source: §C.1 · p. 15
• The proof also relates each Bayes-optimal class output to the conditional probability of its class and the expert-correctness probability.Source: §C.1 · p. 15

C.2. Proof of Theorem 4.1

• The OvA construction decomposes the problem into K + 1 binary risks, with separate terms for each class and the expert-correctness probability.The minimizer for each OvA component corresponds to the minimizer of its binary inner risk.Source: §C.2 · pp. 15–16
• When the binary surrogate is strictly proper composite, its inverse link recovers the expert-correctness probability from the defer score.The proof also uses the binary surrogate minimizers to recover class decisions.Source: §C.2 · p. 16
• The point-wise surrogate-risk minimizer yields the Bayes-optimal classifier and rejection prediction, establishing calibration for 0–1 L2D.The proof considers cases based on the signs of the class and defer scores.Source: §C.2 · pp. 16–17

D.1. ECE values with respect to the classifier correctness

• Table 2 compares classifier calibration on the CIFAR-10 simulation for OvA and softmax parameterizations.The supplied passage does not report the table’s ECE values or which parameterization performs better.Source: Table 2 · p. 17

D.2. Effect of Calibration on System’s Accuracy

• Post-processing calibration of the One-vs-All rejector marginally improves system accuracy.The section uses temperature scaling to further calibrate the rejector and reports that calibration positively correlates with accuracy.Source: §D.2 · p. 17, Figure 5 · p. 18

D.3. Class-wise performance of the simulated MLPMixer expert for HAM10000

• The simulated HAM10000 MLPMixer expert has non-uniform performance across classes, including expertise on class nv.The passage describes this class-varying performance as a valid simulation of a real-world expert who may specialize in some classes.Source: Table 3 · p. 18

E. Additional Information about the Methods

• To compare differentiable triage fairly with other baselines, the authors use the triage-level methodology from Okati et al. (2021).The differentiable-triage algorithm incorporates the budget during training, unlike the other baselines.Source: E. Additional Information about the… · p. 18
• The softmax surrogate ranks samples by the gap between the largest class probability and rejection probability, while the One-Vs-All surrogate uses the same procedure.For a budget b, the softmax method defers up to the smaller of the budgeted sample count and the number of cases where rejection probability is at least the largest class probability.Source: E. Additional Information about the… · p. 18
• The score baseline defers cases with the lowest maximum class probability and temperature-scales its classifier using validation data.Its performance depends on the classifier’s uncertainty estimates.Source: E. Additional Information about the… · p. 18
• The confidence baseline estimates expert correctness without conditioning on the input, then trains using selected samples within each mini-batch.At test time, it sorts samples by the model’s maximum class probability and defers within the budget, subject to the stated count condition.Source: E. Additional Information about the… · pp. 18–19
• Differentiable triage first estimates the predictive model, then approximates a budget-specific policy that defers when expert loss is lower.Unlike surrogate-loss methods using expert predictions, its training assumes access to expert predictive loss; the experiments use negative log-likelihood.Source: E. Additional Information about the… · p. 19

F. Additional Experimental Details

• CIFAR-10 experiments use 28-layer Wide Residual Networks without data augmentation, with the same settings for both surrogate losses.Training uses SGD, momentum 0.9, weight decay 5e-4, a 0.1 initial learning rate, cosine annealing, early stopping, and batch size 1024.Source: F. Additional Experimental Details · p. 19
• HAM10000 experiments simulate the expert with an 8-layer MLPMixer and fine-tune ResNet34 for the main model.The simulated expert is trained without extensive hyperparameter search; the passage also specifies image resizing and optimization settings.Source: F. Additional Experimental Details · p. 19
• The authors use code made available by the respective authors for their other baselines.Source: F. Additional Experimental Details · p. 19