PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Optimal Conformal Classifiers

David Stutz, Krishnamurthy, Dvijotham, Ali Taylan Cemgil, Arnaud Doucet

arXiv:2110.09192v3cs.LGcs.CVstat.MEstat.ML

TL;DR

Post-training conformal prediction leaves the classifier untuned to the confidence sets it will produce, despite CP’s coverage guarantee. ConfTr simulates calibration and prediction during mini-batch training so the model can optimize those sets. Experiments report lower inefficiency and greater control over set contents while retaining CP coverage guarantees.

  • Problem

    Post-training CP does not tune the classifier toward desired confidence-set properties, limiting control over inefficiency and set composition despite its coverage guarantee.

    Source: §1 · pp. 1–2

  • Method

    ConfTr differentiates through simulated conformalization on mini-batches, calibrating on one portion and computing confidence-set losses on another.

    Source: §1 · p. 2

  • Results

    ConfTr consistently reduces the inefficiency of post-training THR or APS and can shape class-conditional inefficiency or confidence-set composition.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    ConfTr can impose application-specific constraints on confidence sets while preserving the coverage guarantee of the CP method used at test time.

    Source: §1 · p. 2, ETHICS STATEMENT · p. 10

    Experiments use standard benchmark datasets; the work contains no experiments with personal or sensitive medical data.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Modern deep learning based classifiers show very high accuracy on test data but this does not provide sufficient guarantees for safe deployment, especially in high-stake AI applications such as medical diagnosis. Usually, predictions are obtained without a reliable uncertainty estimate or a formal guarantee. Conformal prediction (CP) addresses these issues by using the classifier's predictions, e.g., its probability estimates, to predict confidence sets containing the true class with a user-specified probability. However, using CP as a separate processing step after training prevents the underlying model from adapting to the prediction of confidence sets. Thus, this paper explores strategies to differentiate through CP during training with the goal of training model with the conformal wrapper end-to-end. In our approach, conformal training (ConfTr), we specifically "simulate" conformalization on mini-batches during training. Compared to standard training, ConfTr reduces the average confidence set size (inefficiency) of state-of-the-art CP methods applied after training. Moreover, it allows to "shape" the confidence sets predicted at test time, which is difficult for standard CP. On experiments with several datasets, we show ConfTr can influence how inefficiency is distributed across classes, or guide the composition of confidence sets in terms of the included classes, while retaining the guarantees offered by CP.

1 INTRODUCTION

Conformal prediction (CP) adds confidence sets with a coverage guarantee, but post-training conformalization leaves the classifier untuned to those sets. ConfTr trains the model against a simulated conformal wrapper, improving inefficiency and enabling finer control over set contents while preserving CP coverage guarantees.

1 INTRODUCTION

  • CP guarantees that the true class is included with probability at least 1 − α when calibration examples are exchangeable with test data, but this is marginal rather than conditional coverage.The guarantee averages over possible calibration sets as well as test examples.Source: §1 · p. 1
  • Post-training CP does not tune model parameters toward objectives such as reducing expected confidence set size or controlling which classes are included.The paper contrasts this with prior work that omits calibration during training and supports fewer loss objectives.Source: §1 · p. 2
  • ConfTr simulates conformalization during training, splitting each mini-batch between calibration and prediction so losses can optimize the resulting confidence sets.After training, any existing CP method can provide a coverage guarantee.Source: §1 · p. 2
  • ConfTr consistently reduces the inefficiency of post-training THR and APS and can shape class-conditional inefficiency or confidence-set composition.The paper also reports that ConfTr preserves CP’s coverage guarantee.Source: §1 · p. 2
  • Baseline CP methods can differ in inefficiency: on CIFAR, THR outperforms the other discussed approaches, while harder classes have higher inefficiency.The figure reports coverage close to 1 − α and plots class-wise inefficiency at α = 0.01.Source: Figure 2 · p. 3

2 DIFFERENTIABLE CONFORMAL PREDICTORS

The section reviews conformal predictors and their evaluation, then develops differentiable prediction and calibration steps for training through CP. ConfTr uses those steps within mini-batches, while the original non-smooth CP implementation is used at test time for its coverage guarantee.

2.1 CONFORMAL PREDICTORS

  • THR forms confidence sets by thresholding class probabilities and calibrates the threshold from true-class conformity scores.THR can also use logits or log-probabilities; its marginal coverage guarantee is 1 − α.Source: §2.1 · p. 3
  • APS orders class probabilities to construct sets and uses a randomized conformity score, with calibration based on its true-class scores.The random variable U breaks ties, and the stated quantile yields marginal coverage.Source: §2.1 · p. 3
  • The section evaluates CP by coverage and inefficiency, focusing comparisons on inefficiency because empirical coverage across calibration/test splits is close to 1 − α.For the baseline methods shown, THR has lower average inefficiency than THRL and APS.Source: §2.1 · p. 3
  • Fine-grained control over class-conditional inefficiency or confidence-set composition is generally unavailable with CP alone, motivating differentiable implementations for training.The section identifies THR and APS as methods requiring differentiable prediction and calibration steps.Source: §2.1 · p. 3, §2.2 · p. 4

2 DIFFERENTIABLE CONFORMAL PREDICTORS

  • ConfTr performs differentiable CP within each mini-batch, calibrating on one half and predicting and computing losses on the other.The method uses smooth calibration and prediction steps to update model parameters.Source: §2.1 · p. 4

2.2 DIFFERENTIABLE PREDICTION AND CALIBRATION STEPS

  • Prediction is smoothed with a temperature-scaled sigmoid, while calibration uses a differentiable quantile computed with smooth sorting.As temperature and dispersion approach zero, the original non-smooth computations are recovered; at test time, the original implementations provide the coverage guarantee.Source: §2.2 · p. 4

3 CONFORMAL TRAINING (CONFTR): Learning CONFORMAL PREDICTION

ConfTr simulates conformal prediction during training so the model can adapt to confidence-set objectives. Its losses support minimizing set size and controlling set composition while retaining conformal coverage guarantees.

3 CONFORMAL TRAINING (CONFTR): Learning CONFORMAL PREDICTION

  • ConfTr simulates calibration and prediction on each mini-batch, then applies standard conformal predictors after training.It generalizes an approach that differentiates only through prediction with a fixed threshold by including calibration in training.Source: §3 · p. 4

3.1 CONFTR BY OPTIMIZING INEFFICIENCY

  • ConfTr calibrates on one half of each mini-batch and minimizes confidence-set inefficiency on predictions from the other half.The calibrated threshold is recomputed on a held-out calibration set after training to obtain a conformal coverage guarantee.Source: §3.1 · p. 5

3.2 CONFTR WITH CLASSIFICATION LOSS

  • A configurable classification loss can encourage inclusion of the true label and optionally penalize other classes in confidence sets.The loss matrix controls penalties for including class k when the ground truth is y.Source: §3.2 · p. 5

3.3 CONFTR WITH GENERAL AND APPLICATION-SPECIFIC LOSSES

  • Class- or group-weighted size losses can reallocate inefficiency, such as reducing uncertainty for low-risk conditions at the expense of high-risk ones.The weight depends on the ground-truth class.Source: §3.3 · p. 6
  • Positive penalties for selected true-class/class pairs can reduce their co-occurrence in confidence sets, addressing class-pair confusion.The off-diagonal coverage-confusion entries quantify how often class k appears when the true class is y.Source: §3.3 · p. 6
  • For disjoint class groups, pair-specific penalties can reduce inclusion of K1 classes in confidence sets when the true class belongs to K0.The paper frames this as controlling unwanted overlap between groups, such as high-risk conditions appearing with low-risk ones.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments compare conformal training (ConfTr) with standard cross-entropy training and prior work, then examine how ConfTr can shape confidence sets while maintaining coverage guarantees. Across benchmarks, ConfTr consistently improves inefficiency, while targeted changes to class-level behavior can involve trade-offs elsewhere.

4.1 REDUCING INEFFICIENCY WITH CONFTR

  • ConfTr consistently reduces inefficiency for both THR and APS versus standard training, while Bellotti’s method does not improve it consistently across datasets.ConfTr uses THRLP during training; improvements are less pronounced on CIFAR, where models use pre-trained ResNet features.Source: Table 1 · p. 7, §4.1 · p. 7
  • Using THRLP for training regardless of the test-time CP method avoids difficult THR/APS training, but ConfTr may reduce top-1 accuracy by 2–6% when trained from scratch.THR and APS use probabilities, which the authors say limits gradient flow; THRLP uses log-probabilities. Inefficiency can improve despite lower accuracy.Source: §4.1 · p. 7
  • ConfTr can further reduce ensemble inefficiency from 2.40 to 2.35, and its gains generalize to lower EMNIST confidence levels after training at α=0.01.The reported best individual ensemble model has inefficiency 3.10; at α=0.001, ConfTr’s improvement is more significant than at α=0.01.Source: §4.1 · pp. 7–8

4.2 CONFORMAL TRAINING FOR APPLICATIONS: CASE STUDIES

  • Class- or group-conditional inefficiency can fall by 20% or more, including for CIFAR10 cats and CIFAR100 coarse classes, usually with a slight increase in overall average inefficiency.The figure reports reductions above 30% for some CIFAR100 coarse classes.Source: §4.2 · p. 8, Figure 3 · p. 8
  • On Fashion-MNIST, penalizing selected class pairs reduces their coverage confusion by roughly 1% or 0.5% each, but confusion for other pairs can increase.For coat and shirt, the two confusion directions decrease; penalizing a full row of the matrix also redistributes confusion across pairs.Source: Figure 4 · p. 8, §4.2 · p. 9
  • On CIFAR10, ConfTr reduces cat-group mis-coverage from 98.92% to 91.6% and reverse mis-coverage from 36.52% to 26.43%; CIFAR100 group overlap also falls, with a slight inefficiency increase.For CIFAR100, human-made-to-natural mis-coverage falls from 40.09% to 15.77%.Source: §4.2 · p. 9, Figure 5 · p. 9
  • On WineQuality, ConfTr can reduce inefficiency for minority class 0 (“bad wine”) at the expense of class 1, while improving class-conditional coverage for class 0.Class 0 comprises about 37% of examples; the paper also reports that coverage confusion between the two classes can be manipulated.Source: §4.2 · p. 9, Figure 5 · p. 9

5 CONCLUSION

The conclusion presents ConfTr as end-to-end training for conformal predictors that improves inefficiency and enables targeted control of confidence-set properties. These changes retain CP’s marginal coverage guarantee.

5 CONCLUSION

  • ConfTr improves inefficiency for state-of-the-art CP methods including THR and APS, while retaining CP’s marginal coverage guarantee.The conclusion also emphasizes shaping class-conditional inefficiency, coverage confusion, and overlap between class groups.Source: §5 · p. 9
  • ConfTr can reduce inefficiency for selected classes at the cost of higher inefficiency for others, and can control which non-ground-truth classes enter confidence sets.The conclusion also describes extending this control to reduce overlap between groups of classes.Source: §5 · p. 9
  • The paper motivates confidence-set control for medical diagnosis, while describing its results as demonstrations of ConfTr’s possible uses.The conclusion frames the method as relevant where constraints on confidence sets matter alongside CP guarantees and uncertainty estimates.Source: §5 · p. 9

ETHICS STATEMENT

The ethics statement describes conformal prediction as a source of user-specified coverage guarantees and uncertainty estimates, with potential relevance to medical diagnosis. It also notes that the paper evaluates only standard benchmark datasets, not personal or sensitive medical data.

ETHICS STATEMENT

  • The paper expects CP’s performance guarantees and uncertainty estimates could support trustworthiness among doctors and patients, particularly in medical applications.This is presented as an expectation rather than a tested result.Source: ETHICS STATEMENT · p. 10
  • The experiments use standard benchmark datasets and do not include personal or sensitive medical data.Thus, the stated medical relevance is not validated on medical data in this work.Source: ETHICS STATEMENT · p. 10
  • The authors identify settings beyond medical diagnosis where additional confidence-set constraints may matter alongside CP guarantees and uncertainty estimates.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points to detailed setup, evaluation, and implementation information in the appendix. The appendix also covers related work, coverage guarantees, and a differentiable APS implementation.

REPRODUCIBILITY STATEMENT

  • The appendix documents architectures, training procedures, hyperparameters, preprocessing, data augmentation, evaluation splits, and code for key components.The evaluation uses multiple calibration/test splits and training runs to capture randomness.Source: REPRODUCIBILITY STATEMENT · p. 10

A OVERVIEW AND OUTLINE

  • The appendix collects a CoverTr baseline, complementary experiments, hyperparameter and method ablations, formal coverage material, and implementation code.It includes further inefficiency, class-conditional, coverage-confusion, and mis-coverage results.Source: A OVERVIEW AND OUTLINE · p. 14

B RELATED WORK

  • ConfTr follows split CP for classification and is complementary to test-time CP methods; extensions to regression or other CP formulations are described as possible.The related-work discussion contrasts its coverage-guaranteed sets with multiple-choice prediction sets, which have fixed size and no coverage guarantee.Source: B RELATED WORK · p. 14

C COVERAGE GUARANTEE

  • The stated CP guarantee assumes permutation-invariant learning and exchangeable calibration examples drawn from the test distribution.The discussion notes that conditional coverage generally requires additional assumptions, while THR can provide class-conditional coverage.Source: C COVERAGE GUARANTEE · p. 15

D DIFFERENTIABLE APS

  • The differentiable APS conformity score sums sorted class probabilities up to the candidate class, with a uniform random variable breaking ties.The implementation uses a smooth sorting approach and could be extended with the regularizer discussed by Angelopoulos et al. (2021).Source: D DIFFERENTIABLE APS · p. 15

E COVERAGE TRAINING

CoverTr differentiates through confidence-set prediction at a fixed threshold, rather than through calibration, and requires both classification and size losses. Choosing that threshold can be difficult, while ConfTr instead calibrates on each mini-batch and differentiates through calibration.

E COVERAGE TRAINING

  • CoverTr computes confidence sets on each mini-batch at a fixed threshold τ and differentiates through prediction without differentiating through calibration.Unlike ConfTr, it does not split the batch for calibration and prediction.Source: E COVERAGE TRAINING · p. 15
  • Both classification and size losses are necessary: alone, they favor confidence sets containing every class or no classes, respectively.The weight λ balances the two losses, and τ is recalibrated at test time to obtain a coverage guarantee.Source: E COVERAGE TRAINING · p. 15
  • Choosing τ for training is difficult because reasonable values depend on the dataset and model and may evolve as the model learns.ConfTr addresses this by calibrating on each mini-batch and differentiating through calibration, avoiding a larger held-out calibration set during training.Source: E COVERAGE TRAINING · p. 16
  • Bellotti (2021) uses a squared coverage loss targeting level 1−α on each mini-batch, rather than Lclass.The passage describes this approach as a special case of CoverTr using THRL during training.Source: E COVERAGE TRAINING · p. 16
  • CoverTr with THR or APS is difficult to tune, and Bellotti (2021) is restricted to THRL and is consistently outperformed by ConfTr on inefficiency improvements.The reported comparison also says Bel cannot be used for the studied use cases in Sec. 3.3.Source: E COVERAGE TRAINING · p. 16

F EXPERIMENTAL SETUP

The experiments span image and non-image datasets, several model families, and repeated training and test trials. Results show that ConfTr outperforms the THR baseline on both MNIST and Fashion-MNIST, while small batches, higher confidence levels, and some CoverTr configurations present difficulties.

F EXPERIMENTAL SETUP

  • Experiments cover Camelyon2016, GermanCredit, WineQuality, MNIST, EMNIST, Fashion-MNIST, and CIFAR, using fixed training, calibration, and test splits.Models include linear classifiers, MLPs, and ResNets; CIFAR models are also fine-tuned or extended on pretrained features.Source: F EXPERIMENTAL SETUP · pp. 16–17
  • ConfTr hyperparameters were selected by dataset-specific grid search, and results were evaluated across random calibration/test splits and training trials.The study uses 10 random test splits and 10 randomly sampled training sets for reported averages.Source: F EXPERIMENTAL SETUP · pp. 17–19

G IMPORTANCE OF RANDOM TRIALS

  • Test-trial inefficiency varies more than coverage, while training-trial variation makes ConfTr with APS or THR less stable and can worsen inefficiency.Evaluation averages across 10 calibration/test splits and 10 training trials; the table reports a 3.1 standard deviation for training-time APS.Source: Table C · p. 18, G IMPORTANCE OF RANDOM TRIALS · p. 19

H IMPACT OF HYPER-PARAMETERS

  • On MNIST, batch size 100 prevented ConfTr from converging, and the chosen hyperparameters did not generalize well to α=0.1 or 0.05.The authors suggest α=0.01 may be too low when only 50 examples are available for calibration in a batch of 100.Source: H IMPACT OF HYPER-PARAMETERS · p. 19, Table D · p. 19
  • For MNIST hyperparameters, λ and temperature had insignificant impact without Lclass, while λ mattered more for balancing losses with Lclass.The selected hyperparameters do not generalize to higher confidence levels α∈{0.1, 0.05}.Source: H IMPACT OF HYPER-PARAMETERS · p. 19, Table D · p. 19

I COVERTR AND CONFTR ABLATION ON MNIST AND FASHION-MNIST

J ALL INEFFICIENCY RESULTS

ConfTr consistently improves THR and APS inefficiency on the reported multiclass results, though improvements are smaller on CIFAR and not significant on most binary datasets. Standard ConfTr and the identity loss matrix reduce overall inefficiency without changing its class distribution or coverage-confusion pattern.

J ALL INEFFICIENCY RESULTS

  • ConfTr consistently improves inefficiency for THR and APS on the reported multiclass results, including EMNIST.The CIFAR results show smaller gains, which the authors attribute as likely to features already trained to optimize cross-entropy.Source: J ALL INEFFICIENCY RESULTS · pp. 20–21

K EFFECT OF CONFTR ON CLASS-CONDITIONAL INEFFICIENCY AND COVERAGE CONFUSION

  • Standard ConfTr and ConfTr with Lclass using the identity loss matrix reduce overall inefficiency but leave class-conditional inefficiency and coverage-confusion patterns roughly unchanged.The paper therefore uses standard ConfTr for reducing class-conditional inefficiency and Lclass for changing coverage confusion or improving MisCover.Source: K EFFECT OF CONFTR ON… · p. 21

J ALL INEFFICIENCY RESULTS

  • On binary datasets, THRL, THR, and APS perform similarly, and ConfTr generally does not improve significantly over baseline.On Camelyon, the reported result uses the best model rather than an average over training trials because only 280 training examples are available.Source: J ALL INEFFICIENCY RESULTS · p. 20, K EFFECT OF CONFTR ON… · p. 22

L SHAPING CLASS-CONDITIONAL INEFFICIENCY ON OTHER DATASETS

ConfTr can target class- or group-conditional inefficiency, including classes with above- or below-average inefficiency. These reductions can involve a slight increase in average inefficiency, while targeted coverage confusion can also be reduced.

L SHAPING CLASS-CONDITIONAL INEFFICIENCY ON OTHER DATASETS

  • ConfTr can reduce inefficiency for targeted classes or groups whether their initial inefficiency is above or below average.Examples include individual CIFAR10 classes and coarse CIFAR100 classes.Source: L SHAPING CLASS-CONDITIONAL… · p. 22
  • Targeting coverage-confusion cells or a full row can significantly reduce coverage confusion on Fashion-MNIST and CIFAR10.The examples target coat/shirt and cat/dog cells, and show row 6 for Fashion-MNIST and row 3 for CIFAR10.Source: Figure D · p. 23
  • Per-class or group inefficiency reductions are possible across MNIST, Fashion-MNIST, and CIFAR100, sometimes at the cost of a slight increase in average inefficiency.On CIFAR100, reductions of 40% or more are possible; the authors suggest the larger number of classes may provide more flexibility for trade-offs.Source: L SHAPING CLASS-CONDITIONAL… · pp. 22–23

M MANIPULATING COVERAGE CONFUSION ON OTHER DATASETS

Additional-dataset experiments show coverage-confusion and mis-coverage trade-offs vary by dataset and targeted class pair; reductions are not uniformly achievable.

M MANIPULATING COVERAGE CONFUSION ON OTHER DATASETS

  • On MNIST, coverage-confusion reductions are usually larger and roughly linear in L_y,k; on CIFAR10 they can plateau early or fail for some targeted pairs.Reducing confusion for some class pairs can slightly increase it for others.Source: M MANIPULATING COVERAGE CONFUSION ON… · p. 23

N MISCOVER RESULTS ON ADDITIONAL DATASETS

M MANIPULATING COVERAGE CONFUSION ON OTHER DATASETS

  • On MNIST even-vs-odd, L_K0,K1=1 versus ConfTr lowers MisCover 0→1 from 38.84 to 29.36 while raising MisCover 1→0 from 38.69 to 49.08.For Fashion-MNIST shirt-vs-others, the same setting changes MisCover 0→1 from 80.28 to 72.58 and MisCover 1→0 from 20.93 to 25.81.Source: M MANIPULATING COVERAGE CONFUSION ON… · p. 24

O ADDITIONAL RESULTS ON BINARY DATASETS

Binary-dataset experiments show class-specific set behavior can be adjusted unevenly, while the accompanying pseudocode outlines the differentiable THR training path.

O ADDITIONAL RESULTS ON BINARY DATASETS

  • WineQuality permits inefficiency reduction for class 0 (“bad wine”) but not class 1 (“good wine”); class-0 coverage improves and coverage confusion Σ0,1 falls.The class-1 majority comprises approximately 63% of examples; the results generalize less strongly to GermanCredit, which has fewer training and calibration examples.Source: O ADDITIONAL RESULTS ON BINARY DATASETS · p. 24

P PSEUDO CODE

  • The example combines classification and size losses through simulated CP; its trainable and fixed parameters also allow fine-tuning a pre-trained model.The implementation assumes a differentiable quantile routine.Source: P PSEUDO CODE · pp. 25, 27

O ADDITIONAL RESULTS ON BINARY DATASETS

  • On MNIST and Fashion-MNIST, lower average inefficiency accompanies lower class-conditional inefficiency, but its distribution across classes remains similar; CIFAR10 changes little.The coverage-confusion pattern likewise remains roughly the same on MNIST and Fashion-MNIST, and mostly unchanged on CIFAR10.Source: P PSEUDO CODE · p. 26

P PSEUDO CODE

  • The pseudocode splits each mini-batch, calibrates a smooth THR threshold on one half, then predicts confidence sets and computes loss on the other.It implements this flow using smooth calibration and prediction steps for THR.Source: P PSEUDO CODE · p. 27
Loading 2110.09192v3…