Source-linked AI summary
Uncertainty Sets for Image Classifiers using Conformal Prediction
Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, Michael I. Jordan
TL;DR
Reliable uncertainty quantification for image classifiers remains difficult because calibration methods lack formal guarantees, limiting their use in consequential settings. The paper modifies conformal prediction to construct calibrated predictive sets for any classifier, regularizing unlikely-class scores. On ImageNet and ImageNet-V2, RAPS achieves coverage with substantially smaller sets than existing approaches, often by factors of 5 to 10 versus Platt scaling.
Problem
Image classifiers need actionable uncertainty estimates for consequential applications, but probability-calibration methods such as Platt scaling do not provide formal coverage guarantees.
Method
The paper uses split conformal calibration and regularizes small scores of unlikely classes to construct prediction sets from arbitrary classifiers.
Results
RAPS achieves target coverage with smaller sets than naive and APS; for a ResNeXt-101 at α = 10%, APS averages 19 labels while RAPS averages 2.
Takeaways & Limitations
The method provides a simple, computationally lightweight way to quantify image-classifier uncertainty while retaining small average prediction sets.
Takeaways & Limitations
The coverage guarantee is marginal over X and Y, so it holds on average rather than for every particular image.
Abstract
from arXiv · showhide
Convolutional image classifiers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings. Existing uncertainty quantification techniques, such as Platt scaling, attempt to calibrate the network's probability estimates, but they do not have formal guarantees. We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%. The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset. Our method modifies an existing conformal prediction algorithm to give more stable predictive sets by regularizing the small scores of unlikely classes after Platt scaling. In experiments on both Imagenet and Imagenet-V2 with ResNet-152 and other classifiers, our scheme outperforms existing approaches, achieving coverage with sets that are often factors of 5 to 10 smaller than a stand-alone Platt scaling baseline.
1 INTRODUCTION
The paper develops conformal prediction sets for image classifiers that provide user-specified marginal coverage, addressing the limitations of probability calibration in consequential applications. Regularized Adaptive Prediction Sets improve efficiency by producing smaller sets while retaining coverage guarantees.
- Motivation: Prediction sets supplement the most likely class with a set that is guaranteed to contain the true diagnosis with high probability.The motivation centers on actionable uncertainty for high-stakes settings such as medical diagnosis.
- Contribution: Conformal predictors can modify any black-box classifier to produce predictive sets satisfying a pre-specified coverage level.The evaluation focuses on ImageNet classification with CNN base classifiers.
- Limitations of existing approach: Naive probability-threshold sets lack coverage because CNN probabilities are inaccurate and can become large because tail probabilities are miscalibrated.These limitations motivate holdout-based calibration and regularization.
- Proposed methods: APS uses holdout samples to select a threshold that achieves target coverage, while RAPS regularizes noisy tail estimates to produce smaller, more stable sets.Both methods retain formal coverage guarantees and require negligible computation.
- Results: At α = 10%, APS sets average 19 labels while RAPS sets average 2 for a ResNeXt-101, with both achieving coverage almost exactly.Naive does not achieve coverage in this experiment.
- Evaluation context: The evaluation covers ImageNet and ImageNet-V2 and addresses limited prior empirical assessment of conformal methods on state-of-the-art CNNs.The paper positions its evaluation as extensive relative to earlier work.
2 METHODS
The method uses conformal calibration to select a threshold that guarantees target coverage, then regularizes class scores to produce smaller, more stable, and adaptive prediction sets. RAPS retains finite-sample coverage while improving set size relative to naive, APS, and fixed-size procedures.
- Design desiderata: RAPS balances coverage, small set size, and adaptiveness, although coverage and the other two objectives can be in tension.Adaptive sets should be smaller for easy examples and larger when the classifier is uncertain.
- Conformal calibration: Conformal calibration selects the smallest threshold achieving at least 1 −α coverage on an independent calibration set, with a finite-sample correction.The resulting set is guaranteed to have coverage 1 −α for a fresh test observation from the same distribution.
- Conformal calibration: The coverage guarantee is marginal over X and Y, so it holds on average rather than for each particular image.The theorem assumes i.i.d. calibration and test observations, with exchangeability also sufficient as a technical extension.
- RAPS: RAPS orders classes by predicted probability, adds a rank-based regularization penalty, and conformally calibrates the resulting penalized sets.The penalty uses λ and kreg to make lower-ranked classes require larger thresholds, promoting smaller sets.
- Optimality considerations: RAPS with suitable regularization dominates top-k sets by achieving the desired coverage with average set size no larger than the best fixed top-k procedure.Naive and APS do not share this guarantee and usually produce sets larger than k*.
3 EXPERIMENTS
Experiments on ImageNet and ImageNet-V2 compare naive, APS, and RAPS prediction sets across pretrained classifiers, evaluating coverage, size, and adaptiveness. RAPS maintains coverage while generally producing smaller, more stable sets, including under distribution shift when calibration data matches the new distribution.
- Experimental setup: Experiments compare naive, APS, and RAPS on ImageNet and ImageNet-V2 using pretrained classifiers and conformal calibration.The study uses nine standard pretrained ImageNet classifiers and reports repeated-trial coverage and mean set size.
- Coverage and set size: RAPS has smaller average sets than naive and APS while achieving coverage across the evaluated ImageNet experiments.The reported evaluations use two coverage levels and repeated random calibration and evaluation subsets.
- ImageNet-V2: On ImageNet-V2, conformal calibration from the new distribution preserves coverage for models trained on different distributions, though coverage variance is higher with 5K calibration and evaluation sets.The ImageNet-V2 experiment uses the same preprocessing as Experiment 1 and smaller calibration and evaluation sets.
- Regularization and adaptiveness: RAPS regularization controls set-size behavior: λ = 1 clips sets at a maximum size of 5, while larger λ can under-cover hard examples and increase sets for easy examples.This illustrates a tradeoff between smaller average sets and adaptiveness across image difficulty.
- Parameter choice: RAPS performance depends on tuning kreg and λ, although the experiments select them adaptively from data and report strong results across models and coverage levels.Table 3 evaluates multiple parameter choices for ResNet-152 at 90% coverage.
4 ADAPTIVENESS AND CONDITIONAL COVERAGE
The paper evaluates adaptiveness through coverage conditional on prediction-set size rather than exact conditional coverage for each input. It finds that light RAPS regularization improves size-stratified behavior over APS, while conditional coverage remains an imperfect target for high-signal image classification.
- Conditional coverage: Exact conditional coverage requires 1 − α coverage for every x, but distribution-free guarantees for this property are impossible.The paper therefore considers approximate or alternative notions of adaptiveness.
- Size-stratified coverage: Coverage conditional on set size provides a practical adaptiveness criterion because it tests whether coverage remains near 1 − α across prediction-set-size strata.The size-stratified coverage violation measures the worst deviation across strata and can be computed by post-stratifying labeled examples.
- APS behavior: APS can over-cover large sets at 97% when the nominal marginal coverage rate is 90%, indicating imbalanced coverage across set sizes.The paper attributes this behavior to APS over-covering hard examples and under-covering easy ones.
- RAPS behavior: RAPS with λ between .001 and .01 produces more balanced coverage across set sizes than APS, making light regularization preferable for adaptiveness.The paper notes that less regularization may be needed as the training dataset grows and probability estimates become more stable.
- Adaptive tuning: Automatically tuned RAPS always outperforms APS on test-set adaptiveness under the reported coarse size partition and median violation criterion.The tuning searches λ values on a tuning set and reports median size-stratified coverage violation over 10 independent trials.
- Alternative criterion: The paper argues that conditional coverage is a poor adaptiveness goal when the best possible model has high accuracy, motivating coverage guarantees conditioned on set-size events instead.For high-signal problems, the proposed relaxation requires coverage for any collection of prediction-set sizes.
5 DISCUSSION
The method produces predictive sets with guaranteed error control and small average size, supporting uncertainty quantification and several downstream uses in classification.
- Predictive sets guarantee a pre-specified error level while retaining small average size for classifiers with many possible labels.The method is designed as a simple wrapper for arbitrary base classifiers.
- These sets can identify hard test-time examples for active learning when only a small number of points can be labeled.
- A cascade can invoke an expensive classifier only when a cheap classifier returns a large prediction set.
- Predictive sets can also support model development by exposing failure cases and outliers for performance improvement.
- The authors envision screening many diseases from blood samples and referring patients to relevant specialists.
A PROOFS
The proofs establish conformal coverage through calibrated scores and exchangeability, while propositions bound the regularized set size; the supplied tables describe 5% evaluations.
- Proofs: Theorem 1 defines each calibration score as the smallest threshold at which the true label enters the predictive set.
- Proofs: The calibrated threshold is the ⌈(1 − α)(n + 1)⌉ smallest score, and exchangeability yields the coverage argument.
- Proofs: The upper-bound proof requires the conformity scores to be almost surely distinct, which holds when the predicted probability of the true class is positive.
- Proofs: Proposition 2 shows the regularized predictive set has size at most k∗ under the stated calibrated-threshold condition.
- Proofs: Proposition 3 considers the stronger condition that conditional coverage equals 1 − α for every input x.
B RANDOMIZED PREDICTORS
Randomization interpolates between neighboring predictive-set sizes to attain exact target coverage, while differing from the deterministic procedure by at most one element.
- Randomization is needed to achieve exact 1 − α coverage when available set sizes straddle the target.
- The randomized output differs from the non-randomized procedure by at most one element, and deterministic sets can be selected with a flag.
- If size k and k − 1 sets have 91% and 89% coverage, respectively, the procedure chooses each with equal probability to reach 90%.
- In general, the randomization probabilities are chosen so the weighted average of the two coverages equals the desired level.
C IMAGENET AND IMAGENETV2 RESULTS FOR α = 5%
The paper repeats Experiments 1 and 2 at α = 5% and directs readers to tables reporting the resulting evaluations.
- Experiments 1 and 2 were repeated with α = 5%.
- The results are reported in Tables 6 and 7.
D COVERAGE AND SIZE CONDITIONAL ON IMAGE DIFFICULTY
RAPS’s adaptiveness varies with image difficulty: regularization shifts coverage toward easier and medium-difficulty images, while very difficult images remain challenging.
- Increasing λ decreases coverage for more difficult images and increases it for easier images.This pattern is observed when stratifying coverage by the true label’s rank among predicted classes.
- For the most difficult images, APS can produce large sets that still rarely contain the true class.
- Table 8 reports RAPS coverage and size across difficulty levels at a desired coverage of 90%.Difficulty is defined by the ranking of the true class’s estimated probability.
- As λ increases, coverage improves for medium-difficulty examples but not for very rare and difficult ones.
E CHOOSING kreg AND λ TO OPTIMIZE SET SIZE
The method adaptively chooses kreg and λ using held-out data to optimize set size or adaptiveness while preserving conformal coverage through fresh calibration data.
- The procedures for choosing kreg and λ optimize either set size or adaptiveness and outperform APS in both cases.
- Algorithm 4 estimates the smallest fixed set size achieving coverage from ranked class indices and labels.It returns k̂* as the estimated smallest fixed-size set achieving coverage.
- kreg is estimated from a small tuning split, with m ≈1000 usually sufficient because performance is fairly insensitive to kreg.
- λ is selected from a coarse grid by minimizing set size on held-out samples.The tested values were 0.001, 0.01, 0.1, 0.2, and 0.5.
- Using tuning data to select k̂* and λ does not invalidate coverage because conformal calibration uses fresh data.
F COMPARISON WITH LEAST AMBIGUOUS SET-VALUED CLASSIFIERS
Compared with LAC, RAPS offers a trade-off: LAC achieves slightly smaller sets, while RAPS provides substantially better adaptiveness and more balanced coverage across image difficulty.
- LAC achieves correct coverage and systematically smaller sets than RAPS, but the size difference is slight.The larger gaps are between APS and the smaller-set methods.
- LAC has dramatically worse coverage for hard images than for easy ones, whereas RAPS shows a much smaller coverage gap.
- RAPS achieves the best overall adaptiveness on the SSCV metric, outperforming both APS and LAC.
- LAC targets the smallest possible set size rather than adaptiveness, sacrificing coverage balance across difficulty levels.
- The comparison evaluates coverage and set size for nine Imagenet classifiers on Imagenet-Val.