PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformal Prediction Sets Quantify Information Gain: A Theoretical Perspective

Kevin Zhang, Stephen Bates

arXiv:2610.08785v1cs.LG

TL;DR

Conformal set size is often used as an uncertainty heuristic, but its information-theoretic basis is not well established. The paper develops decision-theoretic information measures for set-valued prediction and relates set-size reduction to Shannon mutual information and the data processing inequality.

  • Problem

    Conformal set size is used as an uncertainty heuristic, but prior one-sided bounds do not establish whether its reduction is a principled information measure.

    Source: §1 · p. 1

  • Method

    The paper uses decision-theoretic generalized information measures for set-valued prediction, including an integral representation of Shannon mutual information.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    In standard classification, set-size reduction is bracketed by calibration-dependent conformal information measures and satisfies an approximate data processing inequality; the theory is validated across 11 settings.

    Source: §1 · p. 1, §6 · p. 9

  • Takeaways & Limitations

    These results motivate conformal set-size reduction as a principled information-gain metric.

    Source: §6 · p. 9

    The theory covers classification with probability and APS scores, while experiments use sequential feature selection with a globally fixed ordering.

    Source: §6 · p. 9

Abstract

from arXiv · show

Conformal prediction is a popular tool for uncertainty quantification that outputs prediction sets with finite-sample coverage guarantees. While prediction set size is commonly used as a heuristic measure of uncertainty, the information-theoretic basis for this interpretation remains poorly understood. In this work, we provide such a foundation using a decision-theoretic generalization of entropy tailored to set-valued prediction. In particular, we introduce a family of generalized information measures based on the size and coverage of conformal prediction sets. Notably, Shannon mutual information admits an exact integral representation in terms of these measures. We then show that, in standard classification settings, the reduction in conformal set size from additional information (i) is sandwiched between calibration-dependent members of this family and (ii) obeys a data processing inequality, both up to finite-sample calibration and model error terms. Together, our results formally relate conformal prediction to classical information-theoretic quantities and justify using set-size reduction as an information gain metric. Empirically, we validate our theory across 11 classification settings and show that set-size reduction and Shannon mutual information can rank features differently in a greedy feature selection experiment.

1 INTRODUCTION

Conformal set size is used as a heuristic for uncertainty and motivates measuring information gain, but its relationship to information-theoretic quantities has not established whether set-size reduction is principled. This work supplies a decision-theoretic foundation and relates set-size reduction to generalized information measures and Shannon mutual information.

1 INTRODUCTION

  • Conformal prediction offers finite-sample, distribution-free coverage, while expected set size measures inefficiency and is used as an operational heuristic for instance-level uncertainty.More ambiguous inputs typically produce larger sets.Source: §1 · p. 1
  • Prior one-sided bounds relate conformal set size to conditional entropy but do not establish whether set-size reduction itself is a principled information measure.The paper instead analyzes conformal prediction using decision-theoretic entropy.Source: §1 · p. 1
  • The paper introduces generalized information measures for set-valued prediction that recover Shannon mutual information through an exact integral representation.It also proves that, in standard classification, set-size reduction is approximately bracketed by calibration-dependent measures and obeys a data processing inequality.Source: §1 · p. 1
  • The theory is empirically validated across 11 classification settings, where set-size reduction and Shannon mutual information can produce different feature-selection orderings.Source: §1 · p. 1

2 PRELIMINARIES

The preliminaries set up decision-theoretic information measures and split conformal prediction for categorical classification. They define calibration-based prediction sets and review probability and APS scores, whose choices determine set structure and size.

2 PRELIMINARIES

  • The setting uses a generic feature vector X and categorical target Y; the score function determines conformal-set size and structure.Source: §2 · p. 2
  • Generalized entropy is based on Bayes risk under a chosen loss, and generalized mutual information measures the reduction in that risk given side information.It is nonnegative; Shannon entropy is recovered using log loss and the probability simplex as the action space.Source: §2 · p. 2
  • Split conformal prediction calibrates scores to form a threshold, then includes labels whose test-point scores fall below it; exchangeability and distinct scores give a finite-sample marginal coverage guarantee.The stated bound is at most 1 − α + 1/(n+1).Source: §2 · p. 2
  • Under the probability score, each label receives the negative predicted probability, so the conformal set is a level set determined by the calibrated cutoff.Source: §2 · p. 2
  • APS scores accumulate predicted probability for labels at least as likely as the true label, with randomization for boundary-label inclusion and random tie breaking.Source: §2 · p. 2

3 CONFORMAL ENTROPY AND MUTUAL INFORMATION

The paper develops set-valued generalized information measures and relates conformal set-size reduction to classical information quantities. It establishes calibration- and model-error-qualified bounds and a data processing inequality, with an extension to the APS score.

3.1 GENERALIZED INFORMATION MEASURES FOR THE PROBABILITY SCORE

  • The set-valued loss penalizes prediction-set size and miscoverage, and its induced conformal mutual information uniformly averages over λ to recover Shannon mutual information.For the probability score, the Bayes-optimal set includes labels whose conditional probabilities meet the threshold; with an oracle model it corresponds to a calibrated conformal set.Source: §3.1 · p. 3

3.2 APPROXIMATE INFORMATION GAIN USING CONFORMAL SET-SIZE REDUCTION

  • Conformal set-size reduction is bracketed by calibration-dependent conformal mutual information terms, up to calibration and model error.Under the stated conditions, its expected value is asymptotically non-negative when model divergence is zero and the variance term remains bounded.Source: §3.2 · p. 4

3.3 DATA PROCESSING INEQUALITY FOR CONFORMAL SET-SIZE REDUCTION

  • On average over calibration draws, a degraded feature reduces set size no more than its source feature, up to model imperfection and an O(n^-1/2) calibration term.The result assumes the stated regularity conditions for controlling the finite-sample calibration term.Source: §3.3 · p. 5

3.4 EXTENSION OF CONFORMAL MUTUAL INFORMATION FOR APS SCORE

  • For APS, randomizing the information index and set-valued action extends the sandwich bound, but conditional miscoverage can add slack.For oracle models, this slack vanishes in the upper bound.Source: §3.4 · p. 6

4 RELATED WORK

Prior work has treated conformal set size as predictive inefficiency or an operational uncertainty measure, but its formal information-theoretic basis remains limited. This paper uses decision-theoretic generalized information to connect set-size reduction directly to information gain.

4 RELATED WORK

  • Empirical studies use conformal set size as an operational uncertainty measure, while theoretical work characterizes predictive inefficiency without formally motivating set size as uncertainty or information.Source: §4 · p. 7
  • Earlier theoretical work upper-bounds conditional entropy using set size and coverage, whereas this paper derives two-sided bounds through generalized entropy and links them to Shannon mutual information.The authors characterize this as a direct information-theoretic interpretation not captured by the earlier one-sided approach.Source: §4 · p. 7
  • Information-gain methods are used in active learning and feature acquisition, and prior work has separately studied decision-theoretic information and conformal acquisition based on set size.The paper positions its contribution as identifying theoretical connections among these ideas.Source: §4 · p. 7

5 EXPERIMENTS

Across 11 classification datasets, the experiments assess the sandwich bounds and data processing inequality, then compare conformal set-size reduction with Shannon mutual information in sequential feature selection. Set-size reduction consistently selects features producing smaller prediction sets, while the two criteria can order features differently.

5 EXPERIMENTS

  • The experiments examine theory and feature selection across 11 synthetic, image, and tabular datasets, using oracle conditional distributions for synthetic settings and a fixed evaluation pool for the analyses.Feature selection compares conformal set-size reduction and Shannon mutual information with accuracy-gain and random-selection baselines.Source: §5 · pp. 7–8, §5.2 · p. 9

5.1 SANDWICH AND DPI VALIDATION

  • As calibration and test sizes increase, normalized bound violations approach zero and intervals narrow; DPI violations shrink and the gap concentrates around non-negative values with more calibration data.At very small calibration sizes and large K, the calibration factor can make bounds vacuous; finite-sample effects can also produce negative DPI gaps.Source: Figure 3 · p. 8, §5.1 · p. 8

5.2 FEATURE SELECTION

  • Δα selection consistently yields the smallest sets; accuracy-based selection achieves the highest top-1 accuracy, while Δα matches it on top-5, and CIFAR-100 retraining preserves the ranking difference.The probability-score and APS criteria differ in relative performance.Source: §5.2 · p. 9, Table 1 · p. 9

6 DISCUSSION

The paper connects conformal set-size reduction to generalized information measures and Shannon mutual information, while identifying boundaries on the theory and evaluation. Its analysis is limited to classification with probability and APS scores, and its empirical feature-selection study uses a globally fixed ordering.

6 DISCUSSION

  • Set-size reduction is bracketed by calibration-dependent conformal mutual information measures and satisfies an approximate data processing inequality, supporting its interpretation as information gain.Source: §6 · p. 9
  • The theory covers classification with probability and APS scores, while empirical evaluation is limited to sequential feature selection with a globally fixed ordering.The authors identify other conformal scores, regression, and active feature acquisition as possible future directions.Source: §6 · p. 9

SUPPLEMENTARY MATERIAL FOR CONFORMAL PREDICTION SETS QUANTIFY INFORMATION GAIN: A THEORETICAL PERSPECTIVE

The supplementary proofs establish properties of conformal entropy and mutual information, then derive bounds for probability-score conformal set-size reduction. They also characterize optimal set-valued predictors and establish an information integral representation and a generalized data processing inequality.

A PROOFS

  • The proof roadmap establishes conformal entropy and mutual-information results, concentration results for finite-sample miscoverage terms, and main results for probability and APS scores.Source: A PROOFS · p. 13

A.1 PROPERTIES OF CONFORMAL ENTROPY AND MUTUAL INFORMATION

  • The foundational results cover optimal set-valued predictors, an integral representation of Shannon mutual information, and a data processing inequality for generalized mutual information.Source: §A.1 · p. 13

A.1.1 PROPOSITION 2: OPTIMAL SET-VALUED PREDICTORS

  • For fixed λ, a pointwise risk-minimizing set includes labels with conditional probability above λ and excludes those below λ; the resulting superlevel-set rule achieves conformal entropy.Source: §A.1.1 · p. 13

A.1.2 THEOREM 1: INTEGRAL REPRESENTATION OF SHANNON INFORMATION

  • The proof derives Shannon information's integral representation from the corresponding representation for Shannon entropy.Source: §A.1.2 · pp. 13–14

A.1.3 PROPOSITION 3: OPTIMAL SET-VALUED PREDICTORS FOR RANDOM Λ

  • When Λ depends on X, an optimal predictor includes labels above the input-specific threshold and excludes those below it, with boundary choices allowed through U.Source: §A.1.3 · p. 14

A.1.4 LEMMA 2: GENERALIZED INFORMATION DPI

  • For a finite action space and Markov chain Y → X → X̃, generalized conditional entropy is no smaller given X̃ than given X, yielding the generalized information data processing inequality.Source: §A.1.4 · pp. 14–15

A.2 PROBABILITY SCORE RESULTS

  • The probability-score proofs establish bounds on conformal set-size reduction by first relating conformal entropies at different calibrated thresholds.The proof uses an intermediate conformal entropy term to isolate conformal mutual information.Source: §A.2 · p. 15, §A.2.1 · p. 15

A.2.1 THEOREM 2: PROBABILITY SCORE SANDWICH BOUND

  • The probability-score proof converts a two-sided bound on an intermediate entropy difference into corresponding bounds on Δα.It uses the relationship between conformal prediction sets and Bayes-optimal actions.Source: §A.2.1 · pp. 15–16

A.2.2 CONCENTRATION OF FINITE-SAMPLE MISCOVERAGE DIFFERENCE

This section develops expectation and high-probability controls for finite-sample calibration terms in miscoverage differences, including APS-score sandwich bounds and data-processing results. It also notes where dataset statistics are summarized.

A.2.2 CONCENTRATION OF FINITE-SAMPLE MISCOVERAGE DIFFERENCE

  • With distinct scores and k = ⌈(1 − α)(n + 1)⌉ ≤ n, training-conditional coverage 1 − ε(i) follows Beta(k, n + 1 − k) for i ∈ {1, 2}.The result follows because the calibrated score threshold maps through the score CDF to the k-th order statistic of n uniform samples.Source: §A.2.2 · p. 17

A.2.2 CONCENTRATION OF FINITE-SAMPLE MISCO‌VERAGE DIFFERENCE

  • Although the two miscoverages have the same mean, the weighted calibration term can have nonzero expectation because calibrated thresholds are correlated with miscoverages.The section gives an expectation control when the reciprocal calibrated threshold has finite variance and a high-probability control under almost-sure threshold positivity; under additional smoothness, the expectation bound can improve to O(1/n).Source: §A.2.2 · pp. 17–18

A.2.3 COROLLARY 1: PROBABILITY SCORE SANDWICH BOUND IN EXPECTATION; A.2.4 COROLLARY 3: PROBABILITY SCORE SANDWICH BOUND WITH HIGH PROBABILITY

  • The probability-score sandwich corollaries derive their expectation and high-probability results by applying Lemmas 5 and 6, respectively, to Theorem 2.The supplied passages describe the proof transfers but do not include the resulting inequalities.Source: §A.2.3 · p. 18, §A.2.4 · pp. 18–19

A.2.5 THEOREM 3: PROBABILITY SCORE DPI; A.2.6 COROLLARY 2: PROBABILITY SCORE DPI IN EXPECTATION; A.2.7 COROLLARY 4: PROBABILITY SCORE DPI WITH HIGH PROBABILITY

  • For probability scores, Theorem 3 converts the generalized-mutual-information data-processing inequality into an approximate result for conformal sets at fixed calibration, with expectation and high-probability corollaries.The high-probability corollary holds with probability at least 1 − δ over the calibration draw under the stated positivity condition.Source: §A.2.5 · p. 19, §A.2.6 · p. 19, §A.2.7 · p. 19

A.3.1 REGULARITY OF THE INDUCED APS THRESHOLD

  • For APS scores, a calibrated threshold below 1 ensures the induced probability threshold lies in (0, 1] and has bounded reciprocal expectation and divergence.These regularity bounds support the subsequent APS analysis.Source: §A.3.1 · p. 20

A.3.2 THEOREM 4: APS SCORE SANDWICH BOUND

  • The APS analysis first sandwiches the cross-threshold conformal entropy difference, then uses APS Bayes-optimality to translate that bound into a conformal set-size reduction bound.The theorem considers a fixed calibration dataset and finite bounds.Source: §A.3.2 · pp. 21–22

A.3.3 DECOMPOSITION OF THE APS CALIBRATION TERM

  • Unlike probability scores, APS calibration terms involve input-dependent thresholds and miscoverage, producing covariance terms that typically do not shrink with sample size.Under the oracle model, the first covariance remainder vanishes because randomized APS achieves exact conditional coverage; the corresponding second remainder need not vanish.Source: §A.3.3 · pp. 22–23

A.3.4 COROLLARY 5: APS SCORE SANDWICH BOUND IN EXPECTATION

  • The APS expectation corollary applies the sandwich result under finite-moment conditions and bounds the variance of the average miscoverage difference by 1/n.Its assumptions include the conditions and quantities defined for Theorem 4 and Lemma 9.Source: §A.3.4 · pp. 23–24
  • The proof takes expectations over Dcal in Theorem 4, substitutes Equation 118, and uses E[R(2)] ≥ −E[κ(2)v(2)] and E[R(1)] ≤ E[¯κ(1)¯v(1)] to obtain Equation 117.Source: §A.3.4 · p. 24

A.3.5 COROLLARY 6: APS SCORE SANDWICH BOUND WITH HIGH PROBABILITY

  • Corollary 6 states a high-probability APS-score sandwich bound under distinct calibration scores, i.i.d. calibration and test data, and a stated condition on ηδ/2.The bound holds with probability at least 1 −δ over the calibration data.Source: §A.3.5 · p. 24

A.3.6 THEOREM 5: APS SCORE DPI

  • Theorem 5 extends the data processing inequality to APS scores for a fixed calibration dataset and nominal coverage level.The proof frames the result as an approximate conformal-prediction-set version of the generalized mutual information inequality.Source: §A.3.6 · p. 25

A.3.7 COROLLARY 7: APS SCORE DPI IN EXPECTATION

  • Corollary 7 gives an expectation version of the APS-score data processing inequality under its stated calibration and coverage conditions.The right-hand side is finite.Source: §A.3.7 · p. 25

A.3.8 COROLLARY 8: APS SCORE DPI WITH HIGH PROBABILITY

  • Corollary 8 gives a high-probability version of the APS-score data processing inequality under distinct-score, i.i.d., and ηδ/2 conditions.It holds with probability at least 1 −δ over the calibration data.Source: §A.3.8 · p. 26

B DATASETS

  • Dataset statistics, including sample sizes and feature dimensions, are summarized in Table 2.Source: B DATASETS · p. 27

B.1 DATASET DETAILS

The appendix describes the datasets and feature constructions used in the experiments, then details candidate-feature selection and model training. It also specifies how degradation and hyperparameter selection are implemented.

B.1 DATASET DETAILS

  • S3 uses 40 classes with an 8-class head group holding 70% of prior mass and a 32-class tail group holding the remaining 30%.Its base features combine shared head or tail centers with smaller per-class offsets.Source: §B.1 · p. 27

B.2 FEATURE CONSTRUCTION DETAILS

  • Synthetic features split coordinates into base Z and additional X, tabular features follow natural groupings, and image X is the residual after blurring the original to form Z.For image datasets, the base blur uses standard deviation σ = 3.Source: §B.2 · p. 29

B.3 DEGRADED FEATURE CONSTRUCTION DETAILS

  • Degradation leaves Z unchanged and applies noise to synthetic and tabular X or additional Gaussian blur to image residuals; the main text reports the medium level, γ = 0.5.The experiments consider γ values of 0.25, 0.5, and 1.0.Source: §B.3 · p. 30

B.4 CANDIDATE FEATURE DETAILS

  • Synthetic candidate features are individual coordinates of X, while tabular inputs are candidates individually except that HAR is grouped into 51 sets of 11 features.Image candidates are groups of channels from a frozen backbone representation, intended as meaningful candidates rather than physically acquirable features.Source: §B.4 · p. 30

C.1 MODEL ARCHITECTURES

  • Real datasets use MLPs for tabular data, CNNs for FMNIST and CIFAR, and a fine-tuned pretrained ResNet-18 for ImageNet-1k; synthetic data use oracle probabilities.Tabular models have dataset-specific two- or three-layer widths.Source: C TRAINING DETAILS · p. 31, §C.1 · p. 31

C.2 TRAINING OPTIMIZATION

  • Real-dataset models are trained with Adam and cross-entropy, using validation-loss early stopping and dataset-specific epoch budgets and patience.Image training also uses flips and random crops, with cosine annealing of the learning rate.Source: §C.2 · p. 31

C.3 MASK-CONDITIONED MODEL TRAINING

  • Mask-conditioned tabular models receive the feature vector multiplied by an observation mask alongside the mask, while image models use a frozen backbone representation and a masked prediction head.ImageNet-1k uses a linear head for scalability.Source: §C.3 · p. 32

C.4 HYPERPARAMETER SELECTION

  • Hyperparameters are selected from 27 configurations by augmented-model validation accuracy, then shared across augmented, base, and degraded-feature models.The grid varies learning rate, weight decay, and dropout.Source: §C.4 · p. 32

C.5 ACCURACY AND COVERAGE METRICS

Empirical coverage stays close to nominal across datasets, with greater variability on smaller evaluation pools. The appendix also details bound estimation and validation protocols.

C.5 ACCURACY AND COVERAGE METRICS

  • Classification accuracy generally decreases as features are degraded or removed, while mask-conditioned models perform comparably to full-feature models despite supporting arbitrary feature subsets.The experiments use simplified architectures and training procedures rather than targeting state-of-the-art accuracy.Source: §C.5 · p. 33
  • Both probability and APS scores achieve empirical coverage close to the nominal 0.9 level, with greater variability on smaller evaluation pools such as Wine and HAR.Coverage was assessed over 100 calibration/test splits per dataset.Source: §C.5 · p. 33

D.1 ESTIMATION OF THE BOUNDS

  • For real-world lower-bound estimation, the conditional expectation is approximated using the 10 nearest neighbors; synthetic settings instead use 256 Monte Carlo draws from the known conditional distribution.Neighbor distances use base-feature Euclidean distance for tabular data and learned-embedding cosine distance for image data.Source: §D.1 · p. 34
  • For synthetic data, the bounds use oracle conditional distributions and set D(i) to zero; for real-world data, D(i) is also set to zero, making those bounds approximate.The real-world plug-in estimates cannot determine D(i) without the true conditional distribution; the synthetic experiments assess bounds using true conditional distributions.Source: §D.1 · p. 34

D.2.1 VALIDATION EXPERIMENTS

  • Validation replicates draw disjoint calibration and test sets without replacement; seeds yield independent draws, while samples are nested across sample sizes for a given seed.The sets do not necessarily partition the evaluation pool.Source: §D.2.1 · p. 34
  • Validation varies calibration and test sample sizes, nominal coverage from 0.8 to 0.99, and calibration size for DPI checks; paired procedures share calibration and test sets.DPI experiments use test sets up to 10,000 samples and calibration sizes of 10, 100, or 1,000 when available.Source: §D.2.1 · pp. 34–35

D.2.2 FEATURE SELECTION EXPERIMENTS

In the additional feature-selection results, set-size reduction yields smaller prediction sets than Shannon MI early in some feature-selection trajectories. The appendix describes the selection protocol, its evaluation metrics, and computational constraints.

D.2.2 FEATURE SELECTION EXPERIMENTS

  • Each feature-selection replicate splits samples into 50% selection, 25% calibration, and 25% test sets; conformal criteria use five-fold cross-fitting within the selection set.Shannon mutual information and accuracy are computed directly on the full selection set without calibration.Source: §D.2.2 · p. 35
  • Greedy selection adds the highest-scoring remaining feature at each step, creating one global ordering shared across test samples; each prefix is calibrated and evaluated at 0.9 nominal coverage.The criteria score candidates using predicted probabilities.Source: §D.2.2 · p. 35
  • The APS IΛ criterion is omitted because repeating its k-NN conditional-expectation approximation for every candidate at every step is computationally expensive and adds approximation error.The approximation is needed to estimate the conditional expectation over candidate features given the selected subset.Source: §D.2.2 · pp. 35–36

D.3.1 VALIDATION EXPERIMENTS

  • Validation metrics capture how far set-size reduction falls outside theoretical bounds, how often violations occur, and the bounds' interval width; about 3% of invalid-bound replicates are excluded without changing conclusions.The normalized margin measures violation magnitude, while violation rate counts out-of-bound cases.Source: §D.3.1 · p. 36

D.3.2 FEATURE SELECTION EXPERIMENTS

  • Feature-selection performance is assessed along each ordering using set size, empirical coverage, and top-1 and top-5 accuracy, averaged across intermediate selection steps.The endpoints are excluded because all criteria observe the same features there.Source: §D.3.2 · pp. 36–37

D.4 IMPLEMENTATION DETAILS

  • Presorting class probabilities exactly speeds validation computations, but it cannot be used for feature selection, where criterion-dependent subsets are evaluated naively.The number of possible subsets grows exponentially.Source: §D.4 · p. 37

E.1 SANDWICH VALIDATION RESULTS

Across most feature-selection settings, set-size reduction selects features yielding smaller prediction sets and typically better top-5 accuracy, though Shannon MI can rank features differently. Validation finds that finite-sample deviations shrink with larger samples, while subset-specific evaluation preserves set-size advantages for Δα.

E.1 SANDWICH VALIDATION RESULTS

  • Normalized margin mainly shrinks with larger test sets, while normalized width shrinks with calibration size.With test size fixed, margin is largely insensitive to calibration size; increasing calibration size concentrates conditional miscoverage around nominal coverage and tightens the bounds.Source: §E.1 · pp. 38–39, Figure 7 · p. 39, Figure 9 · p. 40
  • Violation rates generally fall as calibration and test samples grow; APS usually has lower rates, while probability-score Wine remains an exception at roughly 60%–80%.The violation rate is nonzero at smaller sample sizes and decreases toward zero as sample size increases; APS’s lower rate is consistent with wider bounds.Source: §E.1 · pp. 38–39, Figure 8 · p. 39
  • At 99% nominal coverage, all three metrics rise sharply for most datasets, as smaller calibrated thresholds amplify calibration terms and empirical-estimate variance.The bounds remain well-behaved through coverage levels of 0.95 or lower for most datasets, with exceptions for Wine, FMNIST, and CIFAR-10.Source: §E.1 · p. 40, Figure 10 · p. 40

E.2 DPI VALIDATION RESULTS

  • Across degradation levels, DPI gaps can be negative on individual draws but generally concentrate around a non-negative mean as calibration size grows; stronger degradation usually raises the average gap.These trends hold for both probability and APS scores, and the approximate DPI permits negative gaps through finite-sample calibration and model-error terms.Source: §E.2 · p. 41, Figure 11 · p. 41

E.3 FEATURE SELECTION RESULTS

  • Across most settings, set-size-reduction selection yields the smallest prediction sets and typically best top-5 accuracy, while plug-in Iλ often yields larger real-world sets.The Δα criteria yield noticeably smaller sets early on S3 and CIFAR-100, while the probability-score gain on Letter is small; model imperfection is cited for Iλ’s weaker real-world proxy performance.Source: §E.3 · pp. 41, 43, Table 5 · p. 42, Figure 12 · p. 43

E.4 SUBSET-SPECIFIC MODEL ABLATION

  • Subset-specific models reduce set sizes for every criterion, yet Δα still yields smaller sets at every k, with the largest differences in the first two steps.The Δα advantage averaged over k falls from 4.07 to 1.89 under the probability score and from 3.84 to 2.11 under APS.Source: §E.4 · p. 43, Table 6 · p. 44

F COMPUTE RESOURCES

Training and analysis ran on a single Ubuntu 24.04 machine with 32 GB RAM and an NVIDIA GeForce RTX 5070 Ti GPU.

F COMPUTE RESOURCES

  • The hyperparameter grid search took about 10 hours, model training about 12 hours, and downstream analyses about 4 hours on one machine.Validation experiments used a single CPU thread without GPU acceleration; feature-selection mask-conditioned models were evaluated on the GPU.Source: F COMPUTE RESOURCES · p. 45

G EXPERIMENTAL ASSETS

The Human Activity Recognition data came from UCI; ImageNet-1k used a 64 × 64 downsampled version, while other datasets came from OpenML or torchvision.

G EXPERIMENTAL ASSETS

  • The study used UCI Human Activity Recognition, downsampled ImageNet-1k, and other datasets from OpenML or torchvision.The Human Activity Recognition data are under CC BY 4.0; the ImageNet-1k version is the ChocolateDave/imagenet-64 mirror, and pretrained ResNet-18 weights are torchvision IMAGENET1K V1.Source: G EXPERIMENTAL ASSETS · p. 45
Loading 2610.08785v1…