Source-linked AI summary
Flexible distribution-free conditional predictive bands using density estimators
Rafael Izbicki, Gilson T. Shimizu, Rafael B. Stern
TL;DR
Conditional validity generally requires strong assumptions, while conditional responses may be asymmetric or multimodal and optimal prediction regions need not be intervals. The paper proposes Dist-split and CD-split, which achieve asymptotic conditional coverage without a particular target-feature dependence assumption; they converge to oracle intervals or regions, respectively.
Problem
Conditional validity requires strong distributional assumptions, while conditional responses may be asymmetric or multimodal and adequate prediction regions may not be intervals.
Source: §1 · pp. 2–3Method
Dist-split produces intervals, while CD-split uses a data-driven feature-space partition to provide local coverage, including in high-dimensional settings.
Source: §1.1 · p. 3Results
Dist-split converges to an oracle interval, while CD-split converges to an oracle region that is generally smaller than an interval.
Source: §1.1 · p. 3Takeaways & Limitations
CD-split's arbitrary regions are appealing for multimodal data, while Dist-split's intervals are easier to interpret.
Source: §1.1 · p. 3Dist-split may produce larger regions when Y|x is not symmetric and unimodal.
Source: §2 · p. 6
Abstract
from arXiv · showhide
Conformal methods create prediction bands that control average coverage under no assumptions besides i.i.d. data. Besides average coverage, one might also desire to control conditional coverage, that is, coverage for every new testing point. However, without strong assumptions, conditional coverage is unachievable. Given this limitation, the literature has focused on methods with asymptotical conditional coverage. In order to obtain this property, these methods require strong conditions on the dependence between the target variable and the features. We introduce two conformal methods based on conditional density estimators that do not depend on this type of assumption to obtain asymptotic conditional coverage: Dist-split and CD-split. While Dist-split asymptotically obtains optimal intervals, which are easier to interpret than general regions, CD-split obtains optimal size regions, which are smaller than intervals. CD-split also obtains local coverage by creating a data-driven partition of the feature space that scales to high-dimensional settings and by generating prediction bands locally on the partition elements. In a wide variety of simulated scenarios, our methods have a better control of conditional coverage and have smaller length than previously proposed methods.
1 Introduction
Conformal prediction bands offer marginal coverage under an i.i.d. assumption, while conditional coverage for every testing point is generally unavailable without strong distributional assumptions. Dist-split and CD-split target asymptotic conditional coverage without a particular target-feature dependence assumption, with simulations reporting better conditional-coverage control and shorter bands than alternatives.
1 Introduction
- Conditional validity requires strong assumptions about the distribution of (X,Y), motivating methods with asymptotic conditional coverage.Conditional validity seeks adequate coverage for each new instance, rather than only on average.Source: §1 · p. 2
- Existing local-coverage methods have not scaled to high-dimensional settings because choosing a subset large enough for many samples yet small enough for local coverage is challenging.These methods compute conformal bands using only instances that fall in a feature-space subset A.Source: §1 · p. 2
1.1 Contribution
- Dist-split and CD-split target asymptotic conditional coverage without assuming a particular type of dependence between the target and features.They are based on conditional density estimators.Source: §1.1 · p. 3
- Dist-split produces interpretable intervals and converges to an oracle interval, whereas CD-split produces generally smaller arbitrary regions and converges to an oracle region.CD-split also controls local coverage in high-dimensional settings through a data-driven partition of feature space.Source: §1.1 · p. 3
- Across a wide variety of simulation studies, the proposed methods show better conditional-coverage control and smaller band length than alternatives in the literature.The supplied passages report this overall comparison without numerical values.Source: §1.1 · p. 3
- In the toy example, both proposed methods get close to controlling conditional coverage, while CD-split yields smaller bands around modes in the bimodal region.Dist-split is restricted to intervals; CD-split can return a collection of intervals around the modes.Source: §1.1 · p. 3
2 Dist-split
Dist-split uses estimated conditional distributions to construct prediction intervals. Under stated density-estimation and smoothness assumptions, it achieves asymptotic conditional coverage and converges to an optimal oracle interval.
2 Dist-split
- Dist-split fits a conditional CDF on one data split, calibrates its conditional-CDF scores on the other, and uses score quantiles to form an interval.The algorithm randomly splits the data, estimates the CDF using D′, and computes quantiles from the scores in T(D).Source: §2 · pp. 4–5
- Its marginal coverage lies between 1−α and 1−α+1/(n+1).The stated upper excess over the target coverage is at most (n+1)^−1.Source: §2 · pp. 4–5
- Under density-estimator consistency and smoothness assumptions, Dist-split achieves asymptotic conditional coverage and converges to an optimal oracle interval.The required conditions include a consistent conditional density estimator and smoothness of the conditional distribution and its quantiles near the target coverage quantiles.Source: §2 · pp. 5–6
- Dist-split converges to the same oracle as conformal quantile regression methods (Romano et al., 2019; Sesia and Candès, 2019), but usually outperforms them in experiments.Source: §2 · p. 6
- For asymmetric or multimodal conditional distributions, Dist-split may produce regions wider than necessary; two intervals can better represent bimodality.Dist-split outputs intervals, while a union of two intervals can better represent a bimodal distribution.Source: §2 · p. 6
3 CD-split
CD-split forms prediction regions from estimated conditional densities, using a data-driven feature partition to target local coverage. Its profile-based distance groups points with similar conditional densities, including points far apart in Euclidean distance, while preserving the oracle cutoff structure.
3 CD-split
- CD-split uses density-threshold regions, which can be smaller than intervals when the target distribution is multimodal.Dist-split intervals can be wider than necessary in multimodal settings.Source: §3 · p. 6
- The CD-split band is locally valid with respect to its partition and is also marginally valid.Theorems analogous to these validity results hold in the multiclass classification setting.Source: §3 · p. 7, §3.1 · p. 10
- The profile distance groups features by conditional-density similarity, so Euclidean-distant points can share a partition element without compromising too much the approximation of local validity to conditional validity.The profile-based partition is designed to make elements large enough to contain more samples, including in high-dimensional settings.Source: §3 · pp. 7–9, Figure 3 · p. 9
- When conditional density depends only on a subset of features, the profile distance ignores irrelevant features, unlike Euclidean distance, which can be strongly affected by them in high dimensions..Source: §3 · p. 8
- CD-split partitions the feature space by applying k-means++ to discretized density profiles rather than to the original features.The profiles are evaluated on a grid to create the clustering inputs.Source: §3 · pp. 9–10
3.1 Multiclass classification
- In multiclass classification, CD-split uses a similar construction for discrete outcomes, and analogous theorems hold.The passage also distinguishes conditional coverage given a feature value from class-specific coverage.Source: §3.1 · p. 10
4 Experiments
The experiments compare conformal methods across regression settings and a classification setting, assessing conditional-coverage deviation and average prediction-band size. CD-split best controls conditional coverage in the regression settings and improves it over Probability-split in classification, though with larger bands.
Regression experiments
- The regression experiments use d = 20 covariates and include asymmetric, bimodal, heteroscedastic, and homoscedastic response settings.The bimodal setting adds d − 1 irrelevant variables to an example from Lei and Wasserman (2014).Source: §4 · pp. 10–11
Experimental setup
- The comparisons include regression-split, local regression-split, quantile-split, Dist-split, and CD-split, with CD-split partitions of size J = ⌈n/100⌉.The local regression-split score uses an estimate of conditional mean absolute deviation; regression-split uses an estimated regression function.Source: §4 · p. 11
- Each setting is run 5,000 times at each sample size with a 90% target coverage level, using random forests for required estimates and FlexCode for conditional densities.Conditional quantiles for Quantile-split are estimated with quantile forests.Source: §4 · p. 11
Regression experiments
- Across regression settings, CD-split best controls conditional coverage and usually has the smallest prediction bands; Dist-split is frequently second on both measures.Conditional-coverage control is measured by the absolute deviation from nominal coverage; all methods are close to 90% marginal coverage.Source: §4 · p. 11, Figure 4 · p. 12
Classification experiment
- In classification, CD-split better controls conditional coverage than Probability-split, but its prediction bands are larger on average.Probability-split corresponds to CD-split with J = 1 partitions.Source: §4 · pp. 11, 13, Figure 5 · p. 13
5 Final remarks
The paper introduces Dist-split and CD-split for asymptotic conditional coverage without assumptions about dependence between targets and features. Simulations report better conditional-coverage control and smaller prediction bands than other methods across varied settings.
Final remarks
- Dist-split and CD-split obtain asymptotic conditional coverage and converge to optimal oracle bands without assumptions about target-feature dependence.Dist-split converges to an oracle interval, while CD-split converges to an oracle region.Source: §5 · p. 13
- Dist-split returns interpretable intervals, whereas CD-split returns smaller, potentially non-interval prediction regions.The authors note that arbitrary regions are appealing for multimodal data.Source: §5 · p. 13
- Across varied simulated settings, both methods yield smaller prediction bands and better conditional-coverage control than methods in the literature; CD-split also performs well in classification.Source: §5 · p. 13
- CD-split uses a data-driven feature-space metric for defining neighborhoods, particularly in high-dimensional settings; the authors suggest it may also help other conformal methods obtain asymptotic conditional coverage.Source: §5 · p. 13
Proofs
The proofs establish convergence properties for estimated conditional-distribution quantities used by Dist-split and CD-split. They rely on stated assumptions and, for CD-split, conditional exchangeability within feature-space partitions.
Proofs
- Under Assumption 2.3, empirical quantiles of estimated conditional-CDF values U_i = bF(Y_i|X_i) converge in probability to α.The proof compares these values with empirical quantiles of the true conditional-CDF values.Source: Related to Dist-split · pp. 15–16
Related to Dist-split
- For Dist-split, the proof approximates estimated conditional quantiles by true conditional quantiles using the continuity and estimation-error assumptions.The stated argument uses Assumptions 2.3 and 2.4 and the convergence of the relevant empirical quantile to 0.5α.Source: Related to Dist-split · p. 16
Related to CD-split
- For CD-split, conditioning on the test feature belonging to a partition element makes the transformed scores i.i.d.; the proof uses their order statistic to establish coverage.The proof describes this conditional argument for the selected indices in the partition.Source: Related to CD-split · p. 16
- CD-split’s proof also characterizes feature points with identical density profiles as equivalent under the stated continuity argument.Source: Related to CD-split · pp. 16–17