Source-linked AI summary
Conformal Prediction Under Covariate Shift
Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candes, Aaditya Ramdas
TL;DR
Ordinary conformal prediction relies on exchangeability, but training and test covariate distributions may differ. The paper introduces weighted conformal prediction for weighted-exchangeable settings, with covariate shift as a practically important case. The method yields distribution-free prediction intervals and extends to related latent-variable and missing-data settings, while some conditional-coverage problems remain open.
Problem
Conformal prediction must be extended to settings where the training and test data are no longer exchangeable, especially when their covariate distributions differ.
Method
The paper develops weighted conformal prediction for weighted-exchangeable data and applies it under covariate shift using the likelihood ratio between test and training covariate distributions.
Results
Weighted conformal prediction provides distribution-free prediction intervals under covariate shift, while ordinary split conformal prediction can lose coverage on shifted test data.
Takeaways & Limitations
Under covariate shift, the weighted method is computationally as easy as ordinary conformal prediction when the likelihood ratio is known or accurately estimated.
Takeaways & Limitations
Important practical and philosophical problems in assumption-lean conditional predictive inference remain open.
Abstract
from arXiv · showhide
We extend conformal prediction methodology beyond the case of exchangeable data. In particular, we show that a weighted version of conformal prediction can be used to compute distribution-free prediction intervals for problems in which the test and training covariate distributions differ, but the likelihood ratio between these two distributions is known---or, in practice, can be estimated accurately with access to a large set of unlabeled data (test covariate points). Our weighted extension of conformal prediction also applies more generally, to settings in which the data satisfies a certain weighted notion of exchangeability. We discuss other potential applications of our new conformal methodology, including latent variable and missing data problems.
1 Introduction
Conformal prediction constructs distribution-free prediction bands from exchangeable data using symmetric nonconformity scores and quantile-based calibration. Its finite-sample coverage guarantee is nearly exact and does not depend on the underlying distribution or the particular score function.
- Construction: A score function measures how atypical a candidate point is relative to the remaining data, with larger values indicating lower conformity.The score operates on an unordered multiset, so its construction is symmetric in the observations.
- Construction: For each covariate x, the prediction band includes candidate outcomes y whose nonconformity scores satisfy the conformal quantile criterion.The procedure recalculates scores for each candidate y and includes y when the rank-based condition holds.
- Guarantee: Theorem 1 provides finite-sample coverage at least 1 − α for any score function under exchangeability.With almost surely no ties, coverage is also bounded above by 1 − α + 1/(n + 1).
- Guarantee: Exchangeability of the nonconformity scores follows from their symmetric construction, allowing the quantile lemma to establish the coverage result.Random tie-breaking extends the upper-bound statement to settings with ties.
- Foundations: Training and test observations are assumed exchangeable, a weaker condition than i.i.d. sampling.The framework seeks coverage without assumptions on the common distribution P.
2 Covariate shift
Weighted conformal prediction extends distribution-free coverage to covariate shift by reweighting nonconformity scores according to the test-to-training covariate likelihood ratio. In the airfoil example, oracle and estimated weights restore nominal coverage, while weighting reduces effective sample size and can lengthen intervals.
- Under covariate shift, training and test covariate distributions differ while the conditional distribution of Y|X remains unchanged.
- Weighted conformal prediction replaces the ordinary empirical score distribution with likelihood-ratio-weighted scores, yielding scores that resemble those from the test population.
- The method generalizes to weighted exchangeable data, including exchangeable and independent non-identically distributed data as special cases.
- Airfoil data example: 82.2% average coverage under ordinary split conformal during covariate shift rose to 90.8% with oracle-weighted conformal prediction.
- Airfoil data example: Weighted intervals were longer than ordinary intervals at equivalent effective sample size because the regression function was fit on training data from a different distribution.
- Airfoil data example: 91.0% average coverage was achieved with both logistic-regression and random-forest estimated weights, with only slightly greater dispersion than oracle weighting.
3 Weighted exchangeability
The paper generalizes conformal prediction from exchangeable data to weighted exchangeability, yielding weighted coverage results and a corresponding weighted conformal procedure. Independent non-identically distributed draws, including covariate shift, form an important special case, though useful upper bounds require conditions on the weights.
- Proof strategy: Conditioning on the unordered multiset of scores reduces conformal coverage to the probabilities that the test score assumes each observed value.Under ordinary exchangeability, these conditional probabilities are uniform.
- Definition and scope: Weighted exchangeability factorizes the joint density into an observation-specific weight function and a permutation-invariant function.Ordinary exchangeability is recovered when all weights equal one.
- Definition and scope: Independent draws with distributions absolutely continuous relative to a reference distribution are weighted exchangeable, with weights given by the corresponding Radon-Nikodym derivatives.This includes the covariate shift model and extends beyond independent sampling to dependent structures satisfying weighted exchangeability.
- Weighted conformal prediction: The weighted quantile argument extends conformal prediction by assigning permutation probabilities according to the weight functions rather than treating all permutations equally.The resulting weighted conformal band is obtained from the generalized lemma and applies to arbitrary score functions.
- Limitations: Unlike the unweighted case, a useful upper coverage bound may require further conditions because the weighted conditional distribution can have a large maximum jump.In the unweighted case, the corresponding jump is always 1/(n + 1).
- Covariate shift: In covariate shift, training and test observations receive weights one and w(x), respectively, so the general theorem yields the covariate-shift conformal result.The same result carries over to split conformal prediction, where the interval simplifies using the corresponding covariate weight.
4 Discussion
The paper extends conformal prediction to weighted exchangeable data, with covariate shift as a practically important case where computation remains as easy as ordinary conformal prediction. It also develops locally weighted coverage ideas and identifies important unresolved limitations for assumption-lean conditional inference.
- Weighted conformal prediction covers exchangeable data and independent, non-identically distributed data as special cases.
- Under covariate shift, known test-to-training covariate likelihood ratios preserve distribution-free prediction intervals while retaining ordinary conformal computational simplicity.The conditional distribution of Y|X remains unchanged, and unknown ratios can be estimated using unlabeled test covariates.
- Graphical models with covariate shift: Low-dimensional graphical-model variables can provide more tractable likelihood-ratio estimates than directly comparing high-dimensional feature distributions.For Z→X→Y, the method estimates weights from the likelihood ratio of Z between test and training data.
- Missing covariates with known summaries: With missing sensitive covariates, weighted conformal prediction can use a population summary when covariate distributions differ only through the sensitive variable's distribution.This requires the conditional distribution X|Z to be unchanged across hospitals.
- Towards local conditional coverage?: Locally weighted coverage approaches conditional coverage when the bandwidth h is small, but the prediction band must be recomputed for each specified center point x0.The resulting guarantee is a locally smoothed form of conditional coverage rather than one global band valid at all x0.
- Towards local conditional coverage?: Exact conditional coverage for almost all covariate values is impossible with nontrivial finite-length intervals, leaving major practical and philosophical problems open.The paper notes that any method with such coverage must have infinite expected length at any non-atom point, and its construction does not provide one band valid at all x0.