PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformalized Quantile Regression

Yaniv Romano, Evan Patterson, Emmanuel J. Candès

arXiv:1905.03222v1stat.MEstat.ML

TL;DR

Existing conformal intervals can be conservative because their lengths vary little across inputs, while direct quantile-regression intervals lack finite-sample validity. The paper proposes conformalized quantile regression, combining quantile regression with conformal calibration; it retains coverage guarantees and tends to produce shorter, heteroscedasticity-adaptive intervals.

  • Problem

    Existing conformal methods may produce nearly constant-width intervals, while quantile-regression intervals can lack finite-sample coverage guarantees.

  • Method

    CQR fits lower and upper quantile regressors, then conformalizes their interval using calibration data.

  • Results

    CQR achieves nominal 90% coverage for conformal methods and shorter average intervals than standard and locally adaptive conformal prediction.

  • Takeaways & Limitations

    CQR provides finite-sample miscoverage control while adapting interval lengths to heteroscedasticity.

  • Takeaways & Limitations

    Locally adaptive conformal prediction can become less adaptive because training residuals underestimate test prediction errors, especially with overfitting neural networks.

Abstract

from arXiv · show

Conformal prediction is a technique for constructing prediction intervals that attain valid coverage in finite samples, without making distributional assumptions. Despite this appeal, existing conformal methods can be unnecessarily conservative because they form intervals of constant or weakly varying length across the input space. In this paper we propose a new method that is fully adaptive to heteroscedasticity. It combines conformal prediction with classical quantile regression, inheriting the advantages of both. We establish a theoretical guarantee of valid coverage, supplemented by extensive experiments on popular regression datasets. We compare the efficiency of conformalized quantile regression to other conformal methods, showing that our method tends to produce shorter intervals.

1 Introduction

Prediction intervals should provide finite-sample, distribution-free coverage while remaining short and locally adaptive. Conformalized quantile regression combines conformal prediction with quantile regression to pursue both goals, including under heteroscedasticity.

  • Motivation: Prediction intervals quantify uncertainty by providing lower and upper bounds intended to contain responses with high probability.The motivation emphasizes this need in high-stakes applications such as drug efficacy and credit-default risk.
  • Existing approaches: Conformal prediction offers nonasymptotic, distribution-free coverage by calibrating held-out residuals from a fitted regression model.Its interval length and local adaptivity depend on the underlying regression model.
  • Existing approaches: Quantile regression adapts intervals to local variability, but its validity is guaranteed only for specific models and asymptotic or regularity conditions.For nominal 90% coverage, the lower and upper conditional quantiles are estimated at the 5% and 95% levels.
  • Contribution: CQR combines conformal prediction with quantile regression to provide finite-sample, distribution-free validity and statistical efficiency.It can wrap around quantile-regression algorithms including random forests and deep neural networks.
  • Method: CQR calibrates quantile-based intervals on a calibration set after fitting lower and upper quantile regressors on a proper training set.Under exchangeability, the resulting interval is guaranteed to satisfy the coverage requirement for arbitrary joint distributions and sample sizes.
  • Evaluation: Across eleven datasets, CQR produces shorter intervals than competing methods while adapting interval lengths to heteroscedasticity.The paper evaluates miscoverage rate and average interval length against other conformal methods.

2 Quantile regression

Quantile regression estimates conditional quantiles and uses them to form intervals that adapt to heteroscedasticity. However, estimated quantile intervals lack a finite-sample coverage guarantee and can substantially undercover.

  • Quantile regression: Conditional quantile regression estimates a specified quantile of Y given the features X.The median is one example of a conditional quantile.
  • Quantile regression: For miscoverage rate α, the interval uses lower and upper conditional quantiles at α/2 and 1 − α/2.The resulting conditional interval is formed from qαlo(x) and qαhi(x).
  • Quantile regression: Quantile regression can use diverse machine-learning methods, with the pinball loss serving as its check-function objective.The paper illustrates the loss function in Figure 1.
  • Estimated intervals: A direct plug-in interval estimates the lower and upper quantiles and outputs [q̂αlo(Xn+1), q̂αhi(Xn+1)].This approach is often adaptive to heteroscedasticity but does not guarantee the desired finite-sample coverage.
  • Limitations: Quantile-function estimates may be asymptotically consistent under certain regularity conditions and specific models, but finite-sample validity requires conformal prediction.The paper contrasts asymptotic consistency with finite-sample coverage guarantees.

3 Conformal Prediction

Split conformal prediction obtains finite-sample-valid prediction intervals by separating model fitting from calibration. Its standard intervals have fixed length across test inputs, motivating variable-width alternatives.

  • Split conformal prediction: Split conformal prediction divides the data into a proper training set and a calibration set, avoiding the repeated fitting required by full conformal prediction.This computational simplification comes at the cost of splitting the data.
  • Calibration: After fitting a regression model, split conformal computes calibration residuals and an empirical residual quantile.The quantile is Q1−α(R, I2), defined at the adjusted empirical level (1 − α)(1 + 1/|I2|).
  • Guarantee: The resulting prediction interval is guaranteed to satisfy the target coverage requirement.This is the finite-sample guarantee of conformal prediction under the stated assumptions.
  • Limitation: Standard split conformal intervals have length 2Q1−α(R, I2), independent of the new input Xn+1.This fixed-width structure limits their ability to adapt interval length across the input space.

4 Conformalized quantile regression (CQR)

CQR constructs prediction intervals by fitting lower and upper conditional quantiles, then conformalizing their calibration errors. This preserves finite-sample coverage guarantees while adapting interval length to heteroscedasticity, with flexibility across quantile-regression algorithms.

  • Adaptivity: Compared with standard split conformal prediction, CQR calibrates conditional quantiles rather than conditional means, producing intervals adaptive to heteroscedasticity.In the simulated experiment, split conformal achieved 91.4% average coverage with 2.91 average length, while the locally adaptive method achieved 91.7% and 2.86.
  • Procedure: CQR splits the data, fits lower and upper conditional quantile functions, and calibrates their conformity scores on a held-out calibration set.The output interval expands the fitted lower and upper quantiles using an empirical calibration quantile.
  • Procedure: The conformity score measures endpoint errors outside the plug-in interval and is non-positive when the response lies inside it.It therefore accounts for both undercoverage and overcoverage.
  • Guarantees: Under exchangeability, split CQR provides marginal distribution-free coverage, with the finite-sample bound P{Y_n+1 ∈ C(X_n+1)} ≤ 1 − α + 1/(|I2| + 1).If conformity scores are almost surely distinct, the procedure is nearly perfectly calibrated.
  • Flexibility: CQR can wrap around quantile-regression algorithms including random forests and deep neural networks, while retaining rigorous coverage control independent of the underlying algorithm.Hyperparameters can be tuned by cross-validation to minimize average interval length without invalidating the coverage guarantee.
  • Practical considerations: The stronger coverage guarantee can come at the cost of slightly longer intervals, and quantile-regression forests may sometimes produce unnecessarily wide intervals.Tuning nominal quantiles during cross-validation can mitigate this conservatism and may yield shorter intervals.

5 Related work: locally adaptive conformal prediction

Locally adaptive conformal prediction adapts intervals to heteroscedasticity by scaling residuals with an estimated local dispersion, but this adds computation and can weaken adaptivity or inflate intervals. Its limitations are especially pronounced under homoskedasticity and when training residuals poorly represent test errors.

  • Method: Locally adaptive conformal prediction replaces absolute residuals with scaled residuals using an estimated local residual dispersion.The dispersion estimate is typically the conditional mean absolute deviation of |Y − µ̂(x)| given X = x.
  • Coverage: Fitting both the conditional mean and dispersion functions only on the proper training set preserves the conformal coverage guarantee.The two functions therefore satisfy the assumptions required by conformal prediction.
  • Practical considerations: The method requires fitting two functions sequentially, making it more computationally expensive than standard conformal prediction.The conditional mean is fitted first, followed by a regression model for residuals given the inputs.
  • Practical considerations: A positive offset γ is added to the scale estimator to improve numerical stability and statistical performance.The offset enters the scaled residual construction as a constant adjustment to σ̂(x).
  • Limitations: When data is homoskedastic, locally adaptive conformal prediction produces inflated intervals relative to the standard method.The paper attributes this presumably to extra variability from estimating both the dispersion and mean functions.
  • Limitations: Using proper-training residuals can systematically underestimate prediction error, forcing a large correction constant and reducing adaptivity.Training residuals are optimization-biased, whereas calibration residuals are unbiased; overfitting can make this mismatch severe.

6 Experiments

The experiments evaluate conformalized quantile regression against standard and locally adaptive conformal methods across benchmark datasets. CQR generally produces shorter intervals while preserving nominal coverage through conformal calibration.

  • Methods compared: CQR is evaluated with quantile regression forests and neural networks, alongside standard and locally adaptive conformal prediction baselines.Unconformalized quantile regression methods are also included for comparison.
  • Experimental setup: The study averages performance over 2,200 experiments spanning eleven regression datasets and twenty training-test splits.Each split uses 80% of examples for training and 20% for testing, with equal-sized proper training and calibration sets for split conformal methods.
  • Overall results: CQR achieves shorter average prediction intervals than both standard and locally adaptive conformal prediction, and also outperforms unconformalized quantile regression.The comparison averages results across all datasets and training-test splits.
  • Coverage: All conformal methods attain the nominal 90% coverage rate, whereas unconformalized random-forest intervals slightly overcover and neural-network intervals tend to undercover.The reported volatility of unconformalized quantile-neural-network coverage highlights the finite-sample guarantee supplied by conformal prediction.
  • Quantile crossing: Post-processing quantile crossings reduces average interval length from 1.40 to 1.35 for CQR neural networks and from 1.49 to 1.41 for unconformalized quantile neural networks.Coverage rates remain about the same after crossings are eliminated.
  • Individual datasets: On ten out of eleven datasets, CQR outperforms both standard and locally adaptive conformal prediction.CQR random forests are overly conservative on the two Facebook datasets because conformity-score ties invalidate the cited upper-bound condition.

7 Conclusion

The paper concludes that conformalized quantile regression combines conformal prediction’s finite-sample validity with quantile regression’s adaptation to heteroskedasticity. It also points toward conformal predictive distributions as a related extension.

  • Conclusion: CQR controls miscoverage in finite samples under exchangeability while adapting interval lengths to heteroskedasticity.The method combines the advantages of conformal prediction and quantile regression.
  • Future direction: The ideas may extend to conformal predictive distributions, which estimate predictive probability distributions rather than only intervals.The paper notes connections to an independently written paper on conformal distributions.

A Lemmas about quantiles

This section establishes quantile facts for exchangeable samples and uses order-statistic symmetry to support finite-sample probability statements. It defines ordinary and right quantiles and derives the relevant empirical forms.

  • Quantile definitions: The quantile function satisfies Q(α) ≤ z if and only if α ≤ F(z), while the right quantile satisfies F^-(z) ≤ α if and only if z ≤ R(α).The explicit formulas are Q(α) = inf{z ∈ R : α ≤ F(z)} and R(α) = sup{z ∈ R : F^-(z) ≤ α}.
  • Empirical quantiles: The empirical quantile is ˆQ_n(α) = Z(⌈αn⌉), and the right empirical quantile is ˆR_n(α) = Z(⌊αn⌋+1).Here Z(k) denotes the kth smallest value among Z_1, …, Z_n.
  • Exchangeability lemma: For exchangeable random variables, empirical quantiles satisfy finite-sample probability bounds, with an additional distinctness condition yielding sharper statements.The probabilities are taken over all variables in the exchangeable sample.
  • Proof strategy: The proof uses exchangeability and symmetry to equate the probability for each sample index, then applies empirical-CDF inequalities.The argument uses the relationship between ordinary and right empirical quantiles and the bound on their empirical CDF difference.
  • Quantile inflation: The quantile-inflation lemma transfers the relevant order-statistic event between samples of sizes n and n+1.The key equivalence is Zn+1 ≤ Z(k,n) if and only if Zn+1 ≤ Z(k,n+1).

B Synthetic experiment

The synthetic experiment constructs heteroskedastic data with occasional large outliers and compares several 90% prediction-interval procedures. It illustrates locally adaptive conformal prediction using a conditional median estimator.

  • Data generation: The simulation draws 2,000 training predictors uniformly on [1, 5] and generates a 5,000-sample test set from the same model.The response combines a Poisson component, heteroskedastic Gaussian noise, and rare large outliers.
  • Data generation: The outlier term creates few but large outliers, which are displayed in the full-range scatter plot of the test data.Figure 7 provides the full-range view corresponding to the synthetic simulation of Figure 2.
  • Compared procedures: Figure 2 compares 90% intervals from split conformal prediction, locally adaptive split conformal prediction, and CQR.The data are heteroskedastic because the dispersion of Y varies considerably with X.
  • Compared procedures: Locally adaptive conformal prediction is implemented with a random-forest scale estimator, while the standard method uses a random-forest regressor for calibration.The experiment sets γ to zero and is reported as insensitive to that hyper-parameter.
  • Robustness variant: Using quantile regression forests to estimate the conditional median yields average test coverage of 90.14% and average interval length of 2.86.The target coverage is 90%.
Loading 1905.03222v1…