PaperlayerView exact revision on arXiv

Source-linked AI summary

Distribution-Free Predictive Inference For Regression

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, Larry Wasserman

arXiv:1604.04173v2stat.MEmath.STstat.ML

TL;DR

The paper addresses how to construct regression prediction bands without relying on distributional or model assumptions, particularly for high-dimensional regression. It develops conformal methods using general regression estimators and establishes finite-sample marginal coverage alongside oracle-style length guarantees under stability or consistency. The paper also compares computational and statistical variants and extends conformal inference to in-sample prediction, locally varying lengths, and model-free variable importance.

  • Problem

    Regression prediction sets, especially in high-dimensional settings, have limited existing coverage and often rely on strong assumptions about the model, estimator, or errors.

  • Method

    The paper develops full and split conformal inference around general regression estimators, with extensions for in-sample coverage, varying interval length, and LOCO variable importance.

  • Results

    Conformal bands have finite-sample validity without distributional assumptions and approach oracle or shortest valid super oracle bands when the estimator is stable or consistent.

  • Takeaways & Limitations

    Conformal inference supports valid prediction bands for high-dimensional regression methods even when the working model is misspecified, provided the data are i.i.d.

  • Takeaways & Limitations

    The guarantees described are marginal rather than finite-sample conditional coverage, which cannot generally be achieved with finite-length intervals without regularity and consistency assumptions.

Abstract

from arXiv · show

We develop a general framework for distribution-free predictive inference in regression, using conformal inference. The proposed methodology allows for the construction of a prediction band for the response variable using any estimator of the regression function. The resulting prediction band preserves the consistency properties of the original estimator under standard assumptions, while guaranteeing finite-sample marginal coverage even when these assumptions do not hold. We analyze and compare, both empirically and theoretically, the two major variants of our conformal framework: full conformal inference and split conformal inference, along with a related jackknife method. These methods offer different tradeoffs between statistical accuracy (length of resulting prediction intervals) and computational efficiency. As extensions, we develop a method for constructing valid in-sample prediction intervals called {\it rank-one-out} conformal inference, which has essentially the same computational efficiency as split conformal inference. We also describe an extension of our procedures for producing prediction bands with locally varying length, in order to adapt to heteroskedascity in the data. Finally, we propose a model-free notion of variable importance, called {\it leave-one-covariate-out} or LOCO inference. Accompanying this paper is an R package {\tt conformalInference} that implements all of the proposals we have introduced. In the spirit of reproducibility, all of our empirical results can also be easily (re)generated using this package.

1 Introduction

The paper develops conformal prediction for distribution-free, finite-sample-valid regression prediction bands, especially in high-dimensional settings where existing inference methods require strong assumptions. It also provides theoretical guarantees, empirical comparisons, and extensions for in-sample prediction, locally varying interval length, and model-free variable importance.

  • 1 Introduction: The framework constructs distribution-free prediction bands using essentially any regression estimator, including methods for high-dimensional, classical linear, and nonparametric regression.Its central goal is finite-sample validity without assumptions on the data distribution.
  • 1 Introduction: Conformal inference provides generic finite-sample prediction sets, with data augmentation making the procedure immune to overfitting.The framework targets valid average coverage for future responses without requiring the regression model to be correctly specified.
  • 1 Introduction: The paper theoretically studies finite-sample coverage accuracy and distribution-free asymptotic in-sample coverage, and empirically compares full conformal, split conformal, and jackknife methods.The accompanying conformalInference R package reproduces the simulations and implements the proposed methods and extensions.
  • 1 Introduction: If the base estimator is stable, conformal bands are close to an estimator-dependent oracle band; if it is consistent, they approach a shortest-length valid super oracle band.These results establish accuracy and near-optimal length under standard assumptions while preserving distribution-free validity more generally.
  • 1 Introduction: The extensions add valid in-sample prediction intervals, locally varying-length prediction bands, and model-free LOCO inference for covariate importance.LOCO means leave-one-covariate-out inference and is presented as a prediction-based approach to variable importance.

2 Conformal Inference

Conformal inference constructs distribution-free prediction bands with finite-sample marginal coverage under minimal assumptions, while offering full and split procedures with different computational and statistical tradeoffs.

  • Full conformal inference: Sample-quantile exchangeability yields finite-sample coverage for conformal prediction bands without assumptions on the data distribution or regression estimator.The standard validity argument uses the uniform rank of the new observation among exchangeable augmented residuals.
  • Full conformal inference: Conformal intervals are accurate as well as valid, with coverage bounded above by 1 − α + 1/(n + 1) under continuous residual distributions.Continuity is used to avoid ties; random tie-breaking can remove that assumption.
  • Full conformal inference: Improving the regression estimator generally shortens conformal intervals because more accurate fits produce smaller residuals.The interval length depends on the quality of the initial estimator even though coverage does not require estimator assumptions.
  • Coverage scope: The paper distinguishes valid marginal coverage from conditional coverage, which cannot generally be achieved by finite-length intervals without additional regularity and consistency assumptions.Conditional coverage may hold asymptotically under certain conditions, but it is substantially stronger than the paper’s finite-sample marginal guarantee.
  • Computational tradeoffs: Full conformal inference is computationally intensive because it repeatedly refits the estimator on augmented data for candidate feature-response pairs.Efficient full conformal inference remains open for high-dimensional regression using sophisticated nonlinear estimators such as the lasso.
  • Split conformal inference: Split conformal inference greatly reduces computational cost and memory requirements while retaining finite-sample validity and offering approximate in-sample coverage.Its coverage bound has the analogous upper correction 2/(n + 2) under continuous split-sample residuals.
  • Multiple splits: Multiple data splits reduce split-induced variability but can lengthen intervals because the Bonferroni effect dominates as the number of splits increases.The paper therefore recommends using a single split under the stated asymptotic conditions.
  • Jackknife inference: Jackknife intervals use more training data and can be shorter, but lack guaranteed finite-sample out-of-sample coverage and require nontrivial estimator conditions even asymptotically.The conformal method requires none of the uniform asymptotic error, stability, or standard linear-model conditions discussed for jackknife validity.

3 Statistical Accuracy

The paper evaluates conformal prediction-band length by comparing it with idealized oracle bands, showing that estimator consistency or stability yields oracle approximation and near-optimal coverage and length.

  • Oracle benchmarks: Conformal-band accuracy is assessed by comparing interval lengths with a super oracle that knows the regression function and error distribution, and a regular oracle that knows residual distributions.The super oracle provides the ideal benchmark, while the regular oracle accounts for estimation randomness.
  • Oracle benchmarks: The super oracle band uses the regression function plus the α upper quantile of absolute noise and has shortest length under conditional or marginal coverage.Its form is s(x) = [µ(x) − qα, µ(x) + qα].
  • Oracle approximation: If the base estimator is consistent, the regular and super oracle bands have similar lengths, with estimation error affecting oracle width only at second order.The width difference is proportional to E∆n(X)^2 under the theorem’s regularity conditions.
  • Oracle approximation: If the estimator is stable under resampling and perturbations, split and full conformal bands approach the regular oracle without requiring consistency for the true regression function.Sampling stability requires concentration around a potentially different function, rather than closeness to µ.
  • Limitations: When sampling stability fails, split conformal intervals can differ substantially from the oracle because fitted-model residuals may have a different distribution.This is an author-supported limitation of the split-conformal approximation result, not of its finite-sample coverage guarantee.

4 Empirical Study

The empirical study tests conformal intervals across low- and high-dimensional settings, misspecification, heavy tails, heteroskedasticity, correlated features, and multiple base estimators.

  • Overall findings: Conformal prediction bands achieve nearly exact marginal coverage even when the regression model is completely misspecified.The simulations use 90% nominal coverage, averaged over 50 repetitions.
  • Overall findings: In high-dimensional problems, conformal inference often produces much smaller bands than conventional parametric methods.This pattern is attributed to poor linear-regression estimation when the number of features is large.
  • Overall findings: Band length closely tracks the quality of the initial estimator across models and simulation settings.Intervals retain almost exactly 90% coverage across estimators, while lengths are highly correlated with test errors.
  • Low-dimensional comparisons: In low-dimensional misspecified settings, parametric intervals over-cover and become wider, while conformal intervals remain near the nominal 90% coverage.This occurs when linearity, normality, or homoskedasticity assumptions fail.
  • High-dimensional comparisons: In high-dimensional linear-regression settings, full conformal intervals outperform parametric intervals in both length and coverage, whereas jackknife intervals struggle.Split conformal is unavailable when n/2 = 250 and d = 490 because split-sample linear regression is not well-defined.
  • Regularization and computation: Ridge regularization improves every method, while conformal methods still outperform parametric intervals and split conformal offers similar performance at lower computational cost.The computational advantage makes split conformal compatible with expensive cross-validation or bootstrap estimators.

5 Extensions of Conformal Inference

The paper extends conformal inference to in-sample prediction, locally varying interval widths, and broader practical settings while retaining finite-sample guarantees and improving computational efficiency or adaptation to heteroskedasticity.

  • In-Sample Split Conformal Inference: Rank-one-out (ROO) split conformal inference constructs valid in-sample prediction intervals while requiring only two model fits and O(1) additional operations per observation.It addresses the computational and dependence problems of applying ordinary conformal methods separately at each observed feature value.
  • In-Sample Split Conformal Inference: ROO split conformal intervals have an in-sample finite-sample coverage property by symmetry, with empirical average coverage differing from 1 −α by order √(log n/n) with high probability.The theorem assumes i.i.d. observations; an additional continuous-residual assumption is stated for related results.
  • In-Sample Split Conformal Inference: A simpler conservative ROO approximation uses one residual quantile per split, widening each interval by at most one interquantile difference while changing a theorem term from 2/n to 6/n.This approximation preserves the stated theorem guarantees but produces wider intervals.
  • Locally-Weighted Conformal Inference: Locally weighted conformal bands scale residuals by an estimated conditional MAD, producing prediction intervals whose widths vary with the covariate value.The extension is designed to account for nonconstant residual variance, and the validity and accuracy properties of the split methods carry over.
  • Locally-Weighted Conformal Inference: In a heteroskedastic simulation, locally weighted bands had average length 1.105 versus 1.247 unweighted, with average coverage 89.9% versus 89.7%.The weighted band also maintained conditional coverage around the nominal 90% level more evenly across x.
  • Locally-Weighted Conformal Inference: When errors are homoskedastic, locally weighted conformal bands generally have inflated average length because estimating the conditional MAD adds randomness.Thus, local weighting is principally motivated by varying residual spread rather than constant-variance settings.

6 Model-Free Variable Importance: LOCO

LOCO inference measures variable importance through the increase in prediction error after removing a covariate, with conformal intervals providing finite-sample validity. Local and global examples show how these intervals identify predictive variables while preserving stated coverage guarantees.

  • Local prediction bands: 0.899 coverage and 1.105 average length were obtained for locally-weighted split conformal bands, versus 0.897 coverage and 1.247 average length for unweighted bands.The locally-weighted band maintains roughly constant local coverage while varying its length across x.
  • Definition: LOCO measures a covariate’s importance by the increase in prediction error after refitting the model without that covariate.The resulting excess-error variable is the basis for inferential statements about importance.
  • Validity: Conformal LOCO intervals have finite-sample validity without distributional assumptions and are uniformly valid across covariates without multiplicity adjustment.The guarantee applies when intervals are constructed for a fixed, all, or data-selected covariate.
  • Local variable importance: In the additive-model example, intervals for covariates 1–3 often lay above zero where their underlying components deviated from zero, while intervals for covariates 4–6 contained zero.The intervals were computed with ROO split conformal inference at α = 0.1.
  • Global variable importance: Mean-based global inference requires first and second error moments, whereas sign and Wilcoxon procedures provide finite-sample inference under weaker distributional conditions.The sign test requires continuity, while the Wilcoxon test requires continuity and symmetry.
  • Global variable importance: In the high-dimensional example, intervals for the first 5 variables were above zero, while those for the next 12 hovered around zero after multiplicity correction.The intervals targeted median excess test error due to variable dropping and had simultaneous coverage level 0.9.

7 Conclusion

The conclusion emphasizes that conformal inference can turn broad classes of regression estimators into valid prediction bands under model misspecification. It also identifies marginal coverage, split-related randomness, and selective-inference issues as important boundaries and directions for further work.

  • Conclusion: Conformal prediction bands can use nearly any point estimator and retain valid marginal coverage even under gross model misspecification, provided the data are i.i.d.When the estimator is accurate, the resulting band can be near-optimal; otherwise, marginal validity remains.
  • Conclusion: Conformal coverage is marginal, so conditional coverage may exceed 1 − α for some covariate values and fall below it for others.The statistical accuracy of the band remains strongly tied to the base estimator.
  • Open directions: The split method is fast but less accurate, and aggregating multiple splits with a Bonferroni-type correction produces wider bands.Reducing splitting-induced randomness remains an open practical problem.
  • Open directions: LOCO is a first step toward model-free variable selection, but its split-conformal implementation is conditional on the first-half fitted model and raises selective-inference issues.The authors identify detailed comparisons with other high-dimensional inference approaches as future work.

A.1 Proofs for Section 2

These proofs establish finite-sample and asymptotic properties of the conformal procedures through exchangeability, empirical-quantile arguments, and concentration bounds. They also account for split aggregation and rank-one-out adjustments.

  • Finite-sample coverage: The finite-sample coverage proofs for full and split conformal inference rely on symmetry of the test residual with calibration residuals.The arguments also use empirical quantiles of fitted absolute residuals.
  • Multiple splits: Aggregating N split bands requires a Bonferroni-adjusted level α/N, with the proof controlling the probability of simultaneous high-probability events.The analysis explicitly considers the range 2 ≤ N ≤ αn.
  • Length bounds: The split-band length analysis controls empirical residual quantiles using the Dvoretzky–Kiefer–Wolfowitz inequality.The proof compares calibration residual distributions with the target residual distribution.
  • ROO adjustment: The ROO proof matches its effective quantile threshold to split conformal by shifting the target coverage from 1 − α to 1 − (α + 2α/n).The result follows by applying the split-conformal theorem after this coverage adjustment.

A.2 Proofs for Section 3

The proofs for Section 3 analyze how conformal interval endpoints and lengths relate to oracle residual quantiles. They use estimator-error control, density assumptions near target quantiles, and symmetry conditions to derive the stated guarantees.

  • Assumptions: The asymptotic arguments require regularity such as a density bounded below near the target quantile and, for symmetry-based results, symmetry of the noise distribution.These assumptions enter through Taylor expansion and the relation f_0(t) = f_0(−t).
  • Split conformal: The split-conformal proof bounds coverage by comparing fitted residual quantiles with residuals from the target regression function.The argument combines empirical CDF control with quantile regularity near the target quantile.
  • Full conformal: The full-conformal proof shows inclusion below one residual threshold and exclusion above another, with both thresholds incorporating estimator-error terms.The resulting interval length is then controlled using the empirical residual distribution.
  • Asymptotic behavior: Under the stated estimator-consistency event, the lower and upper endpoints of the full-conformal interval are q_α + o_P(1) below and above μ(X), respectively.The proof uses concentration and empirical-quantile arguments to show the relevant index sets have asymptotically appropriate sizes.

A.3 Proofs for Section 5

The proof bounds rank-one-out conformal coverage by analyzing how changing one observation affects inclusion indicators and residual rankings.

  • The proof compares residual order statistics before and after replacing one sample point to control these indicator changes.For indices below m or above m + 1, the inclusion indicators remain unchanged under the stated ordering.
  • Changing one data point can alter rank-one-out inclusion indicators for at most three observations.The affected indices are m, m + 1, and the changed point j.
  • McDiarmid’s inequality converts the bounded influence of each independent sample point into an exponential deviation bound.The argument uses independence of the input sample points after establishing the three-indicator sensitivity bound.
  • The proof obtains the corresponding result for the other data split by switching the two index sets and adding the inequalities.
  • The lower-bound argument applies the standard conformal ranking argument to residuals excluding the held-out observation.Under continuity, residuals are distinct with probability one, enabling the ranking comparison.
  • For the extended set, the lower bound follows from containment, while the upper-bound ranking adjustment changes a probability term from 2/n to 6/n.

B Additional Experiments

Additional experiments test conformal intervals in nonsparse high-dimensional regression and locally weighted bands without heteroskedasticity.

  • Figure 6: In the nonsparse high-dimensional setting, all conformal bands achieve approximately 90% average coverage despite poor test error.The simulation uses n = 200, d = 2000, and a mean function involving s = 100 active variables.
  • Figure 7: The locally weighted method produces mildly varying local lengths and essentially constant local coverage without heteroskedastic noise.
  • Figure 7: The locally weighted method’s average length is not much worse than that of the usual unweighted conformal method.
  • Figure 7: 0.903 coverage and 3.287 average length are reported for split conformal prediction intervals.
  • Figure 7: 0.905 coverage and 3.360 average length are reported for locally weighted split conformal intervals.
Loading 1604.04173v2…