Source-linked AI summary
Adaptive, Distribution-Free Prediction Intervals for Deep Networks
Danijel Kivaranovic, Kory D. Johnson, Hannes Leeb
TL;DR
Research on uncertainty in neural-network predictions for new samples remains limited. This paper pairs three-output networks with prediction-interval procedures, reporting coverage guarantees and improvements in interval length and predictive accuracy under an i.i.d. assumption.
Problem
Research on uncertainty in neural-network predictions for new samples is limited.
Source: §1 · p. 1Method
A three-output network uses a modified quantile-regression loss, with split conformal inference for average coverage and PAV for approximate conditional coverage.
Source: §1 · p. 1, §2.1 · p. 3, §2.3 · p. 5Results
The two procedures have provable coverage guarantees and improve current state-of-the-art methods in interval length and predictive accuracy.
Source: §4 · p. 7Takeaways & Limitations
The procedures' ease and transparency support their application to deep networks to express prediction uncertainty.
Source: §4 · p. 7The guarantees rely on i.i.d. data, and the intervals are generally not valid for out-of-distribution samples.
Source: §1.1 · p. 2, §2.3 · p. 5
Abstract
from arXiv · showhide
The machine learning literature contains several constructions for prediction intervals that are intuitively reasonable but ultimately ad-hoc in that they do not come with provable performance guarantees. We present methods from the statistics literature that can be used efficiently with neural networks under minimal assumptions with guaranteed performance. We propose a neural network that outputs three values instead of a single point estimate and optimizes a loss function motivated by the standard quantile regression loss. We provide two prediction interval methods with finite sample coverage guarantees solely under the assumption that the observations are independent and identically distributed. The first method leverages the conformal inference framework and provides average coverage. The second method provides a new, stronger guarantee by conditioning on the observed data. Lastly, our loss function does not compromise the predictive accuracy of the network like other prediction interval methods. We demonstrate the ease of use of our procedures as well as its improvements over other methods on both simulated and real data. As most deep networks can easily be modified by our method to output predictions with valid prediction intervals, its use should become standard practice, much like reporting standard errors along with mean estimates.
1 Introduction
The paper develops neural-network prediction intervals with coverage guarantees to address the limited uncertainty research and shortcomings of existing approaches. It combines quantile-based interval estimation with sample-splitting procedures and flexible network architectures.
1 Introduction
- The proposed procedures are evaluated as improving on competing methods, which may fail coverage, compromise predictive accuracy, or miss heteroskedasticity.The procedures use sample splitting and provide finite-sample coverage guarantees.Source: §1 · pp. 1–2
1.1 Prediction Intervals
- The paper proposes average-coverage intervals and Probably Approximately Valid (PAV) intervals, whose conditional coverage given the observed data is at least 1−α−ϵ for all but δ100% of data sets.PAV intervals can also control average coverage with appropriate choices of α, ϵ, and δ.Source: §1.1 · p. 2
1.2 The PI-specific loss function
- A three-output network estimates lower and upper interval bounds and a midpoint using a modified quantile-regression loss, but its quantile estimates are not known to be consistent.Finite-sample coverage guarantees are supplied by subsequent sample-splitting modifications.Source: §1.1 · p. 2, §1.2 · p. 3
1.3 Comments on the network architecture
- Users can retain their preferred network architecture or use a pretrained network, while ordering the midpoint and bounds avoids crossing quantiles and keeps predictions inside intervals.The paper also allows changing the midpoint target; under highly skewed distributions, the conditional mean is not guaranteed to lie between the interval quantiles.Source: §1.3 · p. 3
2 Construction of valid PIs with neural networks
The construction splits data between fitting neural-network prediction intervals and calibrating them, yielding average-coverage split-conformal intervals and approximately conditionally valid PAV intervals. The approach estimates quantiles directly rather than adjusting residual-based intervals, and its guarantees rely on i.i.d. data.
2 Construction of valid PIs with neural networks
- The method fits the network on D1 and uses the disjoint D2 to adjust interval widths for coverage.Its calibration score is tailored to the network’s ordered triple, unlike residual-based conformal methods.Source: §2 · p. 3, §2.1 · p. 4
2.1 Average Coverage
- Split-conformal calibration gives the neural-network intervals at least 1 − α coverage for a new observation.The calibration uses the holdout-set order statistic of scores; under the stated tie condition or random tie-breaking, coverage is at most 1 − α + 1/(|D1|+1).Source: §2.1 · p. 4
2.2 PAV
- For PAV intervals, holdout empirical coverage controls deviations from conditional coverage with an exp(−2ϵ^2n2) bound.The construction selects a network from a grid using D2; the grid may depend on data through D1.Source: §2.2 · pp. 4–5
- Using a more conservative level α − ϵ lets the PAV intervals also control average coverage.The stated lower bound is (1 − α + ϵ/2)(1 − K exp(−ϵ^2n2/2)).Source: §2.2 · p. 5
2.3 Discussion of contributions
- Directly estimating quantiles with neural networks avoids relying on residual-based intervals that may not suit skewed or multimodal distributions.The paper’s approximate conditional guarantee conditions on observed data, not input X; its crucial assumption is i.i.d. data, so the intervals are generally not valid for out-of-distribution samples.Source: §2.3 · p. 5
3 Simulations
On real datasets, conf-nn and pav achieve near-nominal average coverage and are among the shortest valid intervals, while their estimation error is similar to conf-fw. In simulations, they provide near-nominal coverage across most of the input domain, whereas unadjusted intervals undercover and other methods fail to capture heteroskedasticity.
3 Simulations
- Experiments used training, validation, and test splits; all data experiments set α = .1 and evaluate on data unseen by the models.Source: §3 · pp. 5–6
3.1 Artificial data
- In simulations, conf-nn and pav provide coverage close to nominal across most of β′X, but intervals are too wide at small β′X where few observations are available.Source: §3.1 · p. 6
- In simulations, conf-fw and high-q over-cover in the left tail and under-cover in the right, while qreg-un under-covers throughout the input space.The authors attribute these patterns to failure to capture heteroskedasticity and the need for calibration, respectively.Source: §3.1 · p. 6
3.2 Real Data
- Conf-nn and pav provide nearly nominal average coverage across repetitions and datasets and are among the shortest valid intervals.Pav is more conservative, consistent with its stronger guarantee, though the coarse selection grid for τ may also contribute.Source: §3.2 · p. 7
- Conf-nn, pav, and conf-fw have similar estimation error across datasets, indicating the proposed loss does not compromise predictive accuracy.Source: §3.2 · p. 7
4 Conclusion
The conclusion presents sample splitting and quantile regression as practical tools for fitting neural networks with accurate predictions and valid intervals. The authors report improvements in interval length and predictive accuracy and advocate broader use of the procedures.
4 Conclusion
- Sample splitting and quantile regression are used to fit neural networks that produce accurate predictions and valid prediction intervals.Source: §4 · p. 7
- The two procedures are reported to improve on current state-of-the-art methods in interval length and predictive accuracy.Source: §4 · p. 7
- The authors argue that larger datasets reduce the cost of using a validation set and support applying these procedures more broadly.Source: §4 · p. 7