Source-linked AI summary
Conformal prediction for time series
Chen Xu, Yao Xie
TL;DR
Time-series prediction needs uncertainty intervals despite complex dependencies, while exchangeability-based conformal prediction is not directly applicable to such data. The paper bounds coverage gaps and introduces EnbPI, an ensemble wrapper; its intervals are asymptotically valid without exchangeability and converge in width to the oracle.
Problem
Time-series prediction needs uncertainty intervals despite complex dependencies, while exchangeability-based conformal prediction is not directly applicable to such data.
Source: §1 · p. 1Method
The paper bounds conditional and marginal coverage gaps and oracle-interval set differences, and introduces EnbPI as an ensemble-based wrapper for prediction algorithms.
Source: §1.1 · p. 2, §7 · p. 24Results
EnbPI's intervals are asymptotically valid without exchangeability, converge in width to the oracle, and empirically maintain validity where competing methods fail.
Source: §7 · p. 24Takeaways & Limitations
The paper notes that EnbPI can also be used for unsupervised sequential anomaly detection.
Source: §7 · p. 24At least 1−α coverage requires incorporating the theorem's upper bounds, but EnbPI does not do so.
Source: §3.1 · p. 12
Abstract
from arXiv · showhide
We develop a general framework for constructing distribution-free prediction intervals for time series. Theoretically, we establish explicit bounds on conditional and marginal coverage gaps of estimated prediction intervals, which asymptotically converge to zero under additional assumptions. We obtain similar bounds on the size of set differences between oracle and estimated prediction intervals. Methodologically, we introduce a computationally efficient algorithm called \texttt{EnbPI} that wraps around ensemble predictors, which is closely related to conformal prediction (CP) but does not require data exchangeability. \texttt{EnbPI} avoids data-splitting and is computationally efficient by avoiding retraining and thus scalable to sequentially producing prediction intervals. We perform extensive simulation and real-data analyses to demonstrate its effectiveness compared with existing methods. We also discuss the extension of \texttt{EnbPI} on various other applications.
1 Introduction
The introduction motivates distribution-free time-series prediction intervals with coverage guarantees and presents EnbPI alongside theoretical and empirical contributions.
1 Introduction
- Time-series intervals matter in high-stakes settings, yet neural prediction intervals may lack theoretical coverage guarantees and general efficient methods for complex models remain under-explored.The introduction gives renewable-energy prediction as an example.Source: §1 · pp. 1–2
1.1 Contributions
- Conditional and marginal coverage gaps, and the set difference from oracle intervals, have bounds that converge to zero under mild assumptions on error dependence and estimation quality.The framework allows asymmetric intervals.Source: §1.1 · p. 2
- EnbPI wraps ensemble estimators without data splitting or expensive retraining, using a bootstrap procedure to construct prediction intervals.The paper identifies network prediction and anomaly detection as potential applications.Source: §1.1 · p. 2
- EnbPI maintains target coverage when competing methods fail, can yield shorter intervals, and is robust to missing data in simulations and real-time-series experiments.The experiments cover simulated and real-time-series data.Source: §1.1 · p. 3
1.2 Literature review
- Conformal prediction relies on exchangeability for marginal coverage, an assumption described as hardly reasonable for time series [Shafer and Vovk, 2008]; neural quantile methods lack coverage guarantees [Wen et al., 2017, Salinas et al., 2020].The cited neural methods also require specialized architectures [Wen et al., 2017, Salinas et al., 2020].Source: §1.2 · pp. 3–4
- The theoretical analysis also applies to other conformal methods, including split conformal and J+aB, providing tools for analyzing conformal prediction in time series.The paper distinguishes its time-series-specific bounds from existing approaches.Source: §1.2 · p. 5
2 Problem setup
The setup specifies a time-series prediction task and an oracle target, then motivates estimated intervals built from leave-one-out residuals to balance fitting and calibration.
2 Problem setup
- The model writes each response as f(X_t) plus error; the errors need not be independent, and their continuous CDF may vary over time.Features may include exogenous sequences or response history.Source: §2 · p. 5
2.1 Oracle prediction interval
- The oracle interval has exact conditional coverage and the shortest width among conditionally valid intervals, assuming perfect knowledge of f and the conditional response distribution.The oracle therefore serves as a benchmark for estimated intervals.Source: §2.1 · p. 6
2.2 Proposed prediction interval
- The proposed interval is centered at the leave-one-out prediction, with width set by two quantiles of the past T leave-one-out residuals.The quantile levels are β̂ and 1−α+β̂.Source: §2.2 · p. 7
- Data splitting trades model-training data against residual calibration quality; leave-one-out residuals use all other training points for each fit, but ordinary leave-one-out retrains repeatedly.EnbPI addresses the retraining cost by constructing leave-one-out estimators from pretrained ensemble models.Source: §2.2 · p. 7
3 Theoretical analysis
The section derives coverage and interval-width guarantees for time-series prediction intervals without requiring data exchangeability. The bounds apply beyond EnbPI and vanish asymptotically under assumptions on error dependence and estimation quality.
3 Theoretical analysis
- The framework bounds conditional and marginal coverage gaps and the set difference between estimated and oracle intervals, with results extending to conformal methods beyond EnbPI.The analysis removes exchangeability assumptions by imposing conditions on the error process and estimator quality.Source: §3 · pp. 7–8, §3.2 · p. 12
3.1 Coverage guarantees
- Conditional coverage gaps converge to zero when the estimation-error term decays, with extensions for linear-process and strongly mixing errors.The section also gives a marginal coverage-gap bound; the strongly mixing result assumes stationary errors with summable mixing coefficients.Source: §3.1 · pp. 10–11
3.2 Width guarantees
- The estimated-versus-oracle interval set-difference is bounded under regularity conditions, with rates adjusted for linear-process or strongly mixing errors.The bounds apply to predictions beyond the first step when the assumptions hold at the relevant indices; split conformal also inherits the results, but its calibration sample size replaces the full training size.Source: §3.2 · pp. 12–13
4 EnbPI Algorithm
EnbPI builds sequential prediction intervals from bootstrap ensembles and past leave-one-out residuals, updating those residuals as outcomes arrive. It avoids model retraining during deployment while supporting adaptations for missing data and anomaly detection.
4 EnbPI Algorithm
- EnbPI aggregates bootstrap predictors into leave-one-out predictors and residuals, then uses past residuals to set interval widths and slides them forward as outcomes arrive.A one-dimensional line search optimizes interval width.Source: §4 · p. 13
4.1 Properties of EnbPI
- Pre-trained ensemble models avoid retraining for each interval; unlike Jackknife+, EnbPI calls the prediction algorithm B times rather than requiring BT model trainings.EnbPI stores its models once and computes residuals from the pre-trained models during prediction.Source: §4.1 · p. 14
- EnbPI avoids data splitting by aggregating bootstrap models not trained on each datum, while a Chernoff bound ensures each leave-one-out ensemble has a balanced model count.The number of models aggregated for each predictor is binomial with parameters B and (1 − 1/T)^T.Source: §4.1 · p. 15
- Without retraining, EnbPI updates a sliding window of past residuals from feedback to adapt interval widths to non-stationarity, trading computation against potentially narrower intervals from retraining.For missing test features, it imputes the feature to compute the interval center; unobserved responses are skipped in the residual window.Source: §4.1 · p. 15, §4.2 · p. 15
4.2 EnbPI on challenging tasks
- EnbPI is presented as flexible for challenging tasks, including a heuristic for missing data and use of prediction intervals to detect anomalous observations.The paper points to a solar dataset with missing data and an example of anomalous traffic-flow detection.Source: §4.2 · p. 15
5 Experimental results
Across simulations and real-data experiments, EnbPI’s interval coverage and width are evaluated, including marginal and conditional coverage and traffic anomaly detection. The experiments report valid coverage in several settings, while identifying poorer sliding coverage near summer shifts.
5 Experimental results
- The experiments cover simulations, real-data marginal and conditional coverage, and traffic-flow anomaly detection.The paper also describes additional time-series examples in the appendix.Source: §5 · p. 16
5.1 Simulation results
- In simulations, EnbPI maintains valid coverage and its interval widths converge to the oracle width as training size grows.For the noisy helix example, test intervals tightly cover responses, and the LOO ensemble predictions closely track the true model.Source: §5.1 · p. 16, Figure 1 · p. 17
5.2 Real-data: Marginal validity and the interval width
- On Atlanta solar data, EnbPI consistently reaches 0.9 coverage across training-data amounts; baselines may be narrower but miss target coverage, and time-series baselines are either conservative or under-cover.At similar interval widths, baselines often still fail to reach valid coverage; EnbPI also has the smallest width standard deviation.Source: Table 1 · p. 18, §5.2 · p. 18, Table 2 · p. 19
5.3 Real-data: Missing data, conditional coverage
- With 25% missing data and five-hour-ahead intervals, EnbPI nearly obtains conditional coverage at each hour, but sliding coverage can worsen near summer data shifts.The authors report that fitting separate morning and afternoon groups can be essential when their data-generating processes are heterogeneous.Source: §5.3 · p. 20, Figure 2 · p. 21
5.4 Real data: Unsupervised anomaly detection
- EnbPI under NN or RNN consistently outperforms competitors in traffic anomaly detection by F1 score, across training sizes and selected sensors.Its F1 scores are also consistent across training sample sizes when RNN is used.Source: §5.4 · p. 21, Table 4 · p. 22
6 EnbPI under change points
EnbPI intervals cover most pre- and post-change observations outside the retraining window, while pre-change models yield wider post-change intervals. Interval width and coverage may also help identify changes.
6 EnbPI under change points
- The simulation changes the underlying model for the final 40% of test data, with distinct post-change specifications for linear, sparse linear, and nonlinear cases.The post-change model is assumed to be known in this setup.Source: §6 · pp. 22–23
- The procedure assumes the change point is known, retrains on 0.1T post-change data, and computes residual quantiles from the most recent T′=100 residuals.The authors note that a change-point detection algorithm could estimate the change point.Source: §6 · p. 23
- Most observations outside indices 50–110 are covered by EnbPI intervals; pre-change models produce wider intervals on post-change data, while AdaptCI fails before retraining.The simulation collects 60 post-change points before refitting at index 110.Source: §6 · p. 23, Figure 4 · p. 23
- Monitoring interval width together with coverage may help detect shifts: model changes can widen intervals and worsen coverage, whereas error-distribution changes may widen intervals with less coverage impact.A sliding window over residuals can adapt to the post-change distribution.Source: §6 · p. 24
7 Conclusions and Discussions
The paper presents theoretical guarantees and EnbPI, an ensemble-based method for time-series prediction intervals without exchangeability. It reports empirical validity across several settings and identifies further applications for future work.
7 Conclusions and Discussions
- The framework bounds conditional and marginal coverage gaps and oracle-versus-estimated interval set differences, with asymptotic convergence under additional assumptions.The bounds depend on estimation quality and error-process assumptions.Source: §7 · p. 24
- EnbPI wraps ensemble predictors to produce intervals without data splitting or retraining, and supports sequential anomaly detection.The paper reports empirical validity even under potentially misspecified models, while its theoretical guarantee requires consistent estimation of the true model.Source: §7 · p. 24
- The authors report that EnbPI works well on time series from varied applications, including network data and data with missing entries.They report maintained validity in settings where other predictive-inference methods fail.Source: §7 · p. 24
- Future directions include adapting EnbPI to classification with non-residual conformity scores and developing online change-point detection and adaptation for time series.The latter direction extends sequential testing ideas based on the Shiryaev–Roberts procedure.Source: §7 · p. 24
A Proofs
The proofs derive coverage and interval-width guarantees by controlling empirical-distribution errors and decomposing the width discrepancy into estimable terms.
A Proofs
- For dependent errors with summable mixing coefficients, the proof bounds empirical-CDF deviations on a high-probability event.The stated event controls the deviation by a term proportional to (log T)^2/3 divided by T^1/3, with a constant involving the mixing-coefficient sum.Source: A Proofs · p. 34
- The width proof defines a Lipschitz-continuous inverse empirical CDF, then decomposes the width discrepancy into terms and bounds each separately.The decomposition uses line-search error, empirical-CDF error, and the difference between estimated and target β.Source: A Proofs · pp. 34–35
- Interpolating the inverse empirical CDF between multiples of T^-1 makes it continuous, and its Lipschitz property supports the subsequent bounds.The interpolation uses the neighboring grid values around α.Source: A Proofs · p. 35
- The width discrepancy separates into line-search approximation, empirical-CDF approximation, and β-estimation terms, each controlled using the corresponding continuity or coverage bound.The proof bounds the line-search term through accurate binning and Lipschitz continuity, and invokes Theorem 1 for the empirical-CDF term.Source: A Proofs · pp. 35–36
B More experiments
The appendix adds experiments on further datasets and reports that EnbPI rarely loses marginal validity while producing shorter intervals than competing methods in those tests.
B More experiments
- The appendix tests California solar and Austin wind data, including multi-step-ahead inference when missing data are present.It first examines interval validity before the missing-data multi-step task.Source: B More experiments · p. 37
- Across additional datasets, EnbPI rarely loses marginal validity and can produce shorter intervals than competing methods.These tests include greenhouse-gas emissions, air pollution, and appliances energy data.Source: B More experiments · p. 37
- The simulation figure displays marginal coverage and width across training sample sizes, plus conditional results for case 3 at the first test point.Its second row overlays prediction intervals, actual responses, and point estimates.Source: Figure 5 · p. 38
B.1 Simulated examples
The simulations test EnbPI across increasingly sophisticated prediction models and report marginal and conditional coverage alongside interval width. Marginal coverage is nearly valid across sample sizes, conditional validity is attained, and interval width approaches the oracle width as training size grows.
Setup
- The simulations use 2,000 observations, split evenly between training leave-one-out ensemble predictors and testing.The response is modeled as Y_t = f(X_t) + ϵ_t.Source: §B.1 · p. 38
- The three cases increase model sophistication, including a linear model, a high-dimensional sparse linear model, and a nonlinear model.The linear case uses skewed-normal errors; the high-dimensional case uses past responses as features; the third case uses an autoregressive error process.Source: §B.1 · pp. 38–39
- EnbPI uses 50 bagging estimators, mean aggregation, and α = 0.05, with linear regression, lasso, or a neural network as prediction algorithms.The prediction interval is slid after each prediction, and predictor settings were not optimized.Source: §B.1 · p. 39
Results
- Marginal coverage is almost always valid at 1−α across sample sizes, conditional coverage attains validity, and interval width converges to oracle width as training size grows.The simulations evaluate marginal and conditional coverage and interval width across three model settings.Source: §B.1 · p. 39
B.2 Setup detail for real-data experiments
The real-data experiments compare EnbPI with time-series and conformal prediction baselines using specified regression and neural-network configurations. The appendix also describes how several competing methods are implemented.
Competing methods
- The comparison includes ARIMA, exponential smoothing, dynamic factor models, split and weighted split conformal prediction, QOOB, AdaptCI, and jackknife+-after-bootstrap.The time-series baselines use default ARIMA settings and, for two methods, damped trends and daily seasonal cycles.Source: §B.2 · p. 40
- WeightedICP estimates distribution-shift weights with logistic regression, while QOOB uses quantile bootstrap estimators and a quantile-based nonconformity score.AdaptCI updates its significance level online and uses a quantile linear model or quantile random forest.Source: §B.2 · p. 40
Regression models
- The baseline regression specifications include ridge regression with generalized cross-validation and random forests with ten trees splitting features rather than samples.The ridge penalty is selected over ten grid points from 0.0001 to 10.Source: §B.2 · p. 40
Neural-network models
- The neural-network baselines specify three-layer feedforward and two-layer LSTM architectures, each with 100 hidden units per layer and Adam optimization.The feedforward network uses dropout after its second layer; both specifications use early stopping.Source: §B.2 · pp. 40–41
B.3 Additional results on solar—Atlanta
Additional Atlanta solar experiments examine changepoints, missing feedback, and multi-step comparisons with QOOB and AdaptCI. Without feedback, conditional coverage decreases even when data are complete and the prediction models are unchanged.
Multi-step ahead inference when s = ∞
- Conditional coverage is poor around changepoints when intervals are not slid, even with no missing data and the same training data.The data near summer have radiation levels very different from the training data.Source: §B.3 · p. 41
Conditional coverage of QOOB and AdaptCI
- Figure 8 shows the detailed hourly conditional coverage of QOOB and AdaptCI, while Table 5 presents a summary of that same conditional-coverage result.Source: §B.3 · p. 41
B.4 Results on solar—California and wind—Austin
The section extends EnbPI to network prediction and reports results on California solar and Austin wind data, including missing-data settings. EnbPI generally resembles its Atlanta results, while comparisons with other methods show differences in coverage and interval performance.
Multi-step comparison
- For multi-step Atlanta forecasts, QOOB and AdaptCI either fail to maintain coverage validity or produce slightly wider intervals than EnbPI.Figure 8 shows their hourly coverage, including coverage over a 30-day sliding window.Source: Table 5 · p. 41, Figure 8 · p. 43
Additional Atlanta results
- On Atlanta solar data without feedback, conditional coverage decreases even with no missing data; this illustrates the need to update past residuals using new feedback.Source: Figure 7 · p. 42
Network prediction
- Network prediction incorporates neighboring nodes’ current and earlier features, and the stated theoretical guarantees hold at each node when EnbPI is applied there.The approach performs nodal prediction using history and neighboring information.Source: §B.4 · p. 42
Overall pattern
- Results on California solar and Austin wind data are generally similar to Atlanta, with the wind analysis focused on multi-step inference with missing data.Source: §B.4 · pp. 42, 44
Marginal validity and interval width
- EnbPI performs similarly to traditional time-series methods, and its ridge-regression Winkler score can be the smallest, while ICP and WeightedICP can lose coverage, especially with multivariate features.The Winkler score is used to assess the balance between validity and efficiency.Source: §B.4 · pp. 42, 44
Missing data, conditional coverage
- EnbPI attains valid conditional coverage on California solar data near noon and on Austin wind data, including when data are missing.For wind, the model uses past response history and trains on all 24 hourly observations because the data do not show clear hourly differences.Source: §B.4 · p. 44
B.5 Results on datasets in other domains
Across three additional datasets, EnbPI’s coverage and interval-width performance varies by dataset and prediction model, while competing conformal methods can lose coverage. The appendix also describes applying EnbPI to traffic-flow anomaly detection and comparing it with eight anomaly detectors.
B.5 Results on datasets in other domains
- On Greenhouse, EnbPI almost never loses coverage, whereas ICP and WeightedICP can significantly under-cover, and EnbPI’s coverage and widths vary less.For the average coverage and width plots, EnbPI with RNN tends to have better coverage and much shorter intervals than other methods, though most methods lose coverage except Dynamic Factor.Source: §B.5 · pp. 44, 46, 47, Figure 10 · p. 48
- On Appliances Energy, ICP and WeightedICP can lose coverage and have wider, more variable intervals; EnbPI with RNN gives the shortest multivariate intervals without coverage loss.With univariate inputs, EnbPI almost always maintains coverage and produces much shorter intervals than time-series methods.Source: §B.5 · pp. 46–47, Figure 10 · p. 48, Figure 11 · p. 49
- On Beijing Air, the three conformal prediction methods perform similarly overall, although ICP and WeightedICP may greatly lose coverage; EnbPI with RF or Ridge can yield shorter univariate intervals with exact coverage.Source: §B.5 · pp. 46–47, Figure 10 · p. 48, Figure 11 · p. 49
- Across the additional datasets, EnbPI is described as stable across prediction algorithms and datasets, while ICP and WeightedICP can severely lose coverage; univariate intervals tend to be shorter.The grouped boxplots summarize 10 independent trials.Source: §B.5 · p. 47, Figure 11 · p. 49
B.6 Details on unsupervised anomaly detection
- The traffic-flow anomaly task labels observations unusually large or small relative to neighboring sensors and/or past values, and the EnbPI setup uses imputed data with 30% missing entries per sensor column.It forms predictors from past flows at eight nearby sensors and flags test points when residuals cross specified quantiles.Source: §B.6 · pp. 47, 50
- The anomaly-detection comparison evaluates EnbPI against eight detectors: four unsupervised methods and four supervised classifiers, using precision, recall, and F1 at each sensor.EnbPI uses 15 pretrained bootstrap models, significance level α = 0.05, and mean aggregation for leave-one-out ensemble predictors.Source: §B.6 · p. 50