PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequential Predictive Conformal Inference for Time Series

Chen Xu, Yao Xie

arXiv:2212.03463v3stat.MLcs.LGstat.ME

TL;DR

Prediction intervals for sequential data are important, but time-series dependence makes many existing conformal methods inapplicable. SPCI adaptively predicts future residual quantiles from past residuals, with asymptotic conditional coverage and narrower real-data intervals without coverage loss.

  • Problem

    Classical prediction-interval methods often impose strong parametric assumptions, while time-series data are non-exchangeable and many existing conformal methods do not apply.

    Source: §1 · p. 1

  • Method

    SPCI uses quantile regression on past prediction residuals to adaptively estimate future residual quantiles for sequential prediction intervals.

    Source: §1 · p. 2, §2 · p. 4

  • Results

    SPCI achieves asymptotic valid conditional coverage for dependent data and significantly narrower real-data intervals without coverage loss.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SPCI is a model-free, distribution-free framework applicable to time series with arbitrary predictive models.

    Source: §6 · p. 9

    The condition (9) comparison method assumes a particular linear form for Y_t|X_t and consistent estimation as sample size grows.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

We present a new distribution-free conformal prediction algorithm for sequential data (e.g., time series), called the \textit{sequential predictive conformal inference} (\texttt{SPCI}). We specifically account for the nature that time series data are non-exchangeable, and thus many existing conformal prediction algorithms are not applicable. The main idea is to adaptively re-estimate the conditional quantile of non-conformity scores (e.g., prediction residuals), upon exploiting the temporal dependence among them. More precisely, we cast the problem of conformal prediction interval as predicting the quantile of a future residual, given a user-specified point prediction algorithm. Theoretically, we establish asymptotic valid conditional coverage upon extending consistency analyses in quantile regression. Using simulation and real-data experiments, we demonstrate a significant reduction in interval width of \texttt{SPCI} compared to other existing methods under the desired empirical coverage.

1. Introduction

The paper introduces SPCI, which uses feedback from sequential prediction residuals to adaptively estimate their future conditional quantiles and construct prediction intervals. It addresses the limits of exchangeability-based conformal prediction for time series, with theoretical coverage guarantees and improved empirical performance reported against existing methods.

1. Introduction

  • Time-series data challenge conformal prediction because they are non-exchangeable, while classical prediction intervals often rely on strong time-series distribution assumptions.The paper also motivates principled uncertainty quantification for complex models such as random forests and neural networks.Source: §1 · p. 1
  • SPCI adaptively re-estimates residual quantiles over time, using temporal dependence among past residuals to construct intervals.The framework uses quantile regression on residuals and can use quantile-regression methods beyond the random forest implementation.Source: §1 · pp. 1–2
  • SPCI establishes asymptotic conditional coverage for dependent data and finite-sample distribution-free marginal coverage when data are exchangeable.The exchangeable-data guarantee matches traditional conformal prediction methods.Source: §1 · p. 2
  • Experiments report competitive or improved empirical performance against baseline conformal methods, including narrower real-data intervals without coverage loss.The paper also reports benefits for multi-step predictive inference.Source: §1 · p. 2

1.1. Literature review

  • Prior sequential conformal methods update residuals or significance levels but do not exploit serial correlation among scores, typically using empirical quantiles of past residuals.The paper contrasts these approaches with SPCI’s conditional quantile modeling of residuals.Source: §1.1 · pp. 2–3, §1.2 · p. 3

1.2. Connection with related works

  • With random forest quantile regression, SPCI can be viewed as adaptively learning data-dependent residual weights, unlike prior non-adaptive weighting schemes.The paper presents this as a step toward investigating data-dependent weights in non-exchangeable conformal prediction.Source: §1.2 · p. 3

2. Problem setup

The setup targets sequential prediction intervals around a user-chosen point predictor, balancing specified coverage with narrow width. Because residuals may be temporally dependent, past residuals can inform prediction of future residuals.

2. Problem setup

  • Sequential conformal prediction uses ordered, potentially non-exchangeable residuals as feedback, updating past residuals with each new prediction residual.Traditional conformal prediction instead assumes exchangeable residuals and uses the same residual set throughout prediction.Source: Figure 2 · p. 3
  • The setup allows continuous scalar outcomes, features from outcome history or exogenous variables, and highly correlated observations without specifying a conditional-distribution form.The first T observations are training data.Source: §2 · p. 3
  • Intervals are constructed sequentially from time T + 1 around a given point prediction, aiming for user-specified coverage 1 − α and the narrowest possible width.The intervals use observations available up to t − 1 and can wrap an arbitrary user-chosen predictive algorithm.Source: §2 · p. 3
  • The setup distinguishes weaker marginal coverage from stronger conditional coverage, while requiring intervals to remain narrow rather than vacuously broad.An interval covering the entire real line is given as an extreme example of a vacuous interval.Source: §2 · p. 3
  • When residuals are temporally correlated, past residuals can inform the distribution of future residuals and help predict a new residual's tail probability.This predictive use of residual history is the stated motivation for SPCI.Source: §2 · p. 4

3. Algorithms

The section moves from split conformal without prediction-time feedback to EnbPI with updated empirical residuals, then introduces SPCI, which estimates future residual quantiles conditionally on past residuals.

3.1. Vanilla split conformal

  • Split conformal fixes calibration residuals during prediction and achieves exact finite-sample marginal coverage under exchangeability, but not valid conditional coverage without further assumptions.Its prediction model is fit on one training split, and residuals from the other split determine interval width.Source: §3.1 · p. 4

3.2. EnbPI: Ensemble version using empirical residuals

  • EnbPI avoids data splitting, uses leave-one-out ensemble predictors, and updates past residuals to adapt interval width during prediction.Its intervals use empirical quantiles of a window of past residuals.Source: §3.2 · p. 4
  • With dependent residuals, EnbPI's empirical quantiles may not yield desired coverage, and its cited guarantee requires a linear conditional model that is consistently estimated.The paper notes that the linear-form assumption can limit practice.Source: §3.2 · pp. 4–5

3.3. Proposed SPCI algorithm

  • SPCI replaces empirical residual quantiles with estimated conditional quantiles, using past residuals to construct sequential prediction intervals.The method refits quantile models as a sliding window of recent residuals advances; the paper uses quantile random forests.Source: §3.3 · p. 5

4. Theory

SPCI has exact finite-sample marginal coverage under exchangeability and asymptotic conditional coverage for dependent data under stated assumptions. Its guarantees are distribution-free and model-free, but interval-width convergence to the oracle is not established.

4.1. Under exchangeability

  • Under exchangeability, SPCI obtains exact finite-sample marginal coverage using the empirical quantile function.The result applies to the special version of SPCI in Algorithm 2.Source: §4 · p. 6, §4.1 · p. 6

4.2. Beyond exchangeability

  • SPCI re-estimates residual quantiles using data-adaptive weights that exploit temporal autocorrelation, unlike the fixed weights of Barber et al. (2022).The quantile regression framework can use methods beyond random forests.Source: §4.2 · p. 6, §4.3 · p. 7
  • The dependent-data guarantee assumes bounded, suitably decaying residual dependence and regularity conditions, without a parametric conditional-quantile form.The dependence condition is intended to rule out dependence strong enough to prevent asymptotic consistency.Source: §4.2 · p. 7
  • For dependent data, SPCI intervals have asymptotic conditional coverage when the quantile random forest estimator satisfies the stated assumptions.The proof extends quantile-estimation consistency analysis from independent observations to dependent observations.Source: §4.2 · pp. 6–7

4.3. Implications of results

  • The coverage guarantees make no explicit residual-distribution or outcome-model assumptions, but oracle interval-width convergence is not theoretically established.The authors state that proving width convergence would require assumptions on the inverse conditional CDF that fall outside their model-free focus.Source: §4.3 · p. 7

5. Experiments

Across simulations and real time series, SPCI generally narrows intervals while maintaining coverage, with some under-coverage in drift and change-point simulations at the original significance level. Its multi-step wind intervals adapt to the series and widen with predictive horizon.

5.1. Simulation

  • In non-stationary and heteroskedastic simulations, SPCI improves interval width over EnbPI without sacrificing valid coverage.Both methods use random forest regression for the point estimator.Source: Table 1 · p. 8, §5.1 · p. 8
  • In drift and change-point simulations, SPCI intervals are much narrower but slightly under-cover at the original significance level; lowering α restores valid coverage with comparable widths.The adjusted α values are 0.09 for distribution shift and 0.075 for change points.Source: §5.1 · p. 8, Table 2 · p. 8

5.2. Real-data examples

  • Rolling real-data results show SPCI largely retains coverage, adapts interval width over time, and has less trial variance than competing methods.Figure 4 reports rolling coverage and width over three time series.Source: Figure 4 · p. 8, §5.2 · pp. 8–9
  • On three real time series, SPCI yields significantly narrower intervals while nearly maintaining the 0.9 target coverage.Table 3 reports marginal coverage and width for all methods.Source: §5.2 · p. 8, Table 3 · p. 9
  • For multi-step wind prediction, SPCI intervals follow the series more closely than EnbPI intervals and widen on average at longer horizons.The divide-and-conquer approach fits S QRF estimators on lagged training data to construct S intervals simultaneously.Source: §5.2 · p. 9, Figure 5 · p. 9

6. Conclusions

SPCI constructs time-series prediction intervals by fitting conditional quantile models to residuals, using their temporal dependence. The paper reports asymptotic valid conditional coverage and improved performance over existing sequential conformal methods, while identifying multivariate and multi-step extensions as future work.

6. Conclusions

  • SPCI uses quantile regression on past residuals to construct adaptive prediction intervals for time series, with asymptotic valid conditional coverage.The framework is model-free and distribution-free and is described for arbitrary predictive models.Source: §6 · p. 9
  • Experimental results consistently show improved performance over existing sequential conformal prediction methods.Source: §6 · p. 9
  • Multivariate confidence regions and more precise modeling of joint future residuals remain proposed extensions.Source: §6 · p. 9

A. Proof

The proof establishes convergence of a conditional distribution estimator and its quantiles under stated dependence, weighting, and regularity assumptions. It then uses quantile convergence to support the asymptotic coverage result for SPCI.

A. Proof

  • The covariance of indicator variables is assumed to depend on index difference, with examples including finite-memory and linear-decay dependence.The finite-memory example is stated to satisfy the required decay condition; the linear-decay example is treated separately.Source: A. Proof · p. 13
  • The assumptions require weights of order O(1/˜T), a Lipschitz conditional distribution, and a continuous, strictly increasing conditional CDF.Source: A. Proof · p. 14
  • The conditional distribution estimator is analyzed by splitting its error into two terms and bounding them using Chebyshev inequalities and the assumptions.Source: A. Proof · pp. 14–16
  • The proof uses convergence of the conditional distribution estimator to establish convergence of estimated residual quantiles, supporting the SPCI coverage result.The argument invokes pointwise convergence of the estimator and convergence of estimated quantiles in probability.Source: A. Proof · p. 16

B. Experimental details

The experiments compare SPCI with three sequential conformal baselines and describe three real-world time series. The section also introduces additional simulation results comparing rolling coverage and interval width with NEX-CP.

B. Experimental details

  • SPCI is compared with EnbPI, AdaptiveCI, and NEX-CP, which use residual calibration, adaptive significance levels, and weighted quantiles, respectively.All three baselines leverage feedback after each sequential outcome is revealed.Source: B. Experimental details · p. 17
  • The real-data experiments use wind speed, Atlanta solar radiation, and electricity transfer between New South Wales and Victoria.The records are sampled every 15 or 30 minutes and span periods from one week to 2.5 years.Source: B. Experimental details · p. 17

B.1. Simulation

  • The simulation subsection adds rolling coverage and interval-width results comparing SPCI with NEX-CP.Source: §B.1 · p. 17

B.1.1. DATA SIMULATION

The simulation setups cover non-stationarity, heteroskedasticity, and distribution drift or changepoints. A comparison without known time features reports narrower intervals for SPCI than the baselines.

B.1.1. DATA SIMULATION

  • The non-stationary setup includes time-related effects, lagged response features, and errors following an AR(1) process with ρ = 0.6.Source: §B.1.1 · pp. 17–18
  • With time information unavailable, SPCI produces narrower intervals than the baselines in the non-stationary simulation comparison.Table A.1 compares the four conformal prediction methods with ˜X_t = X_t.Source: §B.1.1 · p. 18, Table A.1 · p. 18
  • For heteroskedastic errors, the procedure estimates quantiles of normalized residuals and rescales them using estimated conditional standard deviations.Source: §B.1.1 · p. 18
  • The distribution-drift setup linearly interpolates regression coefficients between initial and final values over the simulated series.Source: §B.1.1 · p. 18
  • During testing, the point estimator is refit with a rolling window after a 100-point burn-in, using different window limits for shifts and changepoints.The limits are T0 = 300 under distribution shifts and T0 = 200 under changepoints.Source: §B.1.1 · p. 18

B.1.2. COMPARISON WITH NEX-CP

The comparison discusses residual behavior under distribution shift: residuals may be nearly independent yet non-identically distributed, which can weaken quantile estimation from past residuals. Figure A.1 tracks rolling coverage and width against the target and marks two changepoints.

B.1.2. COMPARISON WITH NEX-CP

  • Residuals are described as nearly independent but non-identically distributed in the distribution-shift and change-point settings.The change-point setting has zero dependence through β_i; the figure marks changepoints at indices 500 and 1500.Source: §B.1.2 · p. 18, Figure A.1 · p. 19
  • Figure A.1 plots rolling coverage and width without adjusted α values, with target coverage 0.9 marked.Source: Figure A.1 · p. 19
  • Drifts or changepoints can change the residual distribution, making a quantile regression forest fit on past residuals a potentially poor estimator for the test residual’s conditional quantile.Source: §B.1.2 · p. 19

B.2. Additional real-data comparisons

On three NASDAQ stock series, SPCI’s interval widths are reported as narrower than AdaptiveCI’s in all three coverage scenarios. The comparison distinguishes cases where both methods maintain coverage, only SPCI does, or both lose coverage.

B.2. Additional real-data comparisons

  • The stock comparison uses closing prices for three randomly selected NASDAQ companies and compares SPCI with AdaptiveCI.Source: §B.2 · p. 19
  • On AJISF, where both methods yield valid coverage, SPCI’s intervals are significantly narrower than AdaptiveCI’s.Source: §B.2 · p. 19
  • On AGTC, AdaptiveCI loses coverage while SPCI maintains it, and SPCI still has significantly narrower intervals.Source: §B.2 · p. 19
  • On AAVL, both methods lose coverage, but SPCI’s coverage loss is smaller and its intervals remain narrower.Source: §B.2 · p. 19

B.3. Multi-step inference

Multi-step SPCI addresses settings where forecasts must be issued before new responses arrive, using separate horizon-specific predictors and residual quantile models. Its computational cost includes additional point-predictor fits, and interval widths are expected to increase with the forecast horizon.

B.3. Multi-step inference

  • Multi-step forecasting is useful when dispatch requires advance forecasts or delayed data collection prevents feedback before all S intervals are constructed.Source: §B.3 · p. 20
  • At each prediction time, the method constructs S intervals together for responses and features that remain unavailable until those intervals have been made.Source: §B.3 · p. 20
  • Rather than estimate the joint distribution of S future residuals, multi-step SPCI uses a divide-and-conquer approach with horizon-specific LOO predictors and quantile regressions.It computes residuals only at times when new feedback is available.Source: §B.3 · p. 20
  • As the multi-step size S increases, SPCI is expected to produce wider intervals because uncertainty grows in the baseline regression or quantile regression fitted on lagged data.For an AR(1) example, the error variance grows with the horizon.Source: §B.3 · p. 20
  • Compared with one-step SPCI, the multi-step version refits S−1 additional sets of LOO point predictors but fits the same number of quantile regressors.Source: §B.3 · p. 20

C. Additional technical details

  • Algorithm 3 fits horizon-specific bootstrap predictors, updates quantile estimators when residuals become available, and builds intervals from aggregated forecasts and learned quantile widths.Source: C. Additional technical details · p. 21

Related papers

Loading 2212.03463v3…