Adaptive Conformal Predictions for Time Series

Margaux Zaffran; Aymeric Dieuleveut; Olivier Féron; Yannig Goude; Julie Josse

Source-linked AI summary

Adaptive Conformal Predictions for Time Series

Margaux Zaffran, Aymeric Dieuleveut, Olivier Féron, Yannig Goude, Julie Josse

arXiv:2202.07282v1Feb 15, 2022stat.MLcs.LG

TL;DR

Standard conformal prediction requires exchangeability, which time series generally lack, motivating methods with stronger temporal handling. This paper analyzes ACI for dependent time series, introduces parameter-free AgACI, and reports strong validity and efficiency in simulations and electricity-price forecasting.

Problem
Conformal prediction’s exchangeability-based validity does not directly apply to time-series data with temporal dependence, while uncertainty quantification remains important for electricity-price forecasting.
Method
The paper analyzes ACI for generally dependent time series and introduces AgACI, which uses online expert aggregation to avoid selecting γ.
Results
AgACI nearly attains validity, with coverage over 89.8% across simulated dependence settings, and achieves the best efficiency among valid methods.
Takeaways & Limitations
ACI can produce effective time-series prediction intervals with a well-chosen γ, while AgACI provides good validity and efficiency without manually choosing it.
Takeaways & Limitations
The best γ remains difficult to choose in complex practical settings, and future work includes studying whether AgACI preserves asymptotic validity.

Abstract

from arXiv · show

Uncertainty quantification of predictive models is crucial in decision-making problems. Conformal prediction is a general and theoretically sound answer. However, it requires exchangeable data, excluding time series. While recent works tackled this issue, we argue that Adaptive Conformal Inference (ACI, Gibbs and Cand{è}s, 2021), developed for distribution-shift time series, is a good procedure for time series with general dependency. We theoretically analyse the impact of the learning rate on its efficiency in the exchangeable and auto-regressive case. We propose a parameter-free method, AgACI, that adaptively builds upon ACI based on online expert aggregation. We lead extensive fair simulations against competing methods that advocate for ACI's use in time series. We conduct a real case study: electricity price forecasting. The proposed aggregation algorithm provides efficient prediction intervals for day-ahead forecasting. All the code and data to reproduce the experiments is made available.

1 Introduction

Predictive uncertainty is important for volatile, dependent electricity markets, but standard conformal prediction relies on exchangeability, which time series violate. The paper studies ACI for dependent time series and proposes AgACI to adapt its learning rate.

Renewable energy increases dependence and volatility in energy markets, strengthening the need for accurate probabilistic electricity-price forecasts.
Conformal prediction builds finite-sample predictive intervals for black-box models without distributional assumptions beyond exchangeability.
Time series violate the exchangeability assumption underlying split conformal validity, while existing online applications show validity varies across markets and regression models.
The paper argues that ACI, developed for distribution shifts, can benefit dependent time series without distribution shifts.
AgACI uses online expert aggregation to avoid choosing ACI’s learning rate γ while targeting validity and efficiency.
The study analyzes ACI theoretically, compares it with competing methods in synthetic experiments, and forecasts French electricity prices with predictive intervals.

2 Setting: ACI for time series

The time-series framework applies ACI sequentially, updating its effective miscoverage level from prior errors while using a sequential calibration split. In an AR(1)-residual example, positive learning rates adapt better and achieve valid coverage than γ = 0.

The framework predicts T1 subsequent responses sequentially after T0 observations, using revealed past data at each prediction step.
ACI combines an online random split with a recursively updated effective miscoverage level αt instead of the fixed target α.
If ACI misses an observation, αt+1 decreases and the predictive interval expands; coverage produces the converse update.
The time-series adaptation replaces ACI’s random split with a sequential split, allowing the method to respond to trends, seasonality, or dependence missed by the fitted model.
In an AR(1)-residual toy example, γ = 0.01 and 0.05 include 90% and 92% of points, compared with 72.6% for γ = 0.
Higher γ reduces adaptation delay, while small residuals allow ACI to shorten intervals when observations lie far from both bounds.

3 Impact of γ on ACI efficiency

ACI’s learning rate γ creates an efficiency trade-off: adaptation can hurt in exchangeable data but improve interval length under strong temporal dependence. Because the best γ depends on dependence strength, AgACI motivates adaptive selection.

ACI’s asymptotic validity guarantee does not determine interval length, and larger γ values can generate frequent infinite intervals.
3.1 Exchangeable case: In the exchangeable case, ACI degrades efficiency linearly with γ; for Gaussian residuals, γ = 0.03 and 0.05 increase length by 1.59% and 3.38% over γ = 0.
3.2 AR(1) case: For highly correlated AR(1) residuals, the analysis assumes clipped residuals, a known bounded stationary quantile function, and a Markov-chain formulation.
3.2 AR(1) case: For high dependence, a strictly positive γ can produce smaller intervals than γ = 0, but excessive γ destabilizes ACI and creates numerous infinite intervals.
3.2 AR(1) case: The optimal learning rate is nonmonotonic in dependence strength: larger γ suits moderate dependence, whereas smaller γ stabilizes intervals for longer-memory processes.
3.2 AR(1) case: Choosing γ remains difficult in complex practical settings because failing to select γ* can produce significantly larger intervals.

4 Adaptive strategies based on ACI

The paper develops adaptive strategies that avoid selecting a fixed ACI learning rate by choosing among ACI experts or aggregating them online.

Naive strategy: The naive strategy selects the historically most efficient γ among those that previously ensured validity, after a warmup period.In simulations, the first 50 steps use γ = 0.
AgACI: AgACI aggregates ACI experts with different γ values instead of selecting one fixed learning rate.Each expert is an ACI procedure parameterized by γk.
AgACI: AgACI performs separate online aggregations for the lower and upper interval bounds using experts’ past losses.The aggregation uses pinball losses with parameters α/2 and 1 − α/2 for the two bounds.
AgACI: The aggregation rule can use gradients and requires thresholding because aggregation is defined for bounded experts.The paper uses Bernstein Online Aggregation and reports similar experimental results for several alternative rules.
Algorithm: AgACI takes as input a miscoverage rate and a grid of candidate γ values, then applies the aggregation procedure sequentially.The algorithm is presented as Online Expert Aggregation on ACI.

5 Numerical evaluation on synthetic data sets

Synthetic experiments compare ACI-based methods with online and ensemble baselines under controlled temporal dependence. AgACI maintains validity and offers strong efficiency, while baseline performance varies with dependence.

Experimental design: The experiments vary ARMA dependence parameters ϕ and θ while holding the noise asymptotic variance at 10.The predictive model is a random forest with fixed hyperparameters across experiments.
Evaluation: The figures jointly display empirical coverage and average median interval length, with the target miscoverage rate set to α = 0.1.A method is assessed by considering validity and efficiency together.
Results: EnbPI loses coverage while producing shorter intervals as dependence increases, whereas EnbPI V2 is closer to target coverage.OSSCP also loses validity as ϕ and θ increase; refitting improves performance relative to offline SCP.
Results: AgACI achieves coverage over 89.8% for all ϕ values and the best efficiency among valid methods.ACI remains robust to increasing dependence except at ϕ = θ = 0.99, while suitable γ values preserve valid coverage.
Results: ACI and AgACI can produce more infinite intervals, degrading efficiency when interval lengths are imputed.This trade-off is especially reported for ACI with γ = 0.05 and AgACI.
Takeaways: The paper concludes that AgACI achieves valid coverage with good efficiency across the simulation settings.It also reports that temporal dependence affects validity and that online procedures outperform offline ones.

6 Forecasting French electricity spot prices

The paper applies conformal prediction to French day-ahead electricity prices, using rolling training and calibration windows. ACI produces the smallest correctly covered intervals, while AgACI remains efficient and valid.

Data and motivation: The case study forecasts French electricity spot prices from 2016–2019, whose volatility motivates probabilistic prediction intervals.The market sets 24 hourly prices for the following day through the Euphemia auction algorithm.
Illustration: Figure 6 illustrates AgACI prediction intervals for French prices on 25 January 2019.The figure covers prices from 2016 to 2019 and provides an example of the predicted intervals.
Method: The study fits one random-forest model per hour and forecasts 2019 with a three-year sliding window.The window uses one year and six months for proper training and the most recent year and a half for calibration.
Results: ACI gives the smallest intervals with correct coverage for γ = 0.01 and γ = 0.05.The authors attribute the interval shrinkage to updating the quantile level.
Results: AgACI is more efficient than γ = 0 while maintaining validity, but slightly over-covers and is less efficient than ACI with a well-chosen γ.OSSCP and EnbPI over-cover in this application, with OSSCP less than its offline version.

7 Conclusion

The paper argues that ACI can support interval prediction for time series with general dependencies and proposes AgACI to avoid selecting its learning rate. It combines theoretical analysis, synthetic experiments, and an electricity-price application, while identifying further theoretical work for AgACI.

ACI’s efficiency is worse than CP in exchangeable settings, while its learning-rate dependence is analysed for autoregressive cases.
AgACI wraps ACI with online expert aggregation to avoid choosing γ.
Extensive synthetic experiments examine ACI against EnbPI and online SCP across different temporal-dependence strengths and structures.
A detailed electricity-price forecasting study evaluates interval prediction in an energy-transition setting.
Theoretical analysis of whether AgACI preserves experimental asymptotic validity remains future work.

Appendices

The appendices review Split Conformal Prediction, its exchangeability requirement, algorithmic construction, and theoretical validity results. They also provide algorithm details and supporting formal statements.

A.1 Split Conformal Prediction: Algorithm 2 specifies random splitting, regression fitting, absolute residual scores, and empirical-quantile computation.
A.2 Theoretical guarantees: Conformal prediction’s validity relies on exchangeability, meaning the joint distribution is invariant to permutations of the observations.
A.2 Theoretical guarantees: The appendix states a quasi-exact validity theorem for Split Conformal Prediction and an additional upper bound when calibration scores have a continuous joint distribution.

A.3 Examples of dependent scores when data noise is exchangeable

The examples show that misspecified predictive models can produce dependent residual scores even when the underlying noise is independent or white noise. A simulated dependent-noise model then supports visual analysis of adaptive intervals.

The appendix constructs examples where scores are non-exchangeable although the data’s true noise is exchangeable.
For an endogenous AR(1) process with an incorrectly estimated coefficient, residuals follow an ARMA(1,1) process.
Thus, dependent residuals can arise even when the underlying model contains only white noise.
With an exogenous misspecified model, the residuals become autoregressive despite i.i.d. Gaussian noise.
Visualization setup: Model A.5 uses deterministic explanatory variables and depicts the covariates, noise, and full process across three subplots.
Visualization setup: The visualization study predicts 0.9-coverage intervals after an initial calibration period and shows that conformity scores, rather than observed values alone, influence CP.

B.1 Proof of Theorem 3.1

The proof develops Markov-chain representations for ACI and establishes stationary behavior under exchangeable scores and clipped AR(1) residuals. It also records numerical median-length behavior and methodological scope boundaries.

Theorem 3.1: Under exchangeable scores, ACI’s effective miscoverage process forms a Markov chain with a stationary distribution under the theorem’s regularity assumptions.
Theorem 3.1: The proof establishes finite state space and irreducibility before deriving stationary moments needed for the Taylor expansion.
Theorem 3.1: The stationary expected interval length converges almost surely to its expectation under the stationary distribution.
Proof scope: The proof’s Taylor-expansion argument requires control of four moments; whether three moments suffice is left unresolved.
Theorem 3.2: For clipped AR(1) residuals, the joint process of the adaptive level and lagged residual is a homogeneous Markov chain with a unique stationary distribution.
Numerical study: The median-length study finds behavior similar to average length, including monotonic dependence on ϕ and a minimum over γ.
Methodological scope: Bootstrap strategies are not adapted to time series in the discussed EnbPI setting, and extending dependent-data improvements is outside the paper’s scope.

C.3.2 DETAILS ON THE IMPLEMENTATION

The section provides a unified framework for comparing computational resources across conformal prediction methods and documents available implementations.

The paper uses a unified experimental framework to summarize implementation resources for the evaluated methods.
Table 1 lists currently available online code for each method alongside the repository’s proposed code.
The implementation summary identifies each codebase’s programming language and, when relevant, its code type.

D Additional experiments on synthetic data sets

The supplementary experiments extend sensitivity and method comparisons across multiple dependent-noise settings, while examining infinite intervals and data-splitting choices. They also identify dependence-specific performance patterns and note that further analysis would be beneficial.

Additional sensitivity experiments: The appendix extends ACI sensitivity comparisons to AR(1) and MA(1) processes with asymptotic variance 10, including the naive strategy and AgACI.
Method comparisons: The comparison of time-series conformal prediction methods is extended to AR(1), MA(1), and ARMA(1,1) noises with asymptotic variances 10 and 1.
Additional sensitivity experiments: All ACI methods produce nearly valid intervals for the additional processes, with coverage above 89.25%.
Dependence effects: For MA(1) noise, θ has little influence, whereas AR(1) and ARMA(1,1) produce similar dependence patterns in the experiments.
Splitting strategies: The sequential split produces larger intervals with slightly higher coverage than the randomized split, and further analysis of splitting effects is left as beneficial future work.

E.1 Details about the data set

The electricity-price dataset contains historical price variables used for forecasting. On January 25, 2019, the forecasts differed substantially from actual values, yet most prediction intervals covered nearly all hourly observations.

Dataset construction: Table 3 presents an extract of the built dataset for French electricity spot-price forecasting.
Forecasting example: On January 25, 2019, prediction intervals included actual values for almost all hours and methods, except after 5 pm and earlier for EnbPI from 11 am.