Source-linked AI summary
Conformal PID Control for Time Series Prediction
Anastasios N. Angelopoulos, Emmanuel J. Candes, Ryan J. Tibshirani
TL;DR
The paper addresses uncertainty quantification when changing time-series distributions undermine standard conformal prediction. It develops conformal PID control, combining quantile tracking, error integration, and scorecasting; experiments find robust performance, with scorecasting helping when predictable score trends remain but potentially increasing volatility otherwise.
Problem
Changing data distributions can degrade predictive performance and invalidate standard uncertainty-quantification techniques such as conformal prediction.
Method
Conformal PID control combines quantile tracking, error integration, and scorecasting to stabilize coverage and adapt prediction sets to changing score sequences.
Results
Quantile tracking with integration consistently yields robust and favorable experimental performance, while scorecasting adds benefits when predictable score trends remain.
Takeaways & Limitations
Scorecaster design, including whether to use one, is an important modeling step alongside base-forecaster design.
Takeaways & Limitations
Scorecasting can increase variance and make coverage and prediction sets more volatile when score trends are absent or the scorecaster is poorly designed.
Abstract
from arXiv · showhide
We study the problem of uncertainty quantification for time series prediction, with the goal of providing easy-to-use algorithms with formal guarantees. The algorithms we present build upon ideas from conformal prediction and control theory, are able to prospectively model conformal scores in an online setting, and adapt to the presence of systematic errors due to seasonality, trends, and general distribution shifts. Our theory both simplifies and strengthens existing analyses in online conformal prediction. Experiments on 4-week-ahead forecasting of statewide COVID-19 death counts in the U.S. show an improvement in coverage over the ensemble forecaster used in official CDC communications. We also run experiments on predicting electricity demand, market returns, and temperature using autoregressive, Theta, Prophet, and Transformer models. We provide an extendable codebase for testing our methods and for the integration of new algorithms, data sets, and forecasting rules.
1 Introduction
The paper addresses uncertainty quantification for time series under distribution shifts by using conformal PID control to stabilize long-run coverage and anticipate systematic score changes. Its framework combines quantile tracking, error integration, and scorecasting, with deterministic guarantees and improvements in forecasting experiments.
- Motivation: Distribution shifts from seasonality, retraining, and changing behavior can degrade prediction and invalidate standard conformal uncertainty guarantees.The paper therefore considers adversarial online sequences without exchangeability or a probabilistic data model.
- Motivation: Without assumptions on sequential data, achieving pointwise coverage may require infinitely large prediction intervals, motivating long-run coverage guarantees.The target is average miscoverage approaching α as the time horizon grows.
- Method: Conformal PID control treats prediction-set construction as a controller that adjusts score quantiles using coverage errors, enabling stable coverage and prospective adaptation to score changes.The process variable is q_t, the output is err_t, and the target set point is α.
- Method: The framework combines quantile tracking, error integration, and scorecasting to react to errors, stabilize cumulative coverage, and predict systematic score trends.These correspond respectively to proportional, integral, and derivative control.
- Theory: Theorem 1 guarantees long-run coverage deterministically for bounded score sequences and scorecaster predictions when the integrator satisfies the required saturation condition.The guarantee does not require a probabilistic model for the data.
- Experiments: In California COVID-19 forecasting, conformal PID reduced a 10-week undercoverage stretch from 20% coverage for the ensemble to 70%, against an 80% nominal level.The method corrected consistent winter-wave underprediction using a scorecaster trained on forecasts, cases, and deaths from all 50 states.
Electricity demand forecasting.
In electricity demand forecasting, conformal PID control uses a Transformer forecaster, a tan integrator, and a Theta scorecaster to anticipate intraday score variation. Compared with ACI, it produces tight coverage without excessively large or infinite prediction sets.
- Electricity demand forecasting: Conformal PID control anticipates intraday score variation and produces sets that closely follow the ground truth without excessively large or infinite sets.The improvement is attributed mainly to the scorecaster’s seasonality component, especially when the base forecaster is imperfect.
- Related work: The broader related work studies online calibration and conformal prediction beyond exchangeable data, with prior approaches relying on probabilistic assumptions.The cited lines of work include adversarial online calibration and time-series conformal prediction.
- Electricity demand forecasting: Figure 3 compares ACI in the left column with conformal PID control in the right column for Transformer-based electricity demand forecasts.The PID configuration uses a tan integrator and a Theta scorecaster.
- Electricity demand forecasting: The electricity-demand experiment uses nominal coverage 1 −α = 0.9 and averages coverage over a trailing window of 50 points.Red dots marking miscoverage events are omitted, with summary statistics reported in Table 2.
2 Methods
The methods recast online conformal prediction as control: quantile tracking provides proportional correction, error integration stabilizes cumulative coverage, and scorecasting anticipates systematic score trends. The resulting procedures provide deterministic long-run coverage under progressively weaker score assumptions, while scorecaster quality governs practical stability and set size.
- 2.1 Quantile tracking: Quantile tracking applies online gradient descent to quantile loss and achieves long-run coverage for bounded scores without requiring the bound to be known.For any learning rate η > 0, the method updates the quantile using sequential score observations and coverage errors.
- 2.1 Quantile tracking: ACI is a special case of quantile tracking and may output infinite or null prediction sets, unlike direct quantile tracking on the original score scale.ACI obtains coverage by transforming unbounded scores into bounded ones, but this can produce degenerate sets when its adaptive level leaves [0,1].
- 2.2 Error integration: Error integration incorporates cumulative coverage error into quantile updates, allowing long-run coverage even when scores are unbounded.A saturating integrator with an admissible, sublinear threshold function controls how often coverage-error corrections occur.
- 2 Methods: Conformal PID control yields deterministic long-run coverage for any scorecaster and any integrator satisfying the required saturation condition.The guarantee is established for arbitrary score sequences within the stated assumptions, rather than relying on a probabilistic data model.
- 2.3 Scorecasting: Scorecasting forecasts the next score quantile to remove systematic trends, making the method forward-looking rather than purely reactive.It is useful when modifying or retraining the base forecaster is difficult, including computationally costly Transformer pipelines.
- 2.3 Scorecasting: A well-executed scorecaster can reduce score variability and tighten prediction sets, whereas an aggressive scorecaster can increase volatility and set size.The practical value of scorecasting therefore depends on whether predictable signal remains in the score sequence.
3 Experiments
Experiments span electricity demand, stock returns, temperature, and multiple forecasting models, with conformal PID variants evaluated under different control settings. Results illustrate trade-offs among coverage stability, set behavior, integrator effects, and scorecasting, while highlighting settings where scorecasting can hurt.
- 3 Experiments: The experiments combine electricity demand, stock returns, and Delhi temperature with AR, Transformer, Prophet, and other forecasters.Except for COVID-19, base forecasters are retrained at each time point, with asymmetric residual scores and a Theta scorecaster.
- 3 Experiments: The COVID-19 experiment uses the Forecast Hub ensemble as its base forecaster, while other settings use retrained models and a Theta scorecaster.COVID-19 instead uses an ℓ1-penalized quantile-regression scorecaster with features from previous forecasts, cases, and deaths.
- 3.1 ACI versus quantile tracking: Tight coverage required a trade-off: at larger learning rates, ACI frequently returned infinite sets, whereas quantile tracking used less aggressive corrections.For Amazon returns, default learning rates produced nontrivial oscillations around 0.9 coverage; larger rates tightened coverage for both methods.
- 3.2 Integrator effects: For Google returns, adding the integrator stabilized coverage for most of the sequence under a deliberately small learning rate.The comparison uses quantile tracking with and without integration, corresponding to P and PI control.
- 3.3 The effect of scorecasting: Scorecasting can improve coverage and set sizes when base-forecaster scores have predictable trends, including examples from the COVID-19 setting.The paper presents these results as conceptual illustrations, with additional evaluations in the appendix and repository.
- 3.3 The effect of scorecasting: Scorecasting is not uniformly beneficial: complex scorecasters can add variance and degrade coverage or set stability, especially when scores lack trends.The authors recommend designing scorecasters carefully rather than relying on out-of-the-box methods in high-stakes settings.
4 Discussion
The framework combines quantile tracking, error integration, and scorecasting to obtain coverage guarantees while adapting prediction sets to changing score sequences. Its extensions include kernel-weighted integration, but scorecaster design and heuristic tuning remain important practical choices.
- 4 Discussion: The framework combines quantile tracking, error integration, and scorecasting as proportional, integral, and derivative control components.Quantile tracking uses online gradient descent, integration accumulates coverage error, and scorecasting predicts systematic score trends.
- 4 Discussion: Quantile tracking and integration were consistently robust, whereas scorecasting helped when predictable score trends remained but could otherwise increase volatility.The scorecaster is therefore a substantive modeling choice alongside the base forecaster.
- 4 Discussion: The experiments outside COVID-19 were illustrative rather than state-of-the-art, and several controller constants were selected with heuristics instead of adaptive online tuning.The authors identify rigorous adaptive tuning as a direction for future work.
- 4 Discussion: Kernel-weighted integration can target trailing-window coverage or group-wise coverage through time-based or covariate-based weights.A trailing window aggregates recent coverage, while bin-based weights support group structure.
D Comparison to clipped ACI
The comparison evaluates the quantile tracker against a clipped ACI variant that prevents infinite-sized prediction sets by clipping them to the largest observed score.
- D Comparison to clipped ACI: Figures 7 and 8 compare the quantile tracker with clipped ACI, which disallows infinite-sized sets by clipping to the largest score seen so far.The comparison focuses on avoiding the infinite-set behavior of unclipped ACI.
E More details on COVID-19 forecasting
The COVID-19 experiments use scorecasters based on recent scores and statewide case and death counts, with analogous results shown for New York and Texas. The supplied summary reports marginal coverage and set-size statistics for the compared methods.
- E More details on COVID-19 forecasting: The COVID-19 scorecaster uses the three most recent ensemble-forecaster scores plus three recent case and death counts from all 50 states.It is implemented as ℓ1-penalized quantile regression with penalty parameter 10.
- E More details on COVID-19 forecasting: The displayed comparison includes AR, Prophet, Theta, and Transformer base forecasters alongside clipped ACI and Conformal PID Control.The supplied labels identify these methods but do not provide their corresponding values.
- E More details on COVID-19 forecasting: Figures 9 and 10 show analogous COVID-19 forecasting experiments for New York and Texas.The supplied visual passages identify the New York and Texas four-week-ahead results.
- E More details on COVID-19 forecasting: 0.85 marginal coverage versus 0.75, with the longest error sequence reduced from 7 to 2, is reported for Conformal PID Control versus the base forecaster.The same comparison reports larger average, median, and upper-quantile set sizes for Conformal PID Control.
F Further experiments
The further experiments provide a broader view across datasets and tuning parameters while fixing autoregression as the base forecaster. Coverage and prediction-set plots vary learning rate, method, and color-coded series, with a tan integrator and Theta scorecaster used throughout.
- F Further experiments: The further experiments examine all datasets across a range of tuning parameters, while restricting the reported comparisons to autoregression as the base forecaster.Results for other base forecasters are referred to the project repository.
- F Further experiments: Each experiment includes coverage and prediction-set plots whose columns represent methods, rows represent learning rates, and colors remain consistent across methods.For the quantile tracker, the learning rate is the multiplier applied to the predicted score quantile.
F.1 Amazon/Google
The experiment uses multivariate stock-price data to forecast the daily opening prices of Amazon and Google on a log scale.
- The data comprise daily prices for 30 blue-chip stocks from January 1, 2006 to December 31, 2014.
- The forecasting targets are Amazon and Google daily opening prices represented on a log scale.
- The scorecaster receives the previous open prices of all 30 stocks as available information.
F.2 Microsoft
The Microsoft experiment uses a univariate stock-price time series covering April 2015 through May 2021.
- The dataset contains a single stock open-price series for Microsoft.
- The Microsoft series spans April 1, 2015 to May 31, 2021.
- This experiment evaluates forecasting with a univariate time-series input.
F.3 Daily temperature in Delhi
The real-data experiments include Delhi weather observations and New South Wales electricity demand, with multiple environmental or market variables available as covariates.
- F.3 Daily temperature in Delhi: The Delhi dataset records temperature, humidity, wind speed, and atmospheric temperature from January 1, 2003 to April 24, 2017.
- F.3 Daily temperature in Delhi: Delhi temperature observations are daily averages computed from eight measurements taken across three-hour periods.
- F.3 Daily temperature in Delhi: The New South Wales electricity-demand dataset contains half-hour observations from May 7, 1996 to December 5, 1998, with related demand, price, and energy-transfer variables as covariates.
- F.3 Daily temperature in Delhi: New South Wales demand is normalized to the interval [0, 1] by default.
F.5 Synthetic data sets
The synthetic experiments examine score sequences with changing behavior and evaluate whether produced quantiles track the target quantile needed for coverage.
- Synthetic score sequences: The synthetic data include change points and other behaviors that are difficult to produce using real data.
- Synthetic score sequences: For synthetic sequences without ground-truth outcomes, the experiments plot scores and algorithm-produced quantiles instead of prediction sets.
- Synthetic score sequences: The target behavior is for q_t to track the 1 − α quantile of s_t, with q_t ≥ s_t indicating coverage.
- Synthetic score sequences: The synthetic study considers i.i.d., noisy increasing, and mixed change-point-and-trend score sequences.
- Synthetic score sequences: Reported coverage values include 90.0%, 89.8%, and 89.7% across several learning-rate configurations.
- Synthetic score sequences: Other reported configurations achieve coverage values from 89.5% to 90.1%.