PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Agnostic Online Certificate-Driven Calibration for Time Series Forecasting Under Distribution Shift

Chenfeng Huang, Zixuan Ma, George Michailidis

arXiv:2609.31960v1cs.LG

TL;DR

Time-series forecasting under distribution shift is difficult to certify when serial dependence and delayed outcomes undermine standard assumptions. OMPB uses martingale PAC-Bayes certificates to regularize online calibration, and the paper reports improved adaptation stability with a fixed-backbone fallback.

  • Problem

    Standard PAC-Bayes analyses rely on independent sampling and distributional stability, assumptions violated by serial dependence and nonstationary shifts in time series.

    Source: §1 · p. 1

  • Method

    OMPB uses martingale PAC-Bayes certificates as surrogate regularizers for online calibration of a gated Bayesian head on a fixed forecasting backbone.

    Source: §1 · p. 2

  • Results

    The certificate-regularized gated residual Bayesian head improves adaptation stability while retaining the fixed backbone as a safe fallback.

    Source: §1 · p. 2

  • Takeaways & Limitations

    When adaptation is uncertain, the gate allows predictions to revert to the backbone baseline.

    Source: §1 · p. 2

    The certificate is defined for bounded proxy quantities, not the raw MAE and MSE reported in experiments.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

Time series out-of-distribution generalization requires forecasters to remain reliable when deployment dynamics differ from training conditions due to covariate shift, concept shift, and temporal dependence. Probably Approximately Correct Bayesian domain adaptation provides computable certificates by decomposing target risk into a source risk term, a source-to-target mismatch term, and a complexity term, but standard analyses rely on independent sampling and distributional stability, assumptions that are violated in time series by serial dependence and nonstationary shift. We propose a model-agnostic online martingale Probably Approximately Correct Bayesian framework that yields finite-sample certificates under temporal dependence and distribution shift. The certificate replaces independent-sample concentration with martingale concentration that adapts to loss scale and predictable variation. We use the certificate as a surrogate regularizer for online calibration by training a gated residual Bayesian head on top of a fixed forecasting backbone, producing a corrective update that reverts to the backbone prediction when the gate is closed. Online calibration combines a source risk anchor, a posterior-shift penalty, and a time-adaptive mismatch term computed from target windows observed before forecasting. It follows a predict-then-update protocol in which outcomes become available only after forecasting and are used to update subsequent predictions. Experiments across convolutional, attention-based, and large language model-based forecasters show improved stability and accuracy under covariate and concept shift.

1 INTRODUCTION

Time-series forecasters face covariate and concept shifts, while serial correlation and delayed outcomes complicate shift detection and online adaptation. OMPB uses martingale PAC-Bayes certificates to regularize a gated Bayesian head, with mismatch signals from antecedent target windows and the backbone as a fallback.

1 INTRODUCTION

  • Covariate and concept shifts can alter inputs or predictive mappings, while serial correlation limits the effective information for distinguishing drift from noise.Seasonality and sensor noise can shift inputs without changing mechanisms; regime changes and interventions can change the predictive mapping.Source: §1 · p. 1
  • Delayed outcome feedback can lag the current regime, creating tension between frequent adaptation and reliable supervision.The method therefore detects changes from inputs available at prediction time and regularizes later updates against delayed-supervision noise.Source: §1 · pp. 1–2
  • OMPB computes a time-adaptive mismatch signal from antecedent target windows, enabling shift detection without instantaneous outcome feedback.The signal uses posterior-weighted prediction disagreement over strictly antecedent windows.Source: §1 · p. 2
  • A gated residual Bayesian head calibrates a fixed backbone conservatively, reverting to its baseline prediction when adaptation is uncertain.The certificate is used as a surrogate regularizer for online calibration.Source: §1 · p. 2
  • OMPB replaces independent-sample concentration with martingale PAC-Bayes regularization to provide finite-sample guarantees for time-series forecasting under distribution shift.The framework uses the certificate as a surrogate objective for online calibration.Source: §1 · p. 2

2 RELATED WORK

Prior work develops PAC-Bayes generalization and domain-adaptation bounds, dependent-stream bounds, and certificate-based learning objectives. The section notes practical difficulties in applying mixing-based bounds and distinguishes the forecasting shift surrogate from the exact classification disagreement decomposition.

Time Series OOD Generalization Bounds

  • Mixing-based bounds address dependent, nonstationary streams, but estimating mixing coefficients is difficult and their constants can loosen under regime shifts.The path-dependent form uses worst-case dependence and realized sequence complexity.Source: §2 · p. 2

PAC-Bayes Generalization and Domain Adaptation

  • PAC-Bayes bounds relate population risk to empirical risk and a KL complexity penalty; domain adaptation adds source-target discrepancy and an irreducible joint-error term not estimable without target labels.The discrepancy can be instantiated as posterior-weighted disagreement.Source: §2 · p. 3
  • For forecasting, OMPB uses a bounded prediction-discrepancy surrogate for shift sensing rather than the exact binary disagreement underlying the classification decomposition.The surrogate functions as an input-driven regularizer.Source: §2 · p. 3

PAC-Bayes Certificates as Surrogate Objectives

  • PAC-Bayes certificates have been optimized as regularized learning objectives, including differentiable objectives for probabilistic neural networks and iterative surrogates designed to reduce optimization costs.This line motivates using a certificate as an active regularizer for deployment-time calibration.Source: §2 · p. 3

3 PRELIMINARIES

Time-series OOD settings include shifts in input marginals and shifts in predictive mappings. The paper replaces i.i.d. concentration with martingale bounds for dependent streams and extends the approach to unbounded forecasting losses using a conditional sub-gamma assumption.

3.1 DISTRIBUTION SHIFT CATEGORIES

  • Covariate shift changes the input marginal while leaving the conditional predictive distribution invariant; concept shift changes the predictive conditional while the input marginal remains stable.Examples include sensor or seasonal changes for covariate shift and interventions or regime changes for concept shift.Source: §3.1 · p. 4

3.2 MARTINGALE PAC-BAYES DOMAIN ADAPTATION

  • Because time-series training data may be dependent and nonstationary, i.i.d. concentration can yield overly optimistic certificates; OMPB instead uses martingale concentration for serially correlated streams.The labeled source sequence is modeled as a filtered stochastic process.Source: §3.2 · p. 4
  • The bounded-loss martingale result uses a bounded-increment assumption, with B set to 1 for losses in [0, 1].Its variance term represents cumulative conditional variance of the loss process.Source: §3.2 · p. 4
  • For unbounded forecasting losses, OMPB replaces bounded increments with a conditional sub-gamma assumption using predictable variance and a scale parameter for sharper Bernstein-type concentration.The resulting guarantee preserves the tractable PAC-Bayes structure.Source: §3.2 · p. 4
  • Figure 1 depicts offline training separately from online calibration, where predictions precede Bayesian-head updates.This predict-then-update ordering is described as ensuring causal validity.Source: Figure 1 · p. 5

4 ONLINE MARTINGALE PAC-BAYES FRAMEWORK (OMPB)

OMPB freezes a source-trained forecasting backbone and recursively calibrates a lightweight Bayesian head under a predict-then-update protocol. Its online certificate combines source risk, martingale correction, and input-side mismatch estimates to regularize calibration without using target labels before forecasting.

4 ONLINE MARTINGALE PAC-BAYES FRAMEWORK (OMPB)

  • At deployment, the backbone stays frozen while the Bayesian head is updated recursively; each forecast precedes outcome reveal and any update, preventing look-ahead bias.The method uses recent unlabeled target inputs for shift sensing and labeled source replay as a stability anchor.Source: §4 · p. 5, §4.3 · pp. 5–6

4.1 GATED RESIDUAL BAYESIAN HEAD

  • The gated residual Bayesian head corrects the backbone forecast while near-zero residual parameters and a small gate preserve the backbone as a safe default.The posterior is placed only on lightweight head parameters, keeping the complexity term tractable.Source: §4.1 · p. 5

4.2 OFFLINE TRAINING

  • Offline training fits the backbone on labeled source data, then optimizes the head using source forecast loss plus KL regularization to its prior.The KL term is normalized by the number of head-training windows, with a fixed training size across datasets.Source: §4.2 · p. 5

4.3 ONLINE CALIBRATION

  • The online certificate combines empirical source-replay risk, a martingale correction, and source–target disagreement estimated from source windows and currently observed unlabeled target windows.It omits the residual labeling-mismatch term because that term depends on target labels unavailable at prediction time.Source: §4.3 · pp. 5–6
  • The certificate uses a bounded proxy discrepancy rather than directly upper-bounding raw MAE or MSE; its mismatch signal depends on posterior forecast dispersion.The disagreement is normalized by horizon and target-variable count; its informativeness requires meaningful posterior epistemic diversity.Source: §4.3 · pp. 5–6

5 PERFORMANCE ASSESSMENT

The evaluation uses three time-series benchmarks spanning load forecasting, influenza surveillance, and global weather, with MAE and MSE as primary point-forecast metrics. OMPB is assessed against online adaptation baselines across three types of forecasting backbones.

5.1 EXPERIMENTS SETUP

  • Experiments use ETTh for hourly electricity load, ILI for epidemiological tracking, and WEATHER-5K for global climatological modeling.The benchmarks represent complementary shift patterns and temporal resolutions.Source: §5.1 · p. 6
  • Main-text comparisons report MAE and MSE against SOLID, OneNet, and PROCEED, using a one-step feedback delay.Probabilistic quality is additionally evaluated with NLL, CRPS, interval coverage and width, and ECE.Source: §5.1 · pp. 6–7
  • OMPB is evaluated with TCN, Autoformer, and GPT4TS backbones representing convolutional, attention-based, and LLM-based forecasting.Online calibration hyperparameters are fixed before target evaluation and are not tuned on target labels.Source: §5.1 · p. 7

A. Covariate Shift Experiment Result:

The ETTh evaluation studies covariate shift by training on ETTh1 and testing on both its in-distribution split and ETTh2. OMPB improves both in-distribution and shifted forecasting results across the tested horizons and backbones.

A. Covariate Shift Experiment Result

  • OMPB achieves the best results across all tested horizons and backbones, improving both ID and OOD metrics under ETTh1-to-ETTh2 covariate shift.Table 1 reports MAE and MSE at H=24, 96, and 336.Source: A. Covariate Shift Experiment Result · p. 7
  • At H=24, TCN MAE rises from 0.4588 on ID ETTh1 to 2.5145 on OOD ETTh2 without adaptation.Unadapted forecasting error remains large at longer horizons.Source: A. Covariate Shift Experiment Result · p. 7
  • Competing baselines reduce OOD error inconsistently and can sacrifice ID accuracy, while OMPB consistently improves GPT4TS on both distributions across horizons.The passage reports especially large OOD MAE reductions for TCN and substantial reductions for Autoformer.Source: A. Covariate Shift Experiment Result · p. 7

B. Concept Shift Experiment Result:

Under the ILI pre-COVID-to-COVID concept shift, OMPB reduces ID and OOD errors across tested backbones and horizons. WEATHER-5K visualizations and ablations further examine near/far covariate shift and the components of the calibration method.

B. Concept Shift Experiment Result

  • OMPB achieves the best results across backbones and horizons under the ILI concept shift, consistently reducing ID and OOD errors.OneNet becomes unstable for TCN across horizons; SOLID and PROCEED provide inconsistent gains sensitive to horizon and backbone.Source: B. Concept Shift Experiment Result · pp. 7–8, Table 2 · p. 8

C. Near vs. Far OOD Generalization

Ablation Results

  • Removing the gate usually modestly degrades performance, while removing the PAC-Bayes certificate or online calibration consistently degrades it.The gate can be comparable under limited data or stronger shift; the ablations report MAE and MSE at H=24.Source: Table 3 · p. 9

D. Ablation Study:

The ablations favor certificate-driven deployment-time calibration, while the discussion frames the certificate as a monitoring signal rather than a direct MAE/MSE bound. The appendix develops PAC-Bayes bounds for dependent streams and a bounded regression proxy, whose tail interpretation has explicit limits.

D. Ablation Study:

  • Removing the certificate or online calibration consistently degrades performance; removing the gate usually causes modest degradation, with exceptions under limited data or stronger shift.The ablations remove the gate, certificate, or deployment-time calibration stage to test their contributions.Source: D. Ablation Study · p. 9

6 DISCUSSION

  • On WEATHER-5K, the certificate tracks recurring shifts and local error spikes, serving as a monitoring signal rather than a direct MAE/MSE bound.It combines a source-risk anchor, martingale correction, and input-side disagreement penalty.Source: §6 · p. 9

A.1 PROOF OF THEOREM 1

  • Theorem 1’s proof starts from bounded losses on an i.i.d. sample and applies the standard McAllester PAC-Bayes argument.The proof uses an exponential-moment bound and a change-of-measure argument.Source: §A.1 · p. 13

A.2 PROOF OF THEOREM 2

  • Theorem 2’s proof combines PAC-Bayes control of source risk with a Gibbs domain-adaptation decomposition, using a classification identity that relates errors to predictor disagreement.The binary-classification identity explains the factor of two on the disagreement term.Source: §A.2 · p. 14

A.3 PROOF OF THEOREM 3

  • Theorem 3 bounds source-risk estimation error for bounded losses with a PAC-Bayes-Freedman martingale correction.The proof reuses the domain-adaptation decomposition and applies the martingale bound uniformly over posteriors.Source: §A.3 · pp. 14–15

A.4 PROOF OF THEOREM 4

  • Theorem 4 replaces bounded increments with a conditional sub-gamma assumption, yielding a variance-adaptive Bernstein-type martingale bound.Its proof applies a PAC-Bayes change-of-measure argument and optimizes the bound over its tuning parameter.Source: §A.4 · p. 15

A.5 REGRESSION INSTANTIATION AND SURROGATE INTERPRETATION

  • The regression instantiation uses bounded forecasting loss and prediction-discrepancy proxies; a certificate on proxy risk bounds the probability of threshold-exceeding forecast error.The tail interpretation follows for the bounded truncated-square proxy loss.Source: §A.5 · p. 16, Define the residual proxy mismatch · p. 17

Define the residual proxy mismatch

  • The online certificate controls bounded proxy quantities, not raw MAE or MSE, and its input-side mismatch cannot identify strong label-mechanism shifts with little disagreement.The proxy is less informative when the threshold makes errors saturate or leaves moderate errors barely affected; the MAE/MSE improvements are empirical validation, not consequences of a raw-loss bound.Source: Define the residual proxy mismatch · pp. 16–17

A.6 DATASET DETAIL

The dataset splits pair source-domain training and in-domain tests with same-sized out-of-distribution windows. The reported shift metrics show distributional separation across ETTh, ILI, and WEATHER-5K, with Miami–NYC the strongest weather-station shift.

A.6 DATASET DETAIL

  • ETTh and ILI use 80% of the source period for training and 20% for in-domain testing, while WEATHER-5K uses 90% of Miami; each setup samples equally many out-of-distribution windows.The weather test windows come from Atlanta and New York City.Source: §A.6 · p. 18

A.7.1 ETTh

  • ETTh1 versus ETTh2 has mean KS 0.751, Wasserstein 15.20, JS 0.455, and MMD 0.867, indicating substantial covariate shift.Figure 4 visualizes feature densities and a PCA projection for the two datasets.Source: Table 6 · p. 18, §A.7.1 · p. 18, Figure 4 · p. 19

A.7.2 ILI

  • Pre-COVID versus COVID ILI has mean KS 0.644, Wasserstein 180,774, JS 0.352, and MMD 1.315, indicating substantial distributional mismatch.Figure 5 shows feature densities and a PCA projection for the two periods.Source: Table 7 · p. 19, §A.7.2 · p. 19, Figure 5 · p. 20

A.8 BACKBONE TRAINING DETAIL

The appendix details backbone training and epoch selection, then evaluates probabilistic forecasting, proxy certificates, temporal certificate behavior, and monitoring-signal correlations. Across these diagnostics, OMPB improves probabilistic metrics and its certificate tracks shifts and realized errors, while remaining a monitoring signal rather than a tight MAE/MSE bound.

A.8 BACKBONE TRAINING DETAIL

  • Backbones use distinct architectures and training setups, while epoch sensitivity selects each backbone’s epoch by its frequency of best results across horizons and splits.The comparison with ID-validation early stopping is reported at H=24; the supplied passages do not give its numerical outcomes.Source: §A.8 · pp. 20–21

B.1 PROBABILISTIC FORECASTING METRICS

  • OMPB achieves the best NLL and lowest ECE in all six ETTh ID/OOD settings, and the best CRPS in five of six.Original models can retain narrow intervals while becoming severely under-covered; some adaptive baselines recover coverage only with very wide intervals.Source: §B.1 · p. 23, Table 15 · p. 23

B.2 REGRESSION-PROXY DIAGNOSTICS

  • In all four representative settings, the full martingale proxy certificate is tighter than the variance-blind i.i.d. certificate.This diagnostic includes a post-hoc residual mismatch term unavailable at prediction time; the online algorithm instead uses source proxy risk, martingale correction, and unlabeled input-side disagreement.Source: §B.2 · p. 23, Table 16 · p. 24, Figure 7 · p. 24

B.3 CERTIFICATE DIAGNOSTICS

  • The certificate spikes near an abrupt ETTh shift, rises gradually through ILI’s COVID-period shift, and follows recurring seasonal shifts on WEATHER-5K.These trajectories illustrate monitoring responses to distinct temporal patterns, not a direct MAE/MSE bound.Source: §B.3 · p. 24, Figure 8 · p. 25

B.4 CORRELATION AND SCATTER DIAGNOSTICS

  • The online certificate is positively correlated with realized target error across datasets, while disagreement is positively associated with most external shift metrics.ETTh and ILI have stronger Pearson certificate correlations; WEATHER-5K has stronger Spearman correlations, consistent with a monotonic but nonlinear relationship under seasonal shift. Disagreement is most informative when posterior samples retain epistemic diversity and its source-side scale is not saturated.Source: §B.4 · p. 25, Table 17 · p. 25, Table 18 · p. 25

B.5 SENSITIVITY ANALYSIS

Sensitivity analyses find that forecast accuracy is stable across tested disagreement scales, posterior sample counts, and moderate variance-proxy misspecification. Boundedness remains essential: removing clipping breaks the proxy-risk interpretation and produces unstable values.

Disagreement scale and posterior-sample sensitivity

  • MAE stays between 0.6728 and 0.6737 as the disagreement-scale factor varies from 0.25 to 64 in the representative ETTh/TCN/H=96 setting.The scale primarily changes certificate and disagreement magnitudes rather than point prediction.Source: §B.5 · p. 26, Table 19 · p. 26, Figure 10 · p. 27
  • Increasing posterior samples from K=2 to K=32 changes MAE by only 4.27 × 10−7 and MSE by 3.24 × 10−6 in the representative ETTh/TCN/H=96 setting.Runtime increases at K=32 because it requires 496 pairwise comparisons.Source: §B.5 · p. 26, Table 20 · p. 26, Figure 10 · p. 27

Variance proxy and disagreement normalization

  • Moderate variance-proxy misspecification mainly changes certificate conservativeness, while clipped and normalized disagreement remain stable and the unclipped variant is numerically unstable.The unclipped variant violates the bounded-proxy requirement; removing clipping breaks the proxy-risk interpretation.Source: §B.5 · pp. 26–27, Table 21 · p. 27, Table 22 · p. 27

B.6 FEEDBACK-DELAY ANALYSIS

The extended covariate-shift experiments report lowest errors for OMPB across ETTh and WEATHER-5K settings. Delay and posterior-miscalibration diagnostics show that stale supervision worsens accuracy and posterior variance changes certificate scale and selectivity.

B.6 FEEDBACK-DELAY ANALYSIS

  • Accuracy worsens as supervised feedback delay grows from one step to 2H, even though current target inputs remain available for shift sensing.The analysis separates immediate input-side monitoring from labels that arrive later and may describe a regime that has already moved on.Source: §B.6 · p. 27, Table 23 · p. 28, Figure 11 · p. 28

B.7 POSTERIOR MISCALIBRATION

  • Scaling posterior variance while holding its mean fixed changes certificate magnitude and selectivity, while the main temporal shift patterns remain visible.Collapsed variance can understate epistemic uncertainty; inflated variance can overstate uncertainty in benign periods, so the certificate is not a tight MAE/MSE replacement.Source: §B.7 · p. 29, Figure 12 · p. 29

B.8 EXTENDED ETTH EXPERIMENT RESULT

  • Across all ETTh models and horizons, OMPB achieves the lowest errors in both ID and OOD evaluations, with improvements most pronounced on OOD ETTh2.The shift-related error gap generally widens at longer horizons, especially for TCN and Autoformer.Source: §B.8 · p. 30, Table 24 · p. 30

B.9 EXTENDED WEATHER-5K NUMERICAL RESULT

  • Across all WEATHER-5K horizons and backbones, OMPB achieves the lowest MAE and MSE on Miami, Atlanta, and NYC.Improvements are largest under the farther NYC shift and longer horizons; Autoformer is unstable under SOLID and PROCEED, while OMPB remains stable.Source: §B.9 · p. 30, Table 25 · p. 31

Related papers

Loading 2609.31960v1…