When the Martingale Never Stops Firing: Anytime-Valid Gating on Real Forecast Streams

Weijia Han; Lisha Qu

Source-linked AI summary

When the Martingale Never Stops Firing: Anytime-Valid Gating on Real Forecast Streams

Weijia Han, Lisha Qu

arXiv:2608.30502v1Aug 31, 2026cs.LGstat.MEstat.ML

TL;DR

Anytime-valid gating depends on exchangeable score streams, a premise rarely tested in dependent, adaptive deployments. This paper evaluates a martingale-gated Kalman adapter on five forecasting streams and finds that 135 of 135 clean-stream runs fired, while robust-update gating preserved the benefit without an anytime-validity claim.

Problem
The paper examines limited evidence for whether anytime-valid gating’s exchangeability premise holds on dependent, adaptive forecast streams.
Method
The study evaluates a conformal test martingale gating online Kalman updates across five forecasting datasets and isolates robust-update and drift-response components.
Results
135 of 135 real clean-stream runs fired, while component controls attributed divergence to the drift response and retained robust-update benefits without an anytime-validity claim.
Takeaways & Limitations
Deployment reports should pair anytime-valid claims with clean-data fire counts, null-calibration controls, and dependence diagnostics, while evaluating guarantee-free components separately.
Takeaways & Limitations
Whiteness diagnostics are inconclusive because Ljung–Box can over-reject under conditional heteroskedasticity.

Abstract

from arXiv · show

Machine learning systems are increasingly corrected while they run, and the decision of when to intervene is increasingly delegated to statistical monitors. Anytime-valid inference promises evidence that can be acted on at any moment, exactly the guarantee this setting needs, and it is moving from theory into deployed monitoring. Conformal test martingales are the change-detection instrument, and Ville's inequality caps their false-alarm probability on exchangeable data. The guarantee is conditional. A deployment inherits it only if the stream it monitors behaves exchangeably. The premise is hardest to satisfy where these monitors are most useful, on dependent data and inside loops where the monitor modifies the learner whose scores it reads. It is also rarely measured. We measure it in a pre-specified case study, where such a monitor gates the online updates of a Kalman adapter correcting frozen time-series foundation models on five forecasting streams. On exchangeable synthetic streams, the same implementation fires in at most 1 of 60 runs. On the real streams, at alpha = 0.05, 135 of 135 clean-stream runs fired. The construction does not explain the firing; the failure comes from the deployed score stream itself. Repeated fires hold the gate's drift response active, and the gated filter amplifies the very transient it was designed to prevent. The component worth keeping makes no validity claim. Huber-style gating of the filter's own updates cuts isolated-spike degradation by an order of magnitude with no dataset specific tuning. Anytime-valid methods proposed for dependent data should therefore be accompanied by null-calibration controls and mechanism traces.

1 A pre-specified martingale-gated deployment

This pre-specified deployment study evaluates whether an anytime-valid conformal martingale gate can safely control online adaptation when monitored innovations may be serially dependent. It motivates the gate with a self-excited weather transient and specifies its score, calibration, decision rules, and streaming protocol.

Motivation: Anytime-valid false-alarm guarantees require exchangeability of the monitored score stream, so the study measures that premise on a deployment frozen before any run.The monitor is intended for online deployment and can act whenever evidence accumulates, but its guarantee is conditional on exchangeability.
Gate construction: The gate uses the filter’s standardized innovation as its nonconformity score and a simple-mixture conformal test martingale over randomized p-values from a 250-step calibration window.It uses K=19 ε-values, threshold 1/α at α = 0.05, and resets on fire.
Gate construction: The decision table classifies steps as normal, huberize, skip, or sustained drift, with drift boosting Q by 10× and applying a covariance soft reset.Drift is checked first using Mt ≥1/α; isolated spikes use a fixed chi-square null threshold at the 99.9th percentile.
Protocol: The protocol covers five datasets, three backbones, and three seeds under stride-one immediate-reveal replay, where consecutive labels overlap in 95 of 96 entries.Test blocks contain 12,922 to 52,117 steps, creating a mechanical source of serial dependence in the scores.

2 Premise failure on real streams and its attribution

The deployed gate fires ubiquitously on clean real-stream runs, violating the firing rate expected under its exchangeability premise. Controls show firing is not specific to one learner, while whiteness diagnostics cannot cleanly attribute the failure because transient heteroskedasticity can produce misleading portmanteau results.

Real-stream firing: 135 of 135 clean-stream runs fired at least once, versus about 7 of 135 expected under Ville’s α = 0.05 bound.All 45 full-gate runs fired; first fires occurred at steps 6 to 4,274, with a median of 460 and 18 to 1,340 total fires.
Attribution: Every SGD-surrogate run fired, showing that firing is not specific to the full gate’s learner.The surrogate uses the same gate driven by a trailing mean-square score on an SGD learner.
Diagnostic limits: 119 of 135 runs rejected Ljung–Box whiteness at p < 0.01, consistent with serially dependent innovations but not conclusive evidence.The portmanteau test can over-reject under conditional heteroskedasticity even for uncorrelated series.
Diagnostic limits: The weather full-gate portmanteau statistic ranged from 0.05 to 0.64 against a χ2_10 null with mean 10, yielding p ≈1 by lower-tail degeneracy.The passage identifies this as an artifact of the very transient traced in Section 3.
Attribution: 60 of 60 drift-free synthetic pipeline runs fired, despite at most 3 of 60 expected under the premise.These runs used smooth deterministic labels plus i.i.d. Gaussian noise, while monitored scores came from the pipeline’s own adaptation; each run had 33 to 42 fires, with first fires inside 67 test-block steps.

3 Fire-triggered amplification of the weather transient

Repeated martingale fires kept the drift response active through a weather transient, causing the gated filter to amplify errors and diverge on some backbones. The event was treated as a normal update because recovery suppression and an adapted conformal window prevented spike-branch activation, while configuration settings compounded the response.

Fire-triggered amplification: The martingale re-fired every 5 to 10 steps, keeping the drift response’s Q-boost above 8 through the elevated-error episode.Ungated per-step MSE reached 8 to 37, versus a typical level below 1.
Fire-triggered amplification: 1.155 × 105 versus 9.8×103: TimesFM 2.5’s event-step MSE under accumulated drift response exceeded the ungated filter’s MSE.Final cumulative MSE rose from 0.36 to between 2.6 and 3.1 across seeds.
Fire-triggered amplification: 6 of 6 clean-stream runs and 54 of 54 contaminated weather runs crossed the magnitude-divergence guard on TiRex and Chronos-2.All clean-stream crossings occurred at step 18,536.
Fire-triggered amplification: The spike branch never activated: the event was recorded as a normal update while recovery-window suppression was active and the conformal p was 0.011.The conformal window had already adapted to the episode, and the gate’s base forgetting factor was pinned at 0.90 below the drift value of 0.95.

4 Component ablations under contamination and drift

The robust-update half preserves spike robustness without the drift response, matching or beating the full gate on three degrading datasets. Sustained contamination produces broad degradation, often making gated arms worse than ungated alternatives.

Robust-update ablation: Robust updates (huberize and skip) deliver spike robustness without drift response, matching or beating the full gate on ETTh2, traffic, and ETTm1.The same gating transfers to SGD through the surrogate score on ETTh2 and ETTm1, but performs worse than ungated SGD on traffic.
Contamination and drift: +7.8 to +19.3: level shift at 5% degrades every arm on ETTh2 and ETTm1, with gated arms often worst.On traffic, every Kalman arm degrades by +9.5 to +18.3.
Contamination and drift: +1.7 to +14.2: the stuck injector degrades the full gate on the four datasets outside weather, versus at worst +1.8 for the ungated filter and +0.5 for SGD arms.The passage also reports that synthetic piecewise drift makes the full gate adapt more slowly, but the supplied text is truncated.

5 Implications for e-value methods

Deployment reports should pair anytime-valid claims with raw clean-data fire counts, null calibration at the deployed horizon, and dependence diagnostics. The detection alternatives described do not remove the need to test the exchangeability premise, while guarantee-free components should be assessed separately.

Reporting implications: Deployment reports should pair every anytime-valid claim with raw fire counts on clean data, null calibration at its horizon, and dependence diagnostics.Guarantee-free components should be evaluated separately from guaranteed ones.
Reporting implications: The robust-update component carried the measured benefit, while the martingale-triggered response caused the damage.Framing the result as anytime-valid gating would credit the benefit to the guarantee and obscure the damage source.
Detection alternatives: 135 of 135 runs alarmed at every tested threshold when e-detectors were replayed on the deployed p-streams.The passage presents this as evidence that swapping detectors does not repair the underlying premise.
Detection alternatives: E-processes can test exchangeability itself and extend across filtrations, while adaptive conformal updating tracks calibration under shift.These methods are presented as related directions rather than repairs demonstrated for the deployed streams.

A Reproducibility and scope

The study fixes a single deployment configuration and supports reproducibility through deterministic replay, pre-specified verdicts, raw p-value counts, and released null-control scripts.

Reproducibility: The authors release deterministic cached-forecast replay, an errata trail for pre-specified verdicts, raw counts beside every p-value, and null-control scripts.These materials are intended to make the reported deployment results reproducible.
Scope: All reported claims use the single gate configuration specified in Section 1, so their scope is limited to that deployment.The passage explicitly frames every claim as deployment-specific.
Reproducibility: Monte Carlo seeds condition on fixed cached streams, while gated arms and contaminated runs receive separate randomized conformal-p-value or RNG draws.Each gated arm draws its own randomized conformal p-values per seed, and each contaminated run spawns its own RNG.

B Component and feedback ablations

The ablations identify immediate full-vector feedback as necessary for the step-18,536 catastrophe, while a martingale-free fixed-threshold cutoff limits isolated-spike degradation but has backbone-dependent clean-weather cost. E-detector replay reproduces all recorded fires and shows severe over-alarming on paper runs despite calibration on i.i.d. streams.

Response components: Soft-reset-only reproduces full-gate divergence in 6 of 6 TiRex and Chronos-2 runs, whereas boost-only diverges in 0 of 9 runs.Soft-reset-only diverges at step 18,536; on TimesFM 2.5, its final MSE range is 2.50 to 3.03 against the full gate’s 2.57 to 3.06.
Delayed feedback: With feedback delayed 96 steps, the full gate diverges in 0 of 9 clean-weather runs, and its final cumulative MSE exceeds the ungated filter on every backbone.The step-18,536 catastrophe therefore requires immediate full-vector reveal.
A martingale-free cutoff: A fixed-threshold huberize-and-skip arm limits isolated-spike degradation to at most +0.05 in every dataset without martingale holds or dataset-specific tuning.Its clean-weather cost ranges from 0.37 against 2.03 on TiRex and 0.23 against 3.58 on Chronos-2 to 3.41 against 0.36 on TimesFM 2.5.
E-detector replay: SR and CUSUM e-detectors reproduce recorded fires in 405 of 405 runs and alarm in 135 of 135 paper runs at thresholds A ∈{103, 104, 105}.For SR at A=104, alarms reach 193/320/1,464 min/median/max per run versus null expected counts of 1.3 to 5.2; detectors remain calibrated on i.i.d. streams.

C Synthetic drift latency

The synthetic-drift study evaluates adaptation latency and final cumulative MSE across three seeded streams with declared breaks and bounded per-break searches. The full gate is slower than the ungated filter on every seed.

Study design: The study uses three seeds, declared break points, and a bounded search after each break.Adaptation latency is measured until trailing-error recovery, with never-recovered breaks capped at the search horizon.
Metrics: Mean adaptation latency counts steps from each declared break until the trailing-error recovery criterion is met.The benchmark also reports final cumulative MSE per arm.
Results: The full gate is slower than the ungated filter on every seed.The comparison is made at the full gate’s motivating task in the synthetic drift benchmark.