PaperlayerView exact revision on arXiv

Source-linked AI summary

Time Series Forecasting Benchmarks Need Scenario-Grounded Stress Testing

Yuyang Zhao, Lian Xu, Hao Xue

arXiv:2610.02608v1cs.AI

TL;DR

TSF benchmarks primarily measure held-out average error, while many deployment failures involve structured events, dependencies, regime changes, and propagation. The paper proposes scenario-grounded testing that specifies failure conditions and difficulty; it concludes that evaluation should ask under what conditions models fail and why.

  • Problem

    TSF benchmarks focus on average held-out error, while existing robustness tests do not represent structured deployment failures and their causal processes.

    Source: §1 · p. 1, §2.3 · p. 4

  • Method

    The paper proposes test instances that pair forecasting data with an explicit parameterized failure operator and an interpretable difficulty score.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    The paper concludes that TSF evaluation should move beyond accuracy to examine the structured conditions under which models fail and why.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Scenario parameters and failure operators make test conditions explicit, so reported performance can be tied to interpretable failure scenarios.

    Source: §1 · p. 2, §4.1 · p. 6

    A metric vocabulary aligned with scenario parameters remains an open task.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

Time series forecasting (TSF) increasingly drives decisions in transportation, energy, finance, healthcare, and infrastructure, yet current evaluation remains overly narrow: standard benchmarks reward low held-out error, while robustness studies typically reduce failure to Gaussian noise, random masking, or bounded adversarial perturbations. This obscures the real failure modes of deployed forecasting systems. Input-side anomalies are not merely noisier inputs: they often reflect structured events that alter temporal dynamics, break cross-variable dependencies, induce regime shifts, or propagate from faulty sensors to downstream decisions. These semantic, causal, and system-level failures cannot be faithfully captured by i.i.d. perturbations alone. The rise of TSF foundation models makes this evaluation gap more urgent, as unauditable pretraining corpora make held-out generalization increasingly unreliable. We therefore advocate scenario-grounded stress testing. Each test instance should include historical inputs and future targets, together with a semantic scenario, an explicit failure operator, and a measurable difficulty level. This shift makes evaluation interpretable, attributable, and deployment-relevant and friendly, enabling the community to ask not only which model is accurate, but under what conditions it fails and why.

1 Introduction

TSF forecasts inform consequential decisions, but benchmarks largely test average error on held-out data or simple perturbations rather than deployment failures with structure. The paper advocates scenario-grounded stress testing that makes failure conditions explicit and interpretable.

1 Introduction

  • Real disruptions can reshape temporal patterns, disturb variable dependencies, change operating regimes, and propagate sensing errors into downstream decisions.The paper cites Winter Storm Uri and COVID-19 lockdowns as examples of structured disruptions.Source: §1 · p. 1
  • Current evaluation centers on held-out average error and robustness tests using Gaussian noise, random masking, or bounded adversarial perturbations.These tests model value-level deviations without representing the event, system state, or causal process that generated them.Source: §1 · pp. 1–2
  • The paper argues that TSF benchmarking should shift from distributional noise to scenario-grounded stress testing with parameterized, causally interpretable failures reported alongside prediction error.Each test instance should reveal the conditions under which a model fails, not only the magnitude of failure.Source: §1 · p. 2
  • The paper identifies five benchmark blind spots and proposes explicit failure operators, four structural failure families, six design principles, and responses to three counterarguments.Source: §1 · p. 2

2 Why Current TSF Benchmarks Fall Short

The paper traces TSF benchmarking's limitations to its evaluation object: average error over a test distribution. It argues that deployment failures require structured, constructed scenarios and explicit reporting of their conditions.

2.1 Definition of the Current Paradigm

  • Current TSF evaluation measures expected loss over a test distribution assumed to match training or differ by mild covariate shift.Progress is measured by average reduction of this expectation across benchmark datasets.Source: §2.1 · p. 3

2.2 The Evolution of TSF Benchmarking

  • Across four benchmark generations, evaluation has remained average error on a chosen distribution, while the proposed next generation centers parameterized failure scenarios.Table 1 summarizes the progression and the proposed reorientation.Source: Table 1 · p. 3, §2.2 · p. 3

2.3 Existing Robustness Research

  • Existing robustness research leaves uncovered failures that are both structured and constructed, despite methods testing noise, masking, adversarial perturbations, and distribution shift.The paper argues this quadrant is prominent in deployments but lacks an established evaluation methodology.Source: Table 2 · p. 3, §2.3 · p. 4

2.4 Structural Blind Spots in Real TSF Failures

  • Real TSF failures share temporal shape, cross-variable structure, mechanism coupling, causal propagation, and attributability, which current benchmarks lack mechanisms to represent.The paper maps these properties to five structural blind spots.Source: §2.4 · p. 4, Table 3 · p. 4
  • The five blind spots concern missing semantically shaped local events, cross-variable dependency disturbances, coupled regime change and missingness, cascading propagation, and exposed scenario parameters.Source: §2.4 · pp. 4–5
  • The paper frames the mismatch as representational: i.i.d. perturbations of clean distributions cannot express the structure of real failure conditions.It argues that addressing the gap requires changing the object of evaluation, not merely enriching the noise model.Source: §2.4 · p. 5

3 Three Arguments for a Paradigm Shift

The paper argues that current TSF evaluation cannot represent structured failures, provide reliable evidence of generalization for foundation models, or answer the condition-specific questions deployment poses.

3.1 Argument I: Structured Failures Are Not Expressible as i.i.d. Perturbations

  • i.i.d. perturbations cannot express temporal shape, cross-variable structure, shared mechanisms, causal propagation, or exposed scenario parameters.The proposed shift is from sampling values from a distribution to constructing instances with an operator and its parameters.Source: §3.1 · p. 5

3.2 Argument II: Held-Out Evaluation Has Lost Its Epistemic Guarantee

  • When pretraining and test data may overlap, strong held-out performance becomes difficult to distinguish from recall.Scenario-grounded evaluation constructs stressed instances at evaluation time using a controlled, previously unexposed Θ, even when the underlying history was seen.Source: §3.2 · p. 5

3.3 Argument III: Deployment Asks a Different Question Than Average Accuracy

  • Deployment requires knowing under what conditions models fail, how severely, whether failures propagate, and what consequences follow—not only average error.Scenario parameters can encode conditions such as heat-wave demand surges, sustained sensor drift, or regime shifts with rising missingness.Source: §3.3 · pp. 5–6

4 Scenario-Grounded Stress Testing: A Framework

The framework replaces ordinary input–target test pairs with scenario-described instances, then specifies how scenario parameters should support interpretable, causal, graded evaluation across structural failure families.

4.1 Formal Definition

  • A test instance becomes a structured quintet built from a clean pair, an explicit failure operator, scenario parameters, and a measurable difficulty score.The workflow applies the operator to generate stressed inputs and targets, then conditions evaluation on Θ.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2 Four Requirements on Scenario Parameters

  • Useful scenario parameters must be semantically interpretable, composable, causally grounded, and linked to difficulty that changes monotonically and predictably.These properties make parameters legible to domain experts, capable of expressing compound failures, mechanism-based, and suitable for graded sensitivity analysis.Source: §4.2 · p. 7

4.3 A Taxonomy of Structural Failure Families

  • The taxonomy crosses bounded local events versus persistent regime changes with single versus interacting variables, yielding four structural failure families.The four families distinguish bounded perturbations, pathological inter-variable structure, persistent shifts with missingness, and cascading propagation; they are a constructive minimum, not an exhaustive catalog.Source: §4.3 · p. 7

4.4 Design Principles for Next-Generation TSF Benchmarks

  • The six design principles operationalize the framework as an evaluation contract for next-generation benchmarks.A compliant benchmark changes what each test instance records—the evaluation conditions as well as the input and target—rather than simply adding harder samples.Source: Table 4 · p. 7, §4.4 · p. 8

5 Alternative Views and Counterarguments

The paper addresses three objections to scenario-grounded stress testing: scaling may cover failures, historical incidents may be more ecologically valid, and exposed parameters may invite gaming. It argues that scenario construction can test novel failures, preserve structural fidelity while expanding sparse coverage, and make overfitting visible.

Alternative View I: Scaling Law Will Solve This

  • Scaling can fit past extremes without testing novel failures or revealing whether a model learned a mechanism rather than memorized a trace.The authors argue scenario-grounded evaluation can hold scaling accountable by constructing failures that have not been seen.Source: §5 · p. 8

Alternative View II: Synthetic Scenarios Are Less Real Than Historical Events

  • Structural fidelity, not historical origin, determines ecological validity; parameterized scenarios can turn sparse incident observations into coverage-adequate evaluation.The paper argues that historical records stripped of causal context may be less valid than structurally faithful synthesized scenarios.Source: §5 · p. 8

Alternative View III: Exposed Parameters Invite Goodhart’s Law

  • Holding out combinations of exposed scenario parameters can test unseen configurations, while transparency makes benchmark overfitting visible rather than preventing it.The authors contrast this with accuracy-centric benchmarks, where gaming may remain concealed.Source: §5 · p. 8

Our Position

  • The three objections do not resolve the need for scenario-grounded evaluation: scale cannot provide attribution, historical fidelity cannot ensure coverage, and transparency exposes gaming.The paper concludes that the evaluation object itself must change.Source: §5 · p. 9

6 Implications and Open Problems

The paper connects scenario-grounded stress testing to current forecasting research and identifies open work on scenario catalogs, foundation-model evaluation, and metrics. These directions include testing uncertainty against structured-failure difficulty and reporting when, where, and whether errors cross operational thresholds.

6.1 Implications for Current Research Directions

  • TSF foundation-model evaluation could use protocols whose validity does not depend on pretraining-data novelty, paralleling behavioral and dynamic benchmarking in NLP.This reframes evaluation beyond preserving held-out integrity against pretraining corpora.Source: §6.1 · p. 9
  • Anomaly-detection methods can construct F1–F4 failure operators, while scenario-grounded benchmarks would provide standardized testbeds for detection research.The paper identifies point, contextual, and collective anomaly types as natural primitives.Source: §6.1 · p. 9
  • Under scenario-grounded evaluation, probabilistic and quantile-based forecasters can be tested on whether predicted uncertainty anticipates structured-failure difficulty δ.The paper presents this as a stronger deployment-calibration criterion than calibration on held-out distributions.Source: §6.1 · p. 9

6.2 Open Problems

  • F1–F4 provide a structural minimum, so shared domain-specific catalogs of ϕ and Θ calibrated to documented failures and cross-domain δ-calibration protocols remain open needs.These are needed to make domain-specific catalogs comparable.Source: §6.2 · p. 9
  • Current TSFMs are zero-shot forecasters whose behavior under structured failure is largely uncharacterized; testing them along F1–F4 could reveal which failure modes scale solves.The paper identifies this testing as a diagnostic basis for understanding foundation-model behavior under structured failure.Source: §6.2 · p. 9
  • Horizon-aggregated MSE, MAE, and probabilistic counterparts treat errors as exchangeable, while scenario-aligned metrics should report timing, propagation, and operational-threshold crossings.Proposed examples include peak-time deviation, time-to-recovery, cross-variable error correlation, exceedance probability, and time-in-violation.Source: §6.2 · p. 9

7 Conclusion

The conclusion argues that TSF benchmarks remain misaligned with consequential deployments because they measure average error on clean distributions while deployed systems encounter structured failures. It calls for evaluation to ask under what conditions models fail and why.

7 Conclusion

  • TSF models inform consequential decisions, but benchmarks still measure average error on clean distributions rather than structured deployment failures.The paper characterizes this as a representational misalignment in what evaluation treats as its object.Source: §7 · p. 9
  • The paper frames the needed change as redefining the evaluation object, rather than building another benchmark dataset.Its closing question is whether evaluation will continue to ask only how accurate models are.Source: §7 · p. 9
  • The conclusion calls for asking under what conditions models fail and why, rather than measuring accuracy alone.It describes this as asking in the structured terms the answer requires.Source: §7 · p. 9

Related papers

Loading 2610.02608v1…