PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Accuracy and Probabilistic Reliability of Zero-Shot Time Series Foundation Models

Panagiotis Michael, Moysis Symeonides, Demetris Trihinas

arXiv:2609.25788v1cs.LGcs.AI

TL;DR

Forecasting research has limited and fragmented evidence on TSFMs, especially regarding probabilistic calibration alongside point accuracy. This paper benchmarks six zero-shot TSFMs across unseen datasets and finds a trade-off between accuracy and reliability, with xLSTM TiRex providing consistent calibration across modalities.

  • Problem

    TSFM evaluations are sparse and fragmented, and the literature emphasizes point accuracy while giving less attention to probabilistic calibration.

    Source: §1 · p. 2

  • Method

    The study compares six zero-shot TSFMs with statistical baselines and PatchTST on newly collected data, evaluating point accuracy and uncertainty reliability.

    Source: §1 · p. 2

  • Results

    TSFMs offer a strong alternative to statistical and lightweight supervised baselines, but accuracy and calibration trade off; xLSTM TiRex has the most consistent, conservative calibration across modalities.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Point accuracy alone is insufficient for deployment decisions, which also depend on uncertainty calibration.

    Source: §5 · p. 9

    The benchmark leaves multivariate forecasting, robustness to missing data, irregular sampling and concept drift, and computational efficiency for future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Time Series Foundation Models (TSFMs) promise a paradigm shift toward zero-shot forecasting by eliminating task-specific training. However, existing works often overlook trade-offs between predictive accuracy and probabilistic calibration. This paper presents a benchmark study of six TSFMs evaluated on energy, traffic, and financial datasets. We contrast their performance against statistical baselines and a supervised DL model. The study reveals that while TSFMs outperform statistical methods and supervised models, they are subject to a fundamental trade-off between point accuracy and probabilistic reliability. Specifically, xLSTM architectures provide robust probabilistic calibration across horizons. In contrast, patch-based transformers offer competitive accuracy but face calibration issues at long horizons, while transformer-based models exhibit context saturation points for optimal zero-shot reasoning. These findings offer evidence-based guidance for balancing generalization and uncertainty quantification in real-world deployments.

1 Introduction

Task-specific forecasting models can require extensive data, compute, and retraining, while evidence on TSFMs remains sparse and fragmented. The paper benchmarks zero-shot TSFMs across accuracy and probabilistic reliability to address that gap.

  • Task-specific DL models require extensive historical data, considerable optimization compute, and frequent retraining under new distributions or domains.Source: §1 · p. 1
  • TSFMs use pretraining to forecast unseen data without task-specific training, offering a potential alternative for cold-start settings.The paper describes mechanisms including token patching, numerical quantization, and generative flow matching.Source: §1 · pp. 1–2
  • Current TSFM evaluations are sparse and fragmented, and often emphasize point accuracy rather than probabilistic calibration.Prior evaluations also report that TSFMs do not exceed finely tuned task-specific baselines and underperform statistical methods for irregular series [12].Source: §1 · p. 2
  • The study benchmarks six TSFMs against statistical baselines and PatchTST, using newly collected data and both point-error and uncertainty-reliability metrics.The reliability measures are Interval Coverage Error (ICE) and Interval Mean Absolute Error (IMAE).Source: §1 · p. 2

2 Related Work

Related work contrasts supervised, task-specific forecasting architectures with TSFMs that use pretraining for zero-shot generalization. It also describes distinct TSFM designs for representing numerical data and modeling uncertainty.

  • Task-specific Transformer models advanced long-range forecasting, but PatchTST remains constrained by domain-specific data and optimization overhead, limiting zero-shot and cold-start use.Informer reduced attention’s quadratic complexity for longer contexts.Source: §2 · p. 2
  • TSFMs use pretraining to generalize across unseen distributions; Chronos-2 [1] quantizes values, while Moirai-2.0 [6] and TimesFM use patch-based autoregressive designs.Moirai-2.0 predicts multiple future patches simultaneously, whereas TimesFM permits output patches longer than input patches.Source: §2 · pp. 2–3
  • Sundial uses flow matching to sample future trajectories and form probabilistic bounds, while TiRex uses an xLSTM backbone to track hidden states over time.Source: §2 · p. 3

3 Experimental Methodology

The methodology evaluates zero-shot forecasting across traffic, energy, and financial data while varying context and horizon. It compares six TSFMs with statistical and supervised baselines using point-accuracy and interval-calibration metrics.

3.1 Datasets

  • The benchmark spans discrete traffic counts, continuous electricity load, and financial prices, with recently curated traffic and grid data outside known TSFM training corpora.The datasets are EdgeTraffic, MixGridPL, and Yahoo! SPY.Source: §3.1 · pp. 3–4

3.2 Problem Description

  • Zero-shot forecasting predicts a future horizon from historical context without gradient updates or fine-tuning on the target dataset.The rolling-window evaluation gives each model the same unseen target window.Source: §3.2 · p. 4

3.3 Experiment Scenarios

  • Two scenarios vary forecast horizon at fixed context or vary context at fixed horizon to assess accuracy scaling and identify context saturation.Context saturation is the point beyond which additional history no longer improves zero-shot reasoning.Source: §3.3 · p. 5

3.4 Benchmark Baselines and TSFMs

  • The benchmark evaluates six TSFMs against statistical baselines and PatchTST, using pre-trained configurations and quantile forecasts for calibration metrics.PatchTST is configured for comparable inference latency; the TSFMs are Moirai-2, TiREX, Chronos-2, Sundial, Toto, and TimesFM.Source: §3.4 · p. 5

3.5 Testbed and Evaluation Metrics

  • sMAPE assesses point accuracy, while ICE and IMAE assess calibration of an 80% prediction interval.ICE measures how often intervals miss their nominal coverage; IMAE measures the magnitude of those violations.Source: §3.5 · pp. 5–6, Figure 2 · p. 6

4 Evaluation

The benchmark reports dataset-specific forecasting accuracy and interval-calibration results, with extended-horizon point accuracy sometimes coexisting with deteriorating uncertainty estimates. Results are presented as dataset-level plots.

4 Evaluation

  • The evaluation presents predictive-accuracy and probabilistic-calibration findings as plots for each dataset in Figures 3–5.Full tabular data and reproduction instructions are available in the benchmark repository.Source: §4 · p. 6

4.1 Predictive Accuracy Evaluation of Forecasting Models

  • 17% sMAPE at H = 10 is the best short-horizon EdgeTraffic result for TimesFM and Chronos-2, versus ARIMA at 22% and RA at 32%.At H = 1800, Chronos-2 records 28%; under FixedH, TimesFM improves from 36% at C = 180 to 21% at C = 1260, then reaches 22.5% at C = 2460, while PatchTST rises from 31.5% to 52%.Source: §4.1 · p. 7, Figure 3 · p. 7
  • 3.92% to 3.95% at H = 12 is achieved by Chronos-2, TiRex, and Sundial on MixGridPL, versus ARIMA at 14% and RA at 13%.At H = 672, Chronos-2 and Moirai-2.0 reach 6.4% and 6.6%, while ARIMA reaches 19%; under FixedH, Chronos-2 improves from 6.70% at C = 120 to 4.4% at C = 624.Source: §4.1 · p. 7, Figure 4 · p. 8
  • At H = 252 on SPY, TiRex reaches 8.34% and Chronos-2 8.64%, improving over ARIMA at 9.68% and RWD at 9.74%.At H = 1, results are nearly identical across RWD (0.79%), TiRex (0.80%), and Chronos-2 (0.82%); under FixedH, TiRex changes from 1.40% at C = 25 to 1.26% at C = 509.Source: §4.1 · p. 7

4.2 Probabilistic Reliability and Interval Calibration

  • Several TSFMs maintain low sMAPE at extended horizons while their uncertainty estimates deteriorate, producing miscalibration.The evaluation uses ICE and IMAE to assess probabilistic reliability.Source: §4.2 · p. 8
  • At H = 1800 on EdgeTraffic, Chronos-2 and TiRex keep ICE below 0.06 across horizons, while TimesFM and Moirai reach ICE values of 0.73 and 0.56.TimesFM and Moirai also reach IMAE values of 2.8 and 3.2; Toto has ICE 0.07, while PatchTST has ICE 0.7 and IMAE 6.8. The passage characterizes Chronos-2 and TiRex's slightly higher average IMAE of ∼0.28 as conservative.Source: §4.2 · p. 8
  • On MixGridPL, TiRex's ICE stays between 0.01 and 0.04 across scenarios, while Sundial and PatchTST remain over-confident at ICE ∼0.45.Under FixedH, Chronos-2's ICE drops from 0.17 at C = 120 to 0.04 at C = 624; Moirai's IMAE rises from 109 to 479 over long-range forecasting.Source: §4.2 · p. 8
  • On SPY, Sundial and PatchTST exceed ICE 0.50, while TiRex reaches ICE 0.01 and IMAE 0.34 at C = 509.The passage identifies TiRex and Chronos-2 as the most robust; it contrasts their calibrated uncertainty with TimesFM and Moirai-2.0, which often lead in sMAPE but become over-confident at long horizons.Source: §4.2 · pp. 8–9, Figure 5 · p. 9

5 Conclusion and Future Directions

The paper concludes that TSFM performance and calibration vary with architecture and signal, so model selection should weigh horizon, volatility, and the cost of underestimated uncertainty. It identifies several extensions for future evaluation.

5 Conclusion and Future Directions

  • TiRex provides the most consistent, conservative calibration across modalities, while Chronos-2 balances competitive accuracy with calibrated intervals.The conclusion contrasts these models with patch-based TimesFM and Moirai-2.0, which often lead in accuracy but can become over-confident at long horizons.Source: §5 · p. 9
  • Future work will extend the benchmark to multivariate forecasting and robustness under missing data, irregular sampling, and concept drift.Computational efficiency, calibration-aware forecasting, and adaptive model-selection strategies are also left for future work.Source: §5 · p. 9
Loading 2609.25788v1…