PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundations without Fundamentals: Zero-Shot Blind Spots in Time Series FMs

Nafiseh Ghoroghchian, Haipeng Zhang, Shuyi Han, Alex Labach, George Stein

arXiv:2610.02058v1cs.LG

TL;DR

Basic temporal reasoning in TSFMs, especially use of predictive covariates, remains under-examined by broad benchmarks. This paper introduces SimpleTimeBench to test fundamental patterns and finds zero-shot blind spots, while fine-tuning improves specific tasks without producing balanced capabilities. The reported covariate-underutilization also appears in real-world evaluations.

  • Problem

    Broad forecasting benchmarks do not clearly reveal whether TSFMs master basic temporal patterns or exploit informative past covariates.

    Source: §1 · pp. 1–2

  • Method

    SimpleTimeBench tests univariate and multivariate trends, periodic signals, and leading covariates, alongside targeted Chronos-2 fine-tuning and real-world covariate evaluations.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Chronos-2, Moirai, and Toto show zero-shot failures on exponential growth and leading indicators, while full-suite Chronos-2 fine-tuning yields only marginal overall gains and leaves covariate use weak.

    Source: §1 · p. 1, §2 · p. 3, §3 · p. 3

  • Takeaways & Limitations

    Real-world evaluations also find little benefit from covariates in many multivariate TSFM results, limiting their reliability where leading indicators matter.

    Source: §4 · p. 4, §5 · p. 4

    The authors caution that current pre-training regimes may lack inductive biases for exogenous leading indicators, and that public benchmarks may need explicit lead-lag relationships to assess this ability.

    Source: §5 · p. 4

Abstract

from arXiv · show

Despite the success of Time Series Foundation Models (TSFMs) on broad benchmarks, their ability to internalize basic temporal logic, especially in settings supported by exogenous covariates, remains under-examined. We introduce SimpleTimeBench, a diagnostic univariate and multivariate "unit test" suite for primitives such as monotonic trends, periodic signals and leading indicator covariates, scenarios where near-perfect forecasts should be trivial. Surprisingly, prominent multivariate TSFMs (Chronos-2, Moirai and Toto) frequently produce suboptimal zero-shot forecasts for these inputs. While fine-tuning Chronos-2 improves its behaviour on specific tasks, we show that this adaptation degrades performance on other fundamental patterns rather than enhancing its generalizable foundational capabilities. This reveals a gap between pre-training scale and basic temporal reasoning, suggesting that current TSFMs could potentially lack the inductive biases needed to capture simple predictable functions. We further demonstrate that these failures are not merely synthetic curiosities: they persist in real-world sensor forecasting, where TSFMs consistently underutilize leading indicators available in observed covariates. This inability to capture simple relationships limits the practical utility and reliability of current multivariate models.

1. Introduction

Broad benchmarks aggregate performance across many series but can obscure whether TSFMs master basic temporal logic, particularly when past covariates predict future targets. SimpleTimeBench probes these capabilities and reveals zero-shot failures on exponential growth and leading indicators.

1. Introduction

  • Aggregate metrics across thousands of series do not clearly identify fundamental weaknesses in TSFMs.The concern applies to benchmarks spanning univariate and multivariate forecasting settings.Source: §1 · p. 1
  • SimpleTimeBench tests whether TSFMs extrapolate trends, continue periodic signals, and exploit leading indicators in transparent, simple tasks.These tasks are designed so classical statistical models or human interpolation should forecast near-perfectly.Source: §1 · p. 1
  • Chronos-2, Moirai, and Toto struggle zero-shot with simple exponential growth and leading-indicator covariates.Figure 1 highlights failures on the exponential dataset and leading-covariate tests, including cases with a perfect leading indicator.Source: §1 · p. 1, Figure 1 · p. 2
  • Prior work has not systematically unit-tested foundational logic across univariate, multivariate, and covariate-supported settings.The stated gap also includes whether models exploit purely past informative variables such as deterministic leading indicators.Source: §1 · p. 2

2. SimpleTimeBench: Evaluating Zero-Shot Temporal Logic

SimpleTimeBench isolates basic forecasting capabilities through univariate, multivariate, and leading-covariate tests, then diagnoses forecast errors beyond standard error metrics. Its zero-shot evaluation finds shared mean-reversion, anti-exponential, and covariate-underutilization failure modes across leading TSFMs.

Univariate mode

  • The univariate mode tests extrapolation of deterministic trends, periodic signals, and noise processes.Examples include linear and exponential trends and sine waves.Source: §2 · p. 2

Multivariate mode

  • The multivariate mode tests whether adding independent series from the same process improves forecasts or distracts models.Each dataset uses multiple independent variates, with n = 2 by default.Source: §2 · p. 2

Leading-covariate mode

  • Leading-covariate tests provide exact, backward-shifted copies of the target to measure use of predictive information.The construction represents an ideal leading indicator that deterministically reveals the target’s future.Source: §2 · p. 2

Benchmark design

  • Each mode spans 28 generative processes organized into five regimes.Datasets contain independent series drawn from the relevant distribution.Source: §2 · p. 2

Diagnostic metrics

  • The diagnostic metrics include relative MAE, dynamics measures, and directional-bias measures.Relative MAE below 1.0 indicates improvement over naive baselines; MAR and PVR compare predicted dynamics with ground truth, while MBR measures directional alignment relative to the historical mean.Source: §2 · p. 2

Zero-shot results

  • Chronos-2, Toto, and Moirai show mean-reversion, anti-exponential, and covariate-underutilization failures across the shared benchmark tasks.On trend and periodic tasks, MBR is 0.69, 0.66, and 0.80, respectively, against an ideal of 0.5; average PVR is 1.90, 1.88, and 2.20, respectively, against an ideal of 1.0. For Chronos-2 on random walk, relMAE changes only from 1.02 in univariate mode to 0.98 with the leading covariate.Source: §2 · p. 3

3. Nature vs. Nurture: Addressing the Zero-Shot Gap via Fine-tuning

Targeted fine-tuning of Chronos-2 tests whether its zero-shot failures reflect correctable behavior or more fundamental limits. Task-specific tuning improves exponential and covariate performance, but full-suite tuning yields only marginal gains and does not produce balanced capabilities.

Deterministic trends

  • Deterministic-trend fine-tuning retains high performance on constant, linear, and staircase trends while improving exponential-family MBR by 26%.Source: §3 · p. 3

Exponential trends

  • Fine-tuning on univariate exponential data improves MAPE, MAR, and PVR by more than 50% and generalizes to multivariate and leading-covariate settings.The experiment tests whether input normalization inhibits modeling unbounded growth.Source: §3 · p. 3

Leading covariates

  • Leading-covariate fine-tuning shows the architecture can support covariate use, with respective boosts of 17%, 24%, and 24% in MAPE, relMAEmean, and relMAElast.Training used leading-covariate datasets with IID white noise and stochastic correlated processes.Source: §3 · p. 3

Full-suite fine-tuning

  • Full SimpleTimeBench fine-tuning yields only marginal overall gains and still fails to effectively leverage informative covariates.The authors suggest a possible architectural bottleneck in learning diverse temporal primitives simultaneously, while noting extended tuning or alternative optimization might mitigate the issue.Source: §3 · p. 3

4. From Synthetic to Real-World Impact

Real-world evaluations find that multivariate TSFMs often gain little from past covariates on GIFT-Eval. In targeted leading-indicator tests, a small DLinear model benefits substantially more than Chronos-2 on river and stock-price data.

GIFT-Eval results

  • On GIFT-Eval, Chronos-2 and Moirai win barely more than half the multivariate-versus-univariate comparisons, while Toto’s wins have negligible margins.Toto’s margins are 1.2% in MASE and 1.5% in CRPS.Source: §4 · p. 4, Table 3 · p. 4

Targeted leading-indicator tests

  • DLinear gains 24.2% in MASE on Delaware River data and 62.0% on NVIDIA data with leading-indicator covariates, versus Chronos-2 gains of 5.9% and 6.3%, respectively.DLinear has 840 parameters and was trained for a few minutes on a CPU.Source: §4 · p. 4

Implications

  • Aggregated public-benchmark metrics can mask Chronos-2’s limited response to leading information in covariates.The targeted comparisons show DLinear exploiting those signals more effectively on the two evaluated datasets.Source: §4 · p. 4

5. Discussion

SimpleTimeBench isolates fundamental forecasting behavior through controlled tasks whose solutions should be transparent. The authors caution that TSFMs may underuse leading indicators and recommend interpreting their zero-shot capabilities carefully, particularly in high-stakes settings.

5. Discussion

  • Current TSFMs may lack inductive biases for using exogenous leading indicators, warranting caution in high-stakes forecasting.The discussion calls for broader primitive-focused synthetic data and benchmarks with explicit lead-lag relationships.Source: §5 · p. 4

A. SimpleTimeBench

  • SimpleTimeBench tests fundamental forecasting primitives with controlled synthetic tasks and metrics designed to expose interpretable failure modes.It focuses on cases where solutions should be transparent and trivial for classical methods.Source: A. SimpleTimeBench · p. 6
  • The benchmark aims to help practitioners probe model characteristics rather than rank models on a leaderboard.It isolates simple components from noisy, complex data as part of a broader range of forecasting tasks.Source: A. SimpleTimeBench · p. 6

A.1. Datasets

SimpleTimeBench generates series from 28 processes with known ideal forecasts, organized into five generative regimes. It samples univariate, multivariate, and leading-covariate task types, with a 48-step default lag for leading covariates.

A.1. Datasets

  • 28 generative processes are designed to have known ideal forecasts, allowing model outputs to be compared with expected solutions.A process is “solvable” either through exact prediction or by recognizing when only an appropriate distribution can be forecast.Source: §A.1 · p. 6
  • The five regimes cover deterministic trends, periodic and evolving-periodic signals, IID white noise, and stochastic correlated processes.They span parametric trends, fixed or changing oscillations, independent samples, and non-stationary correlated randomness.Source: §A.1 · p. 6
  • Table 1 lists the distributions in each generative regime and an assumed forecasting difficulty.Source: §A.1 · p. 6

A.1.1. TASK TYPES

  • Figures 3–5 show samples for univariate, multivariate, and leading-covariate tasks, which use a 48-step default leading-covariate lag.Source: §A.1.1 · p. 6

A.2. Metrics

The metrics diagnose forecast pathologies that aggregate error measures do not distinguish, using comparisons with simple references. They characterize baseline performance, amplitude, path volatility, and mean-direction behavior rather than serving as standalone leaderboard targets.

A.2. Metrics

  • Diagnostic metrics target systematic bias, insufficient path volatility, and attraction toward historical means, while aggregate CRPS does not distinguish these pathologies.The metrics are referenced to straightforward values and are generally not intended as leaderboard targets because no individual metric captures overall forecast accuracy.Source: §A.2 · p. 6
  • relMAElast compares model MAE with persistence, while relMAEmean compares model MAE with a historical-mean forecast; values above 1 indicate worse performance than the respective baseline.Source: §A.2 · p. 10
  • MAR compares target and forecast deviations from the historical mean to diagnose forecast amplitude; values above 1 indicate muted moves and values below 1 indicate oversized moves.Source: §A.2 · p. 10
  • PVR compares forecast and target first differences to diagnose path volatility; values above 1 indicate smoother forecasts, while values below 1 indicate excessive jitter.Source: §A.2 · p. 10
  • MBR measures how often forecasts hedge toward the historical mean rather than commit to the target direction; values near 0.5 are ideal, while values above 0.7 suggest mean-hedging.Source: §A.2 · p. 10

A.3. Results

The results evaluate target-variable forecasts across univariate and past-covariate modes, finding that unrelated past covariates can hurt and leading covariates do not significantly help on unpredictable regimes. Chronos-2 can use a future covariate that replicates the target, unlike a leading past-only covariate.

A.3. Results

  • Forecast accuracy is measured only on the target variable, across univariate and past-covariate modes, including leading and non-leading covariates.Multivariate forecasting and future covariates are left for future work.Source: §A.3 · p. 11
  • Experiments use 1024 samples per dataset, sequence length 256, forecast horizon 48, and context length 208; effects of context length are left for future work.The benchmark datasets use GIFT-Eval-compatible format.Source: §A.3 · p. 11
  • Past unrelated covariates can harm predictions, while adding leading-covariate information does not significantly help on unpredictable regimes.Values above 1 on some unpredictable distributions do not necessarily indicate model problems because the ideal forecast may be neither the mean nor the last value.Source: §A.3 · p. 11

A.4. Chronos-2 Future Variate Usage Validation

  • Chronos-2 nearly perfectly predicts random walks when given a future covariate that replicates the target, unlike when the same information is supplied through a leading past-only covariate.The validation compares past-only and future covariates that replicate the target signal without a lag.Source: §A.4 · p. 11

B. Fine-tuning Experiments

B.1. Experimental Design and Research Questions

Four fine-tuning experiments test trend continuation, exponential trends, leading-covariate use, and performance across heterogeneous benchmark distributions. Specific tasks improve, while the all-in-one experiment yields only marginal gains and negligible improvement on leading-covariate distributions.

Bias Experiment

  • Fine-tuning improves deterministic trend performance, testing whether models can maintain constant, linear, exponential, and staircase trajectories rather than revert toward the mean.The experiment uses 100,000 samples per distribution and a 10% validation split.Source: §B.1 · p. 14, Figure 8 · p. 15

Norm Experiment

  • Fine-tuning on univariate exponential data is reported to enable perfect prediction of exponential trends.The experiment uses 100,000 samples with a 10% validation split; Figure 9 reports performance after fine-tuning.Source: §B.1 · p. 15, Figure 9 · p. 15

Covariate-Underutilization Experiment

  • Fine-tuning for covariate underutilization produces clear aggregate improvement across all three metrics when testing use of leading indicators.The experiment uses 100,000 samples per distribution with a 1% validation split.Source: §B.1 · p. 15, Table 7 · p. 16, Figure 10 · p. 16

All-in-One Experiment

  • All-in-one fine-tuning yields a marginal boost, with negligible improvements on leading-covariate distributions.It uses all SimpleTimeBench data to test whether covariate use can be learned without sacrificing performance on other distributions.Source: §B.1 · p. 15

Fine-tuning Experiments

  • Covariate-underutilization validation loss could fall further with continued pretraining, whereas the longer all-in-one run nearly reached a plateau.The authors note that extended training or optimization might yield marginal gains, while arguing the training window is ample for evaluating learning capacity on these simple patterns.Source: §B.1 · p. 17

C. GIFT-Eval Univariate vs. Multivariate Experiments

The GIFT-Eval comparison examines univariate versus multivariate performance across datasets, forecast horizons, and models. Multivariate mode provides inconsistent benefits across datasets.

Experiment Configuration

  • Experiments cover all GIFT-Eval datasets across short, medium, and long horizons, using each model's specified repository context length.The setup uses TimeCopilot.Source: C. GIFT-Eval Univariate vs.… · p. 17

Result Post-processing

Per-Dataset Performance

D. Real-world Datasets Experiments

On Delaware River and NVIDIA data, Chronos-2 outperforms DLinear in univariate mode, but DLinear performs better with covariate support. Zoomed views show Chronos-2 largely ignoring covariates during rapid transitions while DLinear tracks their leading spikes.

Dataset construction

  • The Delaware River covariate is upstream streamflow with a 14-hour peak lag; the NVIDIA leading covariate is the target price shifted forward 16 days.Forecast horizons match these respective lags.Source: D. Real-world Datasets Experiments · pp. 17, 20

Evaluation

  • Evaluation uses expanding windows over the final 20% of each series, with non-overlapping folds equal to the forecast horizon; DLinear's lookback is tuned on preceding validation data.The comparison reports MSE, MAPE, and MASE, with MASE seasonality set to 1.Source: D. Real-world Datasets Experiments · p. 20

Localized covariate impact

  • At rapid transitions, Chronos-2 remains muted and delayed despite covariates, while DLinear's forecast aligns closely with leading covariate spikes.The zoomed views cover ±100 hours around the Delaware River peak and ±30 days for NVIDIA.Source: D. Real-world Datasets Experiments · p. 20, Figure 14 · p. 21

Real-world results

Loading 2610.02058v1…