Source-linked AI summary
fev-bench: A Realistic Benchmark for Time Series Forecasting
Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, Yuyang Wang
TL;DR
Existing forecasting benchmarks often provide narrow task coverage, weak statistical evidence, and inflexible evaluation infrastructure, limiting reliable comparisons as pretrained models advance. The paper introduces fev-bench and the lightweight fev library to evaluate 100 tasks, including covariate settings, with win rates, skill scores, and bootstrap confidence intervals. Chronos-2 leads the benchmark, while TiRex and TimesFM-2.5 are statistically indistinguishable under the reported pairwise analysis.
Problem
Existing benchmarks often overlook covariates, lack statistical rigor for comparing models, and provide inconsistent or inflexible evaluation infrastructure.
Method
The paper introduces fev-bench and the lightweight fev package, using win rates, skill scores, and bootstrap confidence intervals for reproducible benchmark aggregation.
Results
Chronos-2 achieves the highest average win rate and skill score, while TiRex and TimesFM-2.5 show no clear winner in pairwise SQL comparisons.
Takeaways & Limitations
Covariate support is identified as a promising direction for pretrained forecasting models, and fev-bench plus fev provide a practical foundation for model comparison.
Takeaways & Limitations
fev-bench prioritizes broad coverage and fast evaluation over fully leakage-free evaluation, relying on self-reported training overlap as a practical safeguard.
Abstract
from arXiv · showhide
Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly with the rise of pretrained models. Existing benchmarks often have limited domain coverage or overlook real-world settings such as tasks with covariates. Their aggregation procedures frequently lack statistical rigor, making it unclear whether observed performance differences reflect true improvements or random variation. Many benchmarks lack consistent evaluation infrastructure or are too rigid for integration into existing pipelines. To address these gaps, we propose fev-bench, a benchmark of 100 forecasting tasks across seven domains, including 46 with covariates. Supporting the benchmark, we introduce fev, a lightweight Python library for forecasting evaluation emphasizing reproducibility and integration with existing workflows. Using fev, fev-bench employs principled aggregation with bootstrapped confidence intervals to report performance along two dimensions: win rates and skill scores. We report results on fev-bench for pretrained, statistical, and baseline models and identify promising future research directions.
1 Introduction
fev-bench addresses gaps in forecasting benchmarks by expanding real-world task coverage, adding statistically principled aggregation, and providing lightweight reproducible evaluation infrastructure.
- Benchmark gaps: Existing benchmarks often ignore covariates, even though promotional and pricing data are important for accurate retail demand forecasting.This gap has contributed to limited covariate support in pretrained forecasting models.
- Benchmark gaps: Single-number benchmark summaries make it difficult to distinguish genuine model improvements from random variation or changes caused by benchmark composition.Small gains may disappear or reverse under minor benchmark changes.
- Infrastructure gaps: Many benchmarks lack evaluation code or use rigid, dependency-heavy systems, producing inconsistent results and limiting extensibility and workflow integration.These infrastructure problems undermine reproducibility and long-term practical utility.
- New benchmark: 46 of fev-bench’s 100 tasks include covariates across seven domains, addressing a major omission in existing forecasting benchmarks.The benchmark also includes univariate and multivariate settings.
- Aggregation methods: Bootstrap-based confidence intervals quantify whether performance differences are statistically meaningful and test robustness to benchmark-composition changes.The aggregation approach is designed to support more reliable model comparisons.
- Evaluation package: The lightweight fev package minimizes dependencies while remaining compatible with popular forecasting libraries and supporting reproducible, extensible benchmark development.Researchers can build and share new benchmarks and their results.
2 Preliminaries
The paper formalizes forecasting tasks around multivariate observations, future horizons, covariates, and evaluation procedures, while emphasizing that aggregation determines the reliability of overall benchmark comparisons.
- Problem definition: Forecasting predicts H future values for each series from historical observations and covariates available over the relevant time range.Covariates may be static, past-only dynamic, or known dynamic variables extending through the forecast horizon.
- Problem definition: The framework supports univariate and multivariate series, with probabilistic forecasts represented through distributions or predictive quantiles.Point forecasts such as conditional means or medians are also supported.
- Benchmarks and tasks: A forecasting task specifies a dataset, horizon, evaluation cutoffs, target and covariate columns, and an evaluation metric.Different task parameterizations can be created from the same dataset, but conflicting metrics create ambiguity about the intended objective.
- Evaluation protocol: Tasks use rolling-origin evaluation, producing H-step forecasts at successive cutoffs to mimic deployment and improve robustness.Each task yields W forecast–target pairs across its evaluation windows.
- Aggregation: Aggregation is necessary to answer whether one model is more accurate overall, and its choice directly affects benchmark reliability and interpretability.This motivates treating aggregation as part of the benchmark rather than merely reporting task-level scores.
3 Task definitions
fev-bench defines 100 diverse forecasting tasks spanning datasets, domains, horizons, covariate types, and forecast outputs, evaluated with scale-aware point and probabilistic metrics.
- Benchmark design: fev-bench contains 100 tasks designed to cover real-world domains, frequencies, horizons, univariate and multivariate series, covariates, and point and probabilistic forecasting.The benchmark is intended to provide broad rather than application-specific coverage.
- Datasets: The benchmark uses 96 unique datasets and includes known dynamic, past dynamic, and static covariates across its tasks.The dataset sources include established repositories, Kaggle, and domain-specific collections.
- Forecast horizons: fev-bench avoids duplicating horizons within datasets and instead prioritizes diversity across datasets and application domains with realistic forecasting horizons.Examples include 168 steps for hourly energy demand and 30 steps for daily retail sales.
- Rolling evaluation: Rolling-window evaluation balances computational cost and statistical reliability, using more windows for smaller datasets and fewer for larger ones.Results are averaged across windows, with a minimum-history requirement before the first evaluation window.
- Representative subset: fev-bench-mini provides 20 tasks that preserve diversity while enabling faster model development and ablation studies, with rankings that approximate the full benchmark.The subset is intended for lower-cost iteration.
- Evaluation metrics: Point accuracy is measured with MASE, which is scale-free and normalizes each series using historical seasonal error.This supports comparisons across series with different magnitudes and remains robust when forecast horizons contain zeros.
- Evaluation metrics: Probabilistic accuracy uses SQL over quantile levels 0.1 through 0.9, extending MASE’s scale-independent normalization to quantile forecasts.Unlike WQL, SQL normalizes each series by scale; WQL and WAPE are additionally reported in the supplement.
4 Aggregating the results
The benchmark aggregates task-level errors using marginal win rates and baseline-relative skill scores, then uses pairwise bootstrap intervals to assess whether model differences are robust to benchmark composition.
- 4 Aggregating the results: Win rates and skill scores provide complementary views of model quality, with win rates capturing relative ordering and skill scores measuring error reduction against a fixed baseline.Pairwise versions extend both statistics to direct model comparisons.
- 4.1 Marginal performance: Win rates range from 0 to 1 and treat ties as half-wins, but they ignore performance magnitude and change when new models enter the benchmark.These limitations motivate the baseline-relative skill score.
- 4.2 Skill score: Skill scores use clipped geometric-mean relative errors against a fixed baseline, yielding positive values for average improvement and negative values for underperformance.Clipping uses ℓ=10^-2 and u=100 to limit extreme-value influence.
- 4.2 Pairwise comparisons: Pairwise win rates and skill scores reveal model relationships that marginal rankings can obscure by comparing any two models directly.Pairwise win rate measures the fraction of tasks won, while pairwise skill score measures average error reduction.
- 4.3 Significance of performance differences: 95% paired-bootstrap confidence intervals quantify how pairwise conclusions vary across alternative benchmark compositions.The procedure resamples tasks with replacement using B=1000 bootstrap samples and constructs analogous intervals for win rates and skill scores.
- 4.3 Significance of performance differences: Marginal statistics establish an overall ranking, while pairwise intervals identify differences robust to benchmark changes; lower bounds above 50% indicate high-confidence win-rate superiority.The corresponding null hypotheses are W_jk=0.5 and S_jk=0.
5 Software package
The fev package provides lightweight, reproducible benchmarking infrastructure that standardizes task definition, evaluation, scoring, and aggregation without imposing rigid model or dependency choices.
- Motivation: The library addresses inconsistent dataset-only evaluations and the rigidity of monolithic benchmark systems by supporting comparable results and workflow integration.Existing systems can differ in horizons, cutoffs, metrics, aggregation, dependencies, and bundled components.
- 5 Software package: fev supplies task definition, data loading and splitting, prediction scoring, and result aggregation while avoiding unnecessary constraints.It depends on Hugging Face datasets and pydantic, without fixing versions of common packages such as torch or numpy.
- 5 Software package: fev excludes model implementations and instead provides adapters for popular forecasting libraries, reducing dependence on implementations that may become outdated.Supported libraries include GluonTS, darts, StatsForecast, AutoGluon, and sktime.
- Library API: An EvaluationWindow defines one cutoff-based train–test split, a Task specifies the forecasting problem, and a Benchmark collects tasks.Tasks can be defined in YAML and evaluation summaries include metric values plus the full task specification.
6 Related work
Time-series benchmarks have expanded in domain and setting coverage, but fev-bench combines broad task coverage with standardized, statistically informed evaluation infrastructure.
- 6 Related work: Existing benchmarks evolved from small, inconsistent dataset collections toward broader repositories and systems spanning domains, frequencies, and forecasting settings.The literature includes competitions, long-horizon benchmarks, repositories, domain-specific benchmarks, and pretrained-model evaluations.
- 6 Related work: fev-bench covers domains, frequencies, and forecasting settings while including 46 covariate tasks, complemented by lightweight infrastructure with standardized aggregation and confidence intervals.This combination distinguishes it from benchmarks emphasizing only datasets or narrower settings.
- 6.2 Aggregation strategies: Win rates are preferred over average rank because they are bounded between 0 and 1 and extend directly to pairwise comparisons.Average rank and average win rate induce the same ordering under the benchmark setup.
- 6.2 Aggregation strategies: Average rank, average win rate, and Bradley–Terry scores induce the same ranking when all models are compared on all tasks, so win rates are reported for simplicity.The equivalence holds under the paper’s aggregation setup.
- 6.2 Aggregation strategies: Confidence intervals on win rates convey effect magnitude and uncertainty, unlike conservative rank-based significance tests that provide only binary decisions.Skill scores use the complement of clipped geometric mean relative error to preserve a higher-is-better interpretation.
- 6.2 Aggregation strategies: Figure 1 compares pairwise win rates and skill scores for the top three models against other models under SQL, with bootstrapped 95% confidence intervals.Higher values are better, and full pairwise results appear in Section D.
7 Results
Pretrained models are evaluated across diverse forecasting settings using task-level metrics, aggregation methods, and runtime measurements. Chronos-2 leads overall, while covariates yield larger gains than native multivariate modeling.
- Evaluation protocol: Evaluation uses SQL for probabilistic forecasts and MASE for point forecasts, aggregating results across 100 tasks with win rates and skill scores.Runtime is reported as median end-to-end training and inference time normalized to 100 series per task.
- Overall results: Chronos-2 achieves the highest average win rates and skill scores, while pretrained models are substantially more accurate and faster than TFT and SCUM Ensemble.TiRex and TimesFM-2.5 follow Chronos-2, whereas older pretrained models rank lower.
- Pairwise comparisons: Confidence intervals show a statistically significant gap between Chronos-2 and the remaining models, but no clear winner between TiRex and TimesFM-2.5.Their heavy overlap means benchmark composition or task weighting could change the second-place ordering.
- Forecasting with covariates: 47.0% vs. 40.9% for Chronos-2 and 42.5% vs. 34.1% for TabPFN-TS show higher SQL skill scores with dynamic covariates.The comparison uses 42 fev-bench tasks containing dynamic covariates.
- Multivariate forecasting: 55.0% vs. 54.4% for Chronos-2 and 54.7% vs. 54.1% for Toto-1.0 show modest gains from native multivariate modeling.The comparison uses 35 multivariate tasks and contrasts multivariate with univariate modes.
8 Scope and limitations
fev-bench prioritizes broad coverage, fast evaluation, and model-development utility rather than fully leakage-free evaluation or specialization for one application domain.
- Scope and leakage: fev-bench offers a general view across domains, frequencies, and forecasting settings, so specialized tasks may produce different model rankings.The benchmark relies on self-reported training overlap as a practical contamination safeguard, while leakage-free alternatives have narrower coverage and slower evaluation.
9 Conclusion
The paper concludes that fev-bench and fev provide a broad, reproducible foundation for statistically robust forecasting evaluation. The benchmark spans diverse datasets and settings, while the aggregation analysis establishes equivalent model orderings for several ranking methods.
- Conclusion: fev-bench combines covariates, multivariate tasks, and principled aggregation for statistically robust evaluation across diverse forecasting domains.The benchmark contains 100 forecasting tasks and is complemented by the lightweight fev evaluation package.
- Task design: The dataset suite includes univariate and multivariate problems with dynamic and static covariates, supporting both point and probabilistic forecasting evaluation.The task inventory distinguishes past dynamic, known dynamic, and static covariates.
- Dataset coverage: The benchmark draws on datasets spanning GIFT-Eval, macroeconomic, energy, BOOMLET, competition, and miscellaneous application sources.Its task design includes varied frequencies, horizons, target selections, and covariate types.
- Model evaluation: The evaluation covers pretrained, statistical, global, and task-specific models with documented configurations, runtimes, and publicly available wrappers.Pretrained models are evaluated alongside statistical baselines, LightGBM, CatBoost, DeepAR, PatchTST, and TFT.
- Aggregation: Average win rates, average ranks, and Bradley–Terry scores induce the same model ordering under the benchmark setup.The appendix derives the equivalence through the affine relationship between win rates and average ranks.
C.2 Average win rate and Bradley–Terry (Elo) scores result in the same ranking
The Bradley–Terry model converts pairwise win rates into latent Elo-style skill scores, and its scores induce the same model ordering as average win rates. The section also reports benchmark evaluation materials and robustness intervals for pairwise comparisons.
- C.2 Average win rate and Bradley–Terry (Elo) scores result in the same ranking: The BT model assumes each model has a latent skill θj and models the probability of outperforming another model through a logistic link.The scaling constant λ is positive, and the parameters are estimated by maximum likelihood subject to an identifiability constraint such as fixing one baseline score.
- C.2 Average win rate and Bradley–Terry (Elo) scores result in the same ranking: Strict concavity of the BT log-likelihood ensures a unique solution up to translation, while the score-difference expression preserves the sign of the skill difference.Thus the ordering equivalence is not merely empirical; it follows from the model’s likelihood structure.
- C.2 Average win rate and Bradley–Terry (Elo) scores result in the same ranking: Average win rates Wj and BT/Elo scores θj induce the same ordering of models, with higher win rate corresponding to higher Elo score.At the BT maximum-likelihood estimate, θj > θk exactly when Wj > Wk, and equality likewise coincides.
- C.2 Average win rate and Bradley–Terry (Elo) scores result in the same ranking: The benchmark materials include marginal SQL and MASE results for the full benchmark and an interactive or CSV release of tabular results.The supplied passages also note failures for several models caused by timeouts, out-of-memory errors, or NaNs during training.
- C.2 Average win rate and Bradley–Terry (Elo) scores result in the same ranking: Pairwise win-rate and skill-score figures report SQL and MASE comparisons with bootstrapped 95% confidence intervals, where higher values are better.The skill scores are explicitly non-symmetric, so Sjk need not equal Skj.
E.2 Evaluation results
fev-bench-mini contains 20 representative tasks, and its model rankings, win rates, and skill scores align with those on the full 100-task benchmark. Marginal SQL and MASE results are reported for the mini benchmark.
- E.2 Evaluation results: Marginal probabilistic forecasting performance on fev-bench-mini is reported according to the SQL metric.The results are presented in Table 16.
- E.2 Evaluation results: Marginal point forecasting performance on fev-bench-mini is reported according to the MASE metric.The results are presented in Table 17.