PaperlayerView exact revision on arXiv

Source-linked AI summary

QuitoBench: A High-Quality Open Time Series Forecasting Benchmark

Siqiao Xue, Zhaoyang Zhu, Wei Zhang, Rongyao Cai, Rui Wang, Yixiang Mu, Fan Zhou, Jianguo Li, Peng Di, Hang Yu

arXiv:2603.26017v1cs.LG

TL;DR

Time-series forecasting lacks large, reliable benchmarks that reflect forecasting-relevant structure rather than application labels. The paper introduces QUITO and regime-balanced QUITOBENCH, then evaluates ten models across balanced TSF regimes and operating conditions. Results show a context-length crossover between model families, while compact deep learning models can match or exceed foundation models and training-data scaling is more beneficial than parameter scaling.

  • Problem

    Time-series forecasting lacks unified, reliable benchmarks, while existing evaluations use coarse domain groupings and imbalanced datasets.

  • Method

    The paper builds QUITO from Alipay application traffic and constructs QUITOBENCH by labeling series with trend, seasonality, and forecastability and balancing all eight TSF regimes.

  • Results

    Deep learning models lead at L = 96, foundation models lead at L ≥576, and CrossFormer achieves the best overall mean rank of 2.86 and lowest MAE of 0.279.

  • Takeaways & Limitations

    Model selection should account for context length and TSF regime, because compact task-specific architectures can remain competitive with substantially larger foundation models.

  • Takeaways & Limitations

    The data come from one organisation, the benchmark focuses on point forecasting of application traffic, and the TSF threshold may not suit every use case.

Abstract

from arXiv · show

Time series forecasting is critical across finance, healthcare, and cloud computing, yet progress is constrained by a fundamental bottleneck: the scarcity of large-scale, high-quality benchmarks. To address this gap, we introduce \textsc{QuitoBench}, a regime-balanced benchmark for time series forecasting with coverage across eight trend$\times$seasonality$\times$forecastability (TSF) regimes, designed to capture forecasting-relevant properties rather than application-defined domain labels. The benchmark is built upon \textsc{Quito}, a billion-scale time series corpus of application traffic from Alipay spanning nine business domains. Benchmarking 10 models from deep learning, foundation models, and statistical baselines across 232,200 evaluation instances, we report four key findings: (i) a context-length crossover where deep learning models lead at short context ($L=96$) but foundation models dominate at long context ($L \ge 576$); (ii) forecastability is the dominant difficulty driver, producing a $3.64 \times$ MAE gap across regimes; (iii) deep learning models match or surpass foundation models at $59 \times$ fewer parameters; and (iv) scaling the amount of training data provides substantially greater benefit than scaling model size for both model families. These findings are validated by strong cross-benchmark and cross-metric consistency. Our open-source release enables reproducible, regime-aware evaluation for time series forecasting research.

1 Introduction

The paper addresses unreliable time-series evaluation caused by scarce, inconsistent, and structurally imbalanced benchmarks. QUITOBENCH uses intrinsic TSF regimes, balanced coverage, leakage-free evaluation, and long series to support more reliable model comparison and selection.

  • Motivation: Time series has the lowest benchmark-dedicated publication share among four surveyed domains, at 4.2%.The comparison is 4.2% for time series versus 9.9% for NLP, 7.0% for speech, and 6.8% for vision.
  • Challenges: Existing benchmarks group series by application domain, although intrinsic trend, seasonality, and forecastability better reflect forecasting difficulty.The paper notes that series within one domain can differ more in predictability than series from different domains.
  • Challenges: Current benchmarks lack the scale, balance, and rigour needed to guide choices across context length, forecast horizon, forecasting mode, and intrinsic data characteristics.This gap is especially important when comparing foundation models with much smaller deep learning models.
  • Contribution: QUITOBENCH balances evaluation series across all eight trend×seasonality×forecastability regimes using a billion-scale, single-provenance Alipay traffic corpus.QUITO covers nine business verticals and contains 1.6 B tokens, while the benchmark is designed around forecasting-relevant statistical properties rather than domain labels.
  • Contribution: The benchmark uses leakage-free evaluation and uniformly long series, enabling context lengths up to 1,024 without overlap with public pre-training corpora.A single proprietary operational environment supports the stated contamination-free construction, while series span 5,900–15,300 time steps.
  • Findings: At L = 96 deep learning models lead, whereas foundation models overtake them at L ≥576; CrossFormer matches the best overall accuracy with far fewer parameters.The evaluation spans ten models, 18 configurations, and approximately 1.6×10^7 predictions per model.

2 QUITO Description

QUITO converts Alipay application-traffic telemetry into standardized, leakage-free datasets and a balanced benchmark. Its TSF taxonomy measures trend, seasonality, and forecastability, then stratifies evaluation series across eight regimes.

  • Data sources and series formation: QUITO contains minute-level and hour-level application-traffic series from nine Alipay business verticals, formed from disjoint pools of workload traces.Each trace has five traffic-subtype variates, and the two granularities avoid overlap in item identifiers.
  • Sanitization and standardization: The pipeline retains 14,244 minute-level and 16,746 hour-level series after sanitization and standardization.These are released as QUITO-MIN and QUITO-HOUR.
  • TSF diagnostics and regime labeling: Each series receives a TSF profile measuring trend strength, seasonality strength, and forecastability from normalized spectral entropy.Trend and seasonality use STL decomposition, while forecastability is F = 1 − H.
  • TSF diagnostics and regime labeling: A default threshold τ = 0.4 binarizes each TSF diagnostic into HIGH or LOW, producing 2^3 = 8 regime cells.For multivariate series, diagnostics are averaged across variate channels before binarization.
  • Leakage-free temporal splitting: A global temporal cutoff creates leakage-free train, validation, and test splits shared across all series.Data before 2023-07-28 is divided into 80% training and 20% validation, while later data forms the test set.
  • Balanced benchmark construction: Stratified sampling draws approximately 162 series per TSF cell, yielding 1,290 test series with near-uniform regime coverage.The benchmark contains 773 minute-level and 517 hour-level series, roughly 160 per cell.

3 Experiments

QUITOBENCH evaluates ten forecasting models across diverse configurations and finds that model performance depends strongly on context length, forecastability, regime, parameter efficiency, and scaling strategy.

  • Experimental setup: 232,200 evaluation instances compare ten models across three families, 18 configurations, dense rolling windows, and both multivariate and univariate modes.Each configuration uses three context lengths, three horizons, and two forecasting modes; dense rolling windows generate approximately 1.6×10^7 predictions per model.
  • Overall performance: CrossFormer achieves the best overall mean rank of 2.86 and lowest MAE of 0.279, while deep learning and foundation models have statistically similar category-level MAE.Deep learning averages MAE 0.312 versus 0.319 for foundation models, with Cohen’s d = −0.067 for CrossFormer versus Chronos-2.
  • Context length: At L=96 deep learning leads by 24.6% MAE, but foundation models lead by 22.0% at L=1024 as their long-context gains exceed those of deep learning models.Foundation models improve 43–50% from short to long context, compared with 7–12% for deep learning models.
  • TSF regime analysis: Forecastability is the dominant difficulty driver: mean MAE rises from 0.278 for high-forecastability series to 0.505 for low-forecastability series, while the easiest regime is 3.64× easier than the hardest.The HIGH LOW LOW regime is 56.7% harder than the next-hardest regime; CrossFormer reaches MAE 0.600 there, 2.91× worse than on high-forecastability series.
  • TSF regime analysis: Foundation models win six of eight TSF regimes, whereas CrossFormer leads the two low-seasonality regimes by 38.4% and 17.7%.The regime split associates foundation-model strength with high seasonality or forecastability and deep-learning strength with low-seasonality settings.
  • Efficiency and scaling: CrossFormer matches or exceeds foundation-model accuracy with 58× fewer average parameters, and increasing training data yields larger gains than increasing model size.CrossFormer’s 1 M parameters and MAE 0.279 outperform Chronos-2’s 100 M parameters and MAE 0.314; data-scaling curves show larger improvements than parameter scaling.
  • Robustness of findings: Model rankings remain consistent across metrics and benchmarks, with Spearman ρ = 0.733 for aggregate MAE–MSE rankings and ρ = 0.865 between QUITOBENCH and Timer.CrossFormer ranks first under both metrics and both benchmarks, while QUITOBENCH additionally provides regime coverage and long-context evaluation.

4 Conclusion

The paper presents QUITO and QUITOBENCH as a billion-scale application-traffic corpus and regime-balanced benchmark for rigorous, contamination-free time series forecasting evaluation, with open data, code, and pipelines.

  • QUITO and QUITOBENCH provide a billion-scale application-traffic corpus and regime-balanced benchmark for time series forecasting.The release is designed to support rigorous, contamination-free evaluation.
  • All data, code, and pipelines are openly released.

Appendices

The appendices situate the work against statistical, deep learning, and foundation-model forecasting methods and review existing benchmarks, emphasizing their diversity and leakage risks.

  • Time series forecasting models: Time series forecasting methods span statistical models, deep learning models, and newer foundation models.Statistical approaches model temporal structure under explicit assumptions, while deep learning methods capture nonlinear dynamics with fewer such assumptions.
  • Time series forecasting benchmarks: Existing benchmarks aggregate diverse public series, but heterogeneous open-source data can introduce direct and indirect leakage risks.

B Broader Impact, Limitations, and Future Directions

The release broadens access to realistic forecasting data and uses TSF regimes to identify high-risk scenarios, while remaining bounded by single-organization data, point-forecasting scope, threshold choices, and inference costs.

  • Broader Impact: The open release enables researchers without proprietary data access to develop and evaluate models on realistic, diverse workloads.The data and benchmark are released under CC-BY 4.0.
  • Broader Impact: 3× higher error rates occur in the HIGH LOW LOW regime, which represents 12% of the benchmark and warrants specialized modeling approaches.
  • Limitations: The corpus comes from a single organization, so workload patterns specific to other cloud platforms may be under-represented.The benchmark focuses on point forecasting of application traffic, and its TSF threshold may not suit every use case.
  • Future Directions: Future work includes probabilistic forecasting, anomaly detection, new foundation models, forecastability-aware selection tools, and cross-provenance validation.The authors also identify evaluating large-language-model integration with forecasting pipelines as a promising direction.

C Analysis on GIFT-Eval

This appendix documents the data format, provenance, aggregation, multivariate structure, deduplication, and TSF diagnostics used to characterize series and compare workload regimes.

  • Data Release: QUITO and QUITOBENCH are distributed as Apache Parquet files in complementary corpus and evaluation releases.The benchmark contains complete series for 1,290 sampled items across minute and hourly resolutions.
  • Data Format: The datasets use a long row format in which each row represents one timestamp of one series.Series reconstruction requires filtering by item id and sorting by date time; no silent imputation is applied.
  • Data Construction: Minute and hourly collections come from disjoint application pools and share a common end date, supporting leakage-free splits across granularities.
  • Time Aggregation: Max pooling aggregates 1-second telemetry into fixed, non-overlapping 10-minute or 1-hour bins while preserving bursty workload peaks.The target resolutions are r ∈ {60, 3600} seconds.
  • Workload Diversity: Each workload trace is a five-channel multivariate series associated with a distinct application service, with anonymized traffic subtypes.The corpus spans nine business verticals and includes diverse operational workloads within one production environment.
  • TSF Diagnostics: Each series receives trend, seasonality, and forecastability diagnostics, with larger values indicating stronger presence of the corresponding property.Trend and seasonality use STL decomposition, and the diagnostics are averaged across five variate channels.
  • TSF Regimes: The eight TSF regimes expose distinct dynamic patterns and show why low forecastability is associated with greater model difficulty.

D.7 Cross-Provenance TSF Coverage Analysis

Quito’s single-provenance series cover a broader and more balanced TSF space than Timer’s multi-provenance collection, while model rankings transfer strongly across benchmarks. This supports TSF-based categorization as a forecasting-relevant alternative to application-domain labels.

  • TSF diversity: Quito achieves 1.24–1.57× higher standard deviation than Timer on every TSF axis and 1.67× higher regime entropy.Quito’s regime entropy is 1.99 bits versus Timer’s 1.19 bits.
  • TSF diversity: Timer concentrates 75.6% of its series in one regime, whereas Quito spans wider trend and forecastability values across nine business verticals.Timer clusters around high seasonality and low trend, while Quito reflects broader workload heterogeneity.
  • Cross-benchmark validation: Model rankings transfer from QuitoBench to Timer with Spearman ρ=0.865, indicating strong cross-benchmark consistency despite different provenance.The comparison provides functional validation across single-provenance and multi-provenance data.
  • External consistency: Quito’s TSF diversity is qualitatively consistent with published cloud traces, including strong diurnal, weak, and near-random workload patterns.These patterns span user-facing services, batch pipelines, and bursty events.
  • Categorization: Domain labels do not consistently predict forecasting difficulty, because series from the same domain can span the full difficulty range.The paper therefore categorizes series by intrinsic trend, seasonality, and forecastability.

E.2 Training and Testing Details

QUITOBENCH combines staged training, zero-shot foundation-model evaluation, and dense rolling-window testing across long series. Its protocol uses validation-based checkpoint selection, repeated seeds, and per-series rank aggregation to improve evaluation stability and fairness.

  • Deep learning training: Deep learning models use three stages: hyperparameter tuning, fine-tuning with early stopping, and test evaluation across three random seeds.The reported MAE is averaged across seeds after selecting the best validation-MSE checkpoint.
  • Foundation-model evaluation: Foundation models are evaluated zero-shot from public checkpoints without task-specific fine-tuning or gradient updates.The context length and forecast horizon are passed directly to each model for inference.
  • Scaling experiments: TimesFM-2.5 is selected for scaling experiments because it supports standard end-to-end fine-tuning with MSE loss.Chronos uses tokenization and cross-entropy, while TiRex lacks fine-tuning support in its released codebase.
  • Rolling-window evaluation: Dense unit-stride windows produce W(H) = Ttest − H + 1 evaluation windows per series, independent of context length L.The 70%/20%/10% split and long test segments support hundreds to over a thousand windows per series.
  • Rolling-window evaluation: QUITOBENCH generates approximately 1.6 × 10^7 predictions per model, one to two orders of magnitude more than GIFT-EVAL’s non-overlapping scheme.The larger sample count is intended to stabilize series-level performance estimates.
  • Ranking methodology: Mean model rank is computed by ranking models per series and configuration before averaging, so each series contributes equally.This prevents heterogeneous error scales and outliers from dominating aggregate results.

E.4 TSF Regime Analysis: Forecastability, Specialization, and Pathological Regimes

Forecastability is the dominant difficulty driver across TSF regimes, while model strengths depend on forecastability and context length. Threshold sweeps preserve the main regime conclusions, although extreme thresholds make regime analysis uninformative.

  • Forecastability and difficulty: 3.64× separates the hardest HIGH LOW LOW regime from the easiest HIGH HIGH HIGH regime by mean MAE.The corresponding mean MAEs are 0.749 and 0.205.
  • Forecastability and difficulty: High-forecastability series with low trend and seasonality reach MAE 0.220, only 7% above the easiest regime.Strong structural patterns alone do not guarantee easy forecasts when forecastability is low.
  • Model sensitivity: Deep learning models degrade approximately 2.2–2.3× from high- to low-forecastability series, versus 1.7–1.8× for foundation models.The result suggests greater robustness of foundation-model representations on unpredictable series.
  • Pathological regime: CrossFormer leads the pathological HIGH LOW LOW regime with MAE 0.600, while foundation models cluster from 0.628 to 0.652.This regime is 56.7% harder than the second-most-difficult regime and 2.91× worse for CrossFormer than the easiest regime.
  • Threshold sensitivity: For τTSF ∈[0.40, 0.50], the low-F/high-F MAE ratio remains 1.86×–2.01× and regime rankings remain stable.At thresholds below 0.40, nearly all items become high-forecastability, making the split degenerate.
  • Context-length crossover: At L = 96, deep learning leads by 24.6% MAE, whereas at L ≥576 foundation models lead by 15–22%.CrossFormer wins all six short-context configurations; Chronos-2 leads at longer contexts.
  • Statistical tests: The aggregate deep-learning versus foundation-model MAE difference is not statistically significant: 0.312 versus 0.319, p = 0.114.Family-level averages therefore mask the context- and regime-dependent differences reported elsewhere.

F Cross-Benchmark Consistency: QuitoBench vs. Timer

QUITOBENCH model rankings show strong consistency with Timer despite differences in training paradigm and data distribution. Consistency is strongest among deep learning models, while some individual models shift substantially between benchmarks.

  • Overall consistency: Spearman ρ=0.865 links model rankings on QUITOBENCH and Timer, with p < 0.01 and 95% CI [0.52, 0.97].The comparison uses trained univariate QUITOBENCH results and zero-shot univariate Timer results.
  • Consistency tiers: Chronos-2, TimesFM-2.5, iTransformer, and DLinear remain within one rank across benchmarks.These models form the highly consistent Tier 1 group.
  • Consistency tiers: TiRex, CrossFormer, and PatchTST show rank changes of approximately three to four positions between benchmarks.TiRex moves from fifth on QUITOBENCH to first on Timer.
  • Deep learning robustness: Among deep learning models, ranking consistency rises to 0.891 and 7/8 regimes share the same best model.CrossFormer is the top deep learning model in 8/8 QUITOBENCH regimes and 7/8 Timer regimes.
  • Dataset design: Timer’s 88.1% high-forecastability composition contrasts with QUITOBENCH’s uniform TSF-regime coverage.The paper associates this imbalance with advantages for models specialized to predictable, structured series.

G Metric Robustness: MSE vs. MAE Rankings

MSE and MAE produce highly consistent model rankings across aggregate, configuration, instance, and TSF-regime analyses. This robustness coexists with regime-dependent performance patterns and supports balanced, metric-aware benchmark interpretation.

  • Metric robustness: ρ=0.733 (p=0.016) between aggregate MSE and MAE model orderings, with CrossFormer ranked first under both metrics.The bottom tier remains unchanged, while mid-tier models shift by up to 3 ordinal positions.
  • Metric robustness: ρ=0.847 mean correlation across 18 evaluation configurations, ranging from ρ=0.685 to 0.952.Short-context configurations show the strongest agreement, with ρ≥0.879.
  • Metric robustness: 0.723 Kendall τ between instance-level MSE and MAE ranks, with the best models coinciding in 54.4% of instances.Exact MAE and MSE ranks match in 48.0% of instances.
  • Metric robustness: ρ=0.976 in HIGH LOW HIGH and ρ=0.952 in LOW LOW HIGH, versus ρ=0.442 in HIGH HIGH LOW and ρ=0.600 in LOW HIGH LOW.Weaker agreement occurs in low-forecastability regimes, where MSE is more sensitive to outliers.
  • Regime analysis: 3.64× MAE gap across TSF regimes, from 0.205 in HIGH HIGH HIGH to 0.748 in HIGH LOW LOW.QUITOBENCH assigns approximately 12.5% of evaluation instances to each regime, preventing prevalence-driven aggregate metrics.
  • Regime analysis: 8/8 deep-learning regime wins for CrossFormer, compared with 4/8 overall; Chronos-2 wins 3/8 and TiRex 1/8 overall.Foundation models excel in high-seasonality regimes, while deep learning models dominate trend-driven regimes.
  • Cross-benchmark analysis: 34.5% lower MAE for CrossFormer on Quito than Timer in HIGH HIGH HIGH, with values of 0.165 versus 0.252.The comparison attributes the difference to task-specific training on diverse, balanced data rather than zero-shot generalization.

H.6.1 Key Findings from MAE Rank Analysis

MAE Rank analysis identifies CrossFormer as the most consistent deep learning model across TSF regimes, while foundation models and simple baselines occupy distinct performance niches. The deep-learning ordering remains stable across regimes, and MSE-based analysis reinforces CrossFormer’s consistency.

  • MAE Rank findings: 8/8 groups: CrossFormer is the best deep learning model by MAE Rank across every TSF regime.Its strongest mean rank is 2.11 in HIGH LOW HIGH.
  • MAE Rank findings: 5/8 regimes: CrossFormer achieves the best overall mean MAE Rank, while Chronos-2 wins 2/8 and TiRex wins 1/8.Chronos-2 achieves its best mean rank, 2.59, in LOW HIGH HIGH.
  • Baseline performance: Positions 9–10: ES and SNaive occupy the bottom ranks across all groups.The result indicates weak performance for these simple baselines on application-traffic forecasting.
  • Deep learning ranking: 2nd-best deep-learning rank: PatchTST achieves this position in 6 of 8 groups, with iTransformer taking it in the remaining 2.The stable ordering is CrossFormer, PatchTST/iTransformer, TSMixer, then DLinear.
  • MSE analysis: 3 regimes each: PatchTST and CrossFormer lead by mean MSE value, while Chronos-2 and TimesFM-2.5 win 1 regime each.The HIGH LOW LOW regime has MSE values from 565 to 1208, reflecting the squared-error penalty on unpredictable, strongly trending series.
  • MSE Rank comparison: 8/8 regimes: CrossFormer achieves the best mean MSE Rank, despite winning only 3/8 regimes by mean MSE value.This distinction indicates consistency across instances rather than universal lowest absolute MSE.

I arXiv Benchmark Analysis

The arXiv analysis estimates benchmark activity across four research domains using domain-specific searches and title-based benchmark identification. Its comparisons are reproducible but bounded by keyword, coverage, category, and preprint limitations.

  • Data and domains: 2020–2025: arXiv submissions are counted as a proxy for research activity across four data-modality domains.The analysis uses the arXiv API and covers papers submitted from January 1, 2020 through December 31, 2025.
  • Domain identification: Time-series papers are identified through title or abstract keywords because time series lacks a dedicated arXiv category.Vision, NLP, and speech/audio are identified through dedicated category filters.
  • Benchmark identification: Benchmark papers require title keywords such as benchmark, dataset, introducing, release, or corpus.The title-based criterion is intended to provide high precision for papers introducing new benchmarks or datasets.
  • Share computation: N bench d,y counts papers matching both the domain query and benchmark keywords.The benchmark share is computed from this count relative to N total d,y, with aggregate shares using cumulative counts.
  • Query implementation: 2024: the retrieval query combines time-series terms, benchmark or dataset-introduction title terms, and a submitted-date interval.The total count is extracted from the API response’s opensearch:totalResults field.
  • Limitations: Keyword sensitivity, incomplete arXiv coverage, differing category practices, and preprint status constrain interpretation of the domain comparison.The keyword approach may miss unconventional titles or include false positives, while arXiv submissions may not represent final peer-reviewed publications.
Loading 2603.26017v1…