Source-linked AI summary
It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks
Zhongzheng Qiao, Sheng Pan, Anni Wang, Viktoriya Zhukova, Yong Liu, Xudong Jiang, Qingsong Wen, Mingsheng Long, Ming Jin, Chenghao Liu
TL;DR
Existing TSFM benchmarks are limited by reused data, inadequate quality control, context-misaligned tasks, and coarse dataset-level analysis. TIME addresses these gaps with fresh, context-aligned benchmark construction and pattern-level evaluation, finding differentiated TSFM performance across temporal patterns. The benchmark evaluates 12 models and provides multi-granular analysis through an interactive leaderboard.
Problem
Existing TSF benchmarks suffer from legacy data reuse, insufficient quality assurance, misaligned forecasting configurations, and limited dataset-level analysis.
Method
TIME builds a human-in-the-loop benchmark with fresh data, automated screening, context-aligned tasks, structural pattern encoding, and multi-granular evaluation.
Results
Chronos-2, TimesFM-2.5, and TiRex are the top three performers overall, while model rankings vary with stationarity and other temporal patterns.
Takeaways & Limitations
Pattern-level analysis provides generalizable, diagnostic views of TSFM capabilities beyond aggregate dataset-level rankings.
Takeaways & Limitations
The benchmark’s general-purpose MASE and CRPS metrics may not reflect application-specific operational requirements, motivating future task-specific metrics.
Abstract
from arXiv · showhide
Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation. However, we contend that existing benchmarks exhibit common limitations in four dimensions: constrained data composition dominated by reused legacy sources, compromised data integrity lacking rigorous quality assurance, misaligned task formulations detached from real-world contexts, and rigid analysis perspectives that obscure generalizable insights. To bridge these gaps, we introduce TIME, a next-generation task-centric benchmark comprising 50 fresh datasets and 98 forecasting tasks, tailored for strict zero-shot TSFM evaluation free from data leakage. Integrating large language models and human expertise, we establish a human-in-the-loop benchmark construction pipeline to ensure high data integrity and redefine task formulation by aligning forecasting configurations with real-world operational requirements and variate predictability. Furthermore, we propose a novel pattern-level evaluation perspective that moves beyond traditional dataset-level evaluations based on static meta labels. By leveraging structural time series features to characterize intrinsic temporal properties, this approach offers generalizable insights into model capabilities across diverse patterns. We evaluate 12 TSFMs and establish a multi-granular leaderboard to facilitate in-depth analysis and visualized inspection. The leaderboard is available at https://huggingface.co/spaces/Real-TSF/TIME-leaderboard.
1. Introduction
TIME addresses four shortcomings in TSF benchmarks—legacy-constrained data, weak quality assurance, context-misaligned tasks, and limited analysis perspectives—by introducing fresh-data, task-centric, pattern-level evaluation.
- Existing TSF evaluation is shifting from dataset-specific modeling toward zero-shot, task-centric assessment for TSFMs.
- Legacy dataset reuse limits coverage, slows apparent progress, and increases benchmark contamination risk.
- Quality assurance is often insufficient, allowing extreme outliers and excessive missing values into benchmark data.
- Fixed prediction horizons detach benchmark tasks from application contexts and data frequencies.
- Dataset-level meta labels and scalar error metrics provide limited insight into temporal behavior, model suitability, and deployment reliability.
- TIME contributes 50 fresh datasets, 98 operationally aligned forecasting tasks, pattern-level analysis, and evaluation of 12 TSFMs with an interactive leaderboard.
2. Related Work
TSF benchmarks evolved from heterogeneous small-scale evaluations and established repositories toward large-scale zero-shot assessment, but recent benchmarks still rely heavily on previously released data and face interpretability challenges.
- Early TSF studies evaluated models on varying sets of only 4–6 datasets without unified protocols.
- M4 introduced large-scale univariate evaluation, while LSF and Monash established widely reused standardized and diverse data sources.
- Since 2023, TSFMs have driven large-scale zero-shot benchmarks spanning diverse domains and broader data coverage.
- Recent benchmark efforts still mainly assemble previously released datasets instead of incorporating genuinely fresh data.
- Time-series heterogeneity and weak intuitive semantics motivate structural feature-based characterization for analysis.
3. Preliminary
The paper formalizes benchmarks as hierarchies of tasks, datasets, series, variates, and testing windows, and represents forecasting tasks through datasets and prediction horizons.
- A benchmark contains multiple tasks sharing an evaluation protocol and offering multiple analysis perspectives.
- Datasets contain univariate or multivariate series, whose variates are treated as prediction targets rather than exogenous covariates.
- Testing windows are continuous target segments, and forecasting samples are generated with nonoverlapping rolling windows.
- A task pairs a dataset with a dedicated prediction horizon.
- A time-series pattern is represented by a feature vector encoding curated statistical properties of a single variate's intrinsic temporal dynamics.
4. Benchmark Construction
TIME constructs its benchmark from fresh, quality-screened data and context-aligned tasks, combining automated diagnostics, human review, structural pattern analysis, and rolling-window evaluation.
- Data curation: TIME prioritizes fresh datasets from public, governmental, industrial, and academic sources to reduce contamination risk from legacy data.
- Data curation: Candidate datasets undergo eligibility and metadata checks for valid timestamps, regular sampling, forecastable context, frequency, and time span.
- Automatic screening: Automated screening profiles timestamp consistency, missingness, length, near-constant behavior, predictability, spikes, and cross-variate redundancy.
- Human review: Quality diagnostics are aggregated into a dataset-level summary for final human decision-making.
- Human review: Human and LLM-assisted review distinguishes corruption from expected domain characteristics while preserving application alignment.
- Task formulation: Task horizons and test lengths are selected from dataset frequency, operational constraints, and complete seasonal cycles rather than universal rules.
5. Benchmarking Strategy
TIME defines temporal patterns using interpretable structural features extracted from STL components, then retrieves variates with matching binary pattern codes for targeted evaluation. This enables pattern-specific, cross-dataset model comparisons using aggregated variate-level results.
- Structural feature construction: STL decomposes each variate into Trend, Seasonality, and Remainder components before feature extraction.This isolates temporal components rather than computing metrics directly on raw variates.
- Structural feature construction: 16.6% of benchmark variates are non-stationary, while Figure 4 shows empirical distributions for continuous features F1–F6.Stationarity is omitted from the density plots because it is binary.
- Structural feature construction: Seven features characterize trend strength and linearity, seasonal strength and correlation, residual autocorrelation, complexity, and stationarity.F1–F5 describe component structure; F6 measures spectral entropy, and F7 is a binary stationarity indicator.
- Pattern stratification: The feature distributions support grouping and cross-domain analysis of temporal patterns across datasets and sampling frequencies.The curated features are described as diverse, informative, and suitable for effective stratification.
- Pattern stratification: Each variate’s feature vector is converted into a seven-bit binary pattern code using population medians for continuous features.Variates with identical codes can then be retrieved as pattern-specific evaluation subsets.
- Pattern stratification: MASE and CRPS aggregate variate-level results within retrieved patterns to produce pattern-specific performance estimates and leaderboards.This provides model comparisons across specific temporal dynamics rather than only dataset-level groupings.
6. Experiments
TIME evaluates 12 TSFMs across 50 datasets and 98 tasks using normalized rolling metrics, then analyzes performance by temporal patterns and visual behavior. Overall rankings favor recent models, while pattern-level and qualitative analyses reveal capability differences that task-level aggregation can obscure.
- Evaluation Setup: 12 TSFMs are evaluated across 50 datasets and 98 forecasting tasks using rolling MASE and CRPS evaluation.Distribution-based models are converted to quantiles by sampling before metric computation.
- Evaluation Setup: Task-level scores are normalized against Seasonal Naive and aggregated with a geometric mean across all 98 tasks.A normalized score below 1 indicates improvement over the baseline.
- Overall Performance: Chronos-2, TimesFM-2.5, and TiRex achieve the lowest overall MASE and CRPS, while newer model versions consistently outperform predecessors.The authors interpret this progression as evidence of capability improvements rather than overfitting to established benchmark biases.
- Pattern-specific Performance: Pattern-level analysis retrieves variates by structural features and aggregates normalized metrics to compare model behavior across specific temporal patterns.Figure 6 contrasts variates with and without each feature using the distance between their performance markers.
- Pattern-specific Performance: Trend strength and linearity generally produce larger relative gains for TSFMs, while recent models improve across trend conditions.The reported gains suggest that models exploit strong and linear trends without making overall progress dependent on trend intensity.
- Pattern-specific Performance: Pattern-specific rankings differ from overall rankings, and recent models show distinct sensitivities to seasonality, stationarity, and complexity.Chronos-2 leads stationary sequences, whereas TimesFM-2.5 surpasses it on non-stationary data; high-complexity sequences remain difficult for all models.
- Qualitative Analysis: Visual inspection shows that competitive MASE or CRPS can coexist with conservative flat forecasts that fail to capture high-variation dynamics.The platform therefore pairs quantitative scores with prediction visualizations to distinguish captured structure from safe mean-like behavior.
7. Conclusion
TIME addresses benchmark contamination and limited evaluation granularity through fresh data and pattern-level analysis, while visual inspection complements aggregate metrics.
- TIME constructs a contamination-free repository of 50 fresh datasets with context-aligned task formulations.
- Performance gains on existing benchmarks have slowed, complicating separation of genuine modeling progress from noise, implementation details, and data biases.
- Public legacy datasets increase contamination risk because TSFMs may have absorbed them during large-scale pretraining.
- Time-series contamination is difficult to detect because weak semantics, fragmented versioning, and inconsistent naming obscure shared data origins.
A.2. Limited Evaluation Perspective
Dataset-level categories and scalar metrics provide limited insight into temporal behavior and deployment readiness. TIME therefore combines pattern-aware aggregation with interactive qualitative inspection.
- Existing evaluations organize results by domain or frequency, although predictive temporal patterns often cut across these categories.
- Scalar time-series error improvements do not establish that forecasts capture temporal behavior or are reliable for deployment.
- TIME groups test instances by interpretable temporal features and complements aggregate metrics with qualitative forecast inspection.
- Table 2 reports task frequencies, forecast horizons, rolling test windows, and physical durations, with blank medium- or long-term entries indicating single-task configurations.
B. Datasets
The benchmark datasets cover environmental, energy, transportation, health, economic, and other domains across varied resolutions and observation periods.
- The collection includes water-quality, ocean-current, chlorophyll, and coastal temperature-salinity datasets from Australian monitoring sources.
- Singapore weather, PM2.5, carpark, and Northeast China wind datasets provide meteorological, air-quality, urban, and regional weather series.
- Australia Solar, EPF Electricity Price, OpenElectricity NEM, and EWELD Load represent solar generation, electricity markets, energy demand, and extreme-weather consumption.
- Finland Traffic and Port Activity capture transportation and maritime activity at fine-grained or aggregated temporal resolutions.
- ECDC COVID and Global Influenza provide daily hospital-occupancy and weekly influenza-surveillance series across European regions and selected countries.
C.1. Data Screening Pipeline
The screening pipeline combines automated quality checks with human and LLM-assisted review, then supports context-aware horizon selection and pattern-driven evaluation.
- The automated pipeline processes datasets and variates through frequency-aware checks, producing a quality summary and cleaned dataset for later decisions.
- Univariate checks reject nonnumeric, short, excessively missing, constant, low-entropy, or white-noise series and detect extreme outliers.
- LLM-assisted review can confirm or override automated deletions using application context, preserving indispensable series with expected statistical imperfections.
- The LLM configures short-, medium-, and long-term horizons using sampling frequency, domain context, operational cycles, periodicity, and predictability limits.
- MASE normalizes forecast error against a seasonal-naive baseline, while CRPS evaluation uses nine equidistant quantiles for probabilistic forecasts.
- Pattern-driven analysis retrieves variates sharing coded temporal characteristics and aggregates their metrics for cross-dataset model insights.
D.3. Feature Distribution Analysis
The analysis tests whether median-based binary grouping yields discriminative and complementary time-series feature strata. Across 6,625 variates, all six features show strong separation, large effects, and statistically significant group differences.
- Analysis setup: 6,625 time-series variates across the benchmark were analyzed using six extracted features and median-based binary grouping.The analysis reports Fisher Score and Cohen’s d to assess group separation and effect size.
- Discriminative power: Every feature achieved a Fisher Score above the 0.25 threshold, ranging from 1.19 for Trend Linearity to 4.46 for Trend Strength.These scores indicate meaningful distributional separation between the median-defined groups.
- Effect sizes: Cohen’s d ranged from 1.54 for Trend Linearity to 2.98 for Trend Strength, indicating very large practical differences between high and low groups.All reported effect sizes exceed the conventional large-effect threshold of 0.8.
- Feature complementarity: Most feature pairs had low to moderate correlations, while Trend Linearity was nearly independent and Seasonality Strength correlated most with Seasonality Correlation at r = 0.68.Trend Linearity had |r| ≤0.03 with all other features; most pairs had |r| < 0.7.
- Statistical significance: Mann-Whitney U tests found statistically significant differences for every median-based grouping, with p < 0.001 for all features.The results support using these groupings for stratified model-performance analysis.
E. Full Results
The full-results section combines overall benchmark scores, pattern-specific comparisons, dataset-level results, and visual inspection of forecast behavior. It reports normalized MASE and CRPS alongside rankings and prediction examples spanning multiple temporal patterns.
- Overall results: Overall TIME performance is summarized in Table 4 using task-level results normalized by Seasonal Naive and aggregated with a geometric mean.Bold and underlined entries identify the best and second-best performance, respectively.
- Pattern-level results: Pattern-specific evaluation compares normalized MASE and CRPS for variates exhibiting a feature pattern and those that do not.The comparison is reported in Table 5 and visualized through feature-specific variate analyses.
- Dataset-level results: Dataset-level results are provided for all 50 datasets in Tables 6 and its continuation tables.These tables complement the aggregate and pattern-level views with per-dataset results.
- Visual inspection: The interactive platform supports zooming and probabilistic-quantile display, while visualizations distinguish training history, test sets, and target windows by color.Blue marks training history, yellow the overall test set, and red the evaluated target window.
- Forecast behavior: Case studies use global and local views to inspect forecasts across multiple seasonalities, regular dips, level shifts, trends, traffic seasonality, and spike structures.Examples include accurate pattern capture in several series and smooth forecasts that miss distinct spikes in MetroPT-3.
G. Limitations and Future Directions
The paper identifies two limitations: binary pattern grouping restricts analysis granularity because finer intersections become sparse, and generic metrics may not reflect operationally actionable forecast quality.
- Pattern granularity and data sparsity: Median-based binary pattern grouping limits finer-grained analysis because combining multiple feature levels causes subgroup counts to grow exponentially and creates severe data sparsity.The paper proposes expanding the variate scale and improving pattern retrieval to preserve statistical validity.
- Task-aligned evaluation metrics: MASE and CRPS may fail to capture downstream operational value, such as directional accuracy in financial forecasting, and are sensitive to scaling and aggregation.Future evaluation aims to incorporate metrics aligned with specific downstream requirements rather than generic error minimization alone.