PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

Shuang Liang, Chaochuan Hou, Xu Yao, Shiping Wang, Hailiang Huang, Songqiao Han, Minqi Jiang

arXiv:2605.26562v1cs.LG

TL;DR

Deep MTSF research has emphasized complex holistic models, while existing benchmarks provide limited evidence about the contributions of their internal components. TSCOMP systematically deconstructs and evaluates these components with constrained orthogonal experiments, then uses the resulting corpus for automated construction that consistently outperforms current state-of-the-art solutions.

  • Problem

    Existing benchmarks evaluate complete forecasting models as unified entities, obscuring component contributions and limiting comprehensive analysis across the MTSF pipeline.

  • Method

    TSCOMP hierarchically deconstructs deep MTSF methods across pipeline, dimension, and component levels and evaluates them with a constrained orthogonal experimental protocol.

  • Results

    TSCOMP-constructed MTSF models significantly outperform current state-of-the-art solutions across distinct data characteristics.

  • Takeaways & Limitations

    The fine-grained performance corpus supports automated construction of forecasting models customized to different datasets and forecasting scenarios.

Abstract

from arXiv · show

While previous research in multivariate time series forecasting has focused on developing complex holistic models, this work advocates for a shift toward a granular, component-level understanding of their impacts. We propose TSCOMP, the first large-scale benchmark that systematically deconstructs deep forecasting methods into their core, fine-grained components--spanning series preprocessing, encoding strategies, network architectures including specific and large time-series models, and optimization methods. Using constrained orthogonal experimental design and extensive evaluations, we conduct multi-view analyses that reveal component effectiveness across different backbones, data characteristics, and their interactions. Beyond providing insights, this benchmark establishes a fine-grained performance corpus comprising over 20,000 model-dataset evaluations, which supports the learning of automated component selection, enabling zero-shot model construction on new datasets. Our experiments demonstrate that the corpus-driven approach, despite its simplicity, consistently outperforms state-of-the-art methods, validating the soundness of our evaluation design and confirming that systematic component selection surpasses manually designed complex architectures. All code and the performance corpus are publicly available at https://github.com/SUFE-AILAB/TSCOMP.

1 Introduction

TSCOMP addresses the limitations of holistic deep MTSF evaluation by systematically decomposing forecasting models into hierarchical components and benchmarking their effects. Its corpus supports automated, data-adaptive model construction that consistently surpasses state-of-the-art methods.

  • Benchmarking Methodology: The benchmark evaluates preprocessing, encoding, network architecture, and optimization through component-, dimension-, and pipeline-level analyses.Reported evaluation uses performance metrics including MSE, MAE, OWA, SMAPE, MASE, and standardized MSE.
  • Existing benchmarks evaluate models holistically, obscuring internal mechanism contributions and hindering combinations of effective designs.
  • Hierarchical Deconstruction: TSCOMP hierarchically deconstructs MTSF methods into pipeline stages, component dimensions, and concrete components for systematic benchmarking.The design spans four pipeline stages, 11 component dimensions, and 49 deconstructed components.
  • Benchmarking Methodology: A constrained orthogonal protocol isolates individual component contributions while enabling analyses across backbones, data characteristics, domains, and component interactions.
  • Automated Model Construction: TSCOMP establishes a fine-grained performance corpus that supports automated model construction and data-adaptive component selection.The framework uses a pre-trained meta-predictor for zero-shot selection on new datasets.
  • Automated Model Construction: The corpus-driven construction approach consistently achieves better results than state-of-the-art methods.

2 Related Work

Related work progresses from statistical and deep forecasting models toward modular strategies for non-stationarity, temporal dependencies, tokenization, and channel treatment. However, existing benchmarks largely assess complete models, leaving comprehensive component-level evaluation of the MTSF pipeline unresolved.

  • MTSF research spans statistical methods, RNNs, Transformers, MLPs, decomposition, multi-scale analysis, and foundation-model approaches.
  • Recent designs address non-stationarity through normalization or decomposition and capture temporal dependencies through multi-scale modeling.
  • Architectures balance channel-independent processing with channel-dependent correlation modeling, while tokenization ranges from point-wise to series-wise representations.
  • Existing forecasting benchmarks conduct broad evaluations but generally treat models as monolithic entities.
  • TSCOMP addresses this limitation by evaluating the entire MTSF pipeline through fine-grained component analysis and large-scale empirical benchmarking.
  • Automated MTSF methods select or combine models through ensembles, meta-learning, or sample-level fusion, operating largely at the model level.

3 TSCOMP: Benchmarking and Automating Deconstructed Components in Deep MTSF

TSCOMP deconstructs deep MTSF methods into a hierarchical component space and evaluates combinations systematically, then uses the resulting corpus for data-adaptive model construction.

  • Benchmark Framework: TSCOMP organizes MTSF designs across pipeline, dimension, and component levels to evaluate modular forecasting choices rather than only complete models.The pipeline includes series preprocessing, series encoding, network architecture, and network optimization.
  • Empirical Outcome: Large-scale evaluations yield systematic component insights across data characteristics and enable data-adaptive pipelines that consistently outperform state-of-the-art methods.The reported comparison concerns the assembled forecasting pipelines produced from component-level insights.
  • Design Space Construction: 4 pipeline stages, 11 component dimensions, and 49 deconstructed components define the benchmark’s structured design space.The component combinations are formed through a Cartesian product, with invalid combinations excluded for architectural soundness.
  • Experimental Design: A constrained orthogonal experimental protocol covers component interactions while standardized datasets, metrics, and training configurations support fair comparison.The protocol uses valid model pools and evaluates extensive combinations under controlled conditions.
  • Performance Corpus: The benchmark evaluates constraint-validated configurations across training datasets to construct a performance matrix capturing fine-grained component–data interactions.This corpus is intended to support automated model construction without exhaustive experimentation on new datasets.
  • Automated Model Construction: The meta-predictor uses dataset meta-features and component embeddings to predict normalized configuration rankings and select top-k components for new datasets.At test time, recommendations require no neural network training on the new dataset’s test split.

4 Experiments

TSCOMP evaluates forecasting components at multiple levels, finding that preprocessing and encoding generally matter more than architecture and optimization. Component effects also vary by backbone and dataset characteristics, while corpus-based recommendations achieve strong automated selection and forecasting performance.

  • Component-Level Analysis: RevIN and Stationary provide strong normalization gains, Channel Independence generally improves performance, and Inverted and Ortho tokenization outperform Point-wise Encoding.HUBER and MAE losses also significantly outperform MSE, while decomposition methods increase MSE on average.
  • Component Interaction Analysis: A simple MLP combined with Sparse Feature Attention performs well, whereas standard MSE performs poorly without Series Normalization.These interactions illustrate both lightweight architectural synergy and sensitivity of MSE to distribution shifts.
  • Dimension-Level Analysis: 63.0% of performance variance is explained by Series Normalization, exceeding the contributions of Channel Independence (11.1%) and Series Tokenization (7.1%).The ANOVA identifies normalization as the primary driver within the contemporary design space.
  • Pipeline-Level Analysis: Series Preprocessing explains 66.6% of total variance, compared with 18.3% for Series Encoding, 8.0% for Network Architecture, and 7.1% for Network Optimization.The preprocessing advantage persists across MAE, RMSE, and MASE and across evaluated scenarios.
  • Architecture-Specific Analysis: Component preferences differ across architectures: LLMs benefit from Moving Average Decomposition and Multi-scale Mixing, while Transformers favor Orthogonal Encoding and advanced loss functions.MLPs rely heavily on preprocessing, TSFMs emphasize encoding, and Transformers and LLMs rely more on network architecture.
  • Data-Specific Analysis: Channel Independence significantly degrades performance on highly correlated datasets, while Stationary Norm is preferred under strong distribution shift.Other mechanisms also show context dependence: Auto-correlation helps highly autocorrelated data, and Destationary Attention helps strongly non-stationary data.
  • Automated Model Construction: 98% of meta-predictor recommendations fall within the top quartile, and over 99% within the top half of selections.This exceeds random selection and indicates learnable patterns in the benchmark corpus.
  • Performance Comparison: TSCOMP dominates M4 short-term forecasting and achieves state-of-the-art results on 10 of 14 long-term forecasting tasks.The gains are obtained with simple MLPs, supporting the importance of precise component selection over architectural complexity.

5 Conclusions and Future Work

The conclusion presents TSCOMP as a framework for fine-grained component analysis and automated construction of specialized MTSF models. Its experiments report superior performance to current solutions, while future work will keep the performance corpus current through an LLM-agent workflow.

  • Conclusions: TSCOMP decomposes MTSF pipelines into component dimensions and design choices to support customized forecasting models.The released code, results, and performance corpus are intended to benefit the MTSF community.
  • Conclusions: TSCOMP-constructed models significantly outperform current MTSF state-of-the-art solutions across distinct data characteristics.The conclusion also states that this effectiveness does not require exhaustive coverage of all state-of-the-art components.
  • Future Work: Future work will use an LLM-agent-powered workflow to turn the static performance corpus into a continuously updated knowledge base.The workflow is intended to incorporate emerging papers and systematically decompose their methods.

A Datasets

The experiments use standard long-term and short-term forecasting datasets with task-specific horizons and metrics. TSCOMP organizes model design into four pipeline stages and evaluates component choices using dataset characteristics and meta-features.

  • Datasets: The evaluation covers 13 standard long-term forecasting benchmarks and the M4 dataset for short-term forecasting.The long-term set includes ETT variants, Electricity, Traffic, Weather, Exchange, ILI, FRED-MD, NASDAQ, NYSE, and Covid-19.
  • Forecast Horizons: Long-term horizons are {96, 192, 336, 720}, with shorter horizon sets for limited-sample datasets and M4.ILI, NYSE, NASDAQ, Fred-MD, and Covid-19 use {24, 36, 48, 60}; M4 uses {6, 8, 13, 14, 18, 48}.
  • Evaluation Metrics: Long-term tasks use MSE and MAE, while short-term tasks use SMAPE, MASE, and OWA.The metrics follow the experimental setup of prior work.
  • Pipeline Design: TSCOMP unifies MTSF design into Series Preprocessing, Series Encoding, Network Architecture, and Network Optimization.These stages contain 11 component dimensions.
  • Pipeline Components: The preprocessing stage includes normalization, decomposition, and series sampling or mixing, while encoding covers channel strategy and tokenization.Examples include RevIN, Stationary-related methods, moving-average decomposition, Channel Independence, and patch-based tokenization.
  • Pipeline Components: Network Architecture includes backbones, feature attention, and retrieval-augmented generation, while optimization includes sequence length and specialized loss functions.The optimization choices define the final training objective for the assembled pipeline.
  • Dataset Characterization: Automated selection uses dataset characterization because handcrafted marginal statistics may miss conditional dependencies between historical observations and future targets.The paper motivates richer dataset representations for adaptive model selection.

D.2.1 Details of Meta-Features.

TSCOMP converts each dataset into a proxy tabular task, uses TabPFN to encode conditional temporal dynamics, and combines these dataset meta-features with component representations for zero-shot configuration ranking.

  • Meta-feature construction: TSCOMP samples channel–time instances from multivariate series and discretizes continuous future targets into K categorical bins.The resulting proxy dataset maps historical windows to future states.
  • Meta-feature construction: TabPFN processes the proxy tabular dataset, and mean pooling of intermediate representations produces a dataset-level meta-feature.The encoder is a pretrained tabular foundation model.
  • Meta-feature interpretation: The proxy mapping captures conditional predictive relationships P(Y|X), so embeddings represent temporal transition laws rather than only marginal statistics.This design encodes the dataset’s intrinsic predictive logic.
  • Empirical validation: Meta-features show a negative correlation between dataset distances and model-performance rank inconsistencies, indicating semantically consistent dataset representations.The representation strategy substantially surpasses TSFEL and TimeFuse.
  • Meta-predictor: The meta-predictor combines dataset meta-features with dense embeddings of decomposed component indices to learn configuration performance rankings.It is optimized with Pearson loss to emphasize correlation between predicted and actual rankings.
  • Automated model construction: A corpus of approximately 20,760 entries supports zero-shot selection of top-k component combinations on new datasets without exhaustive target-dataset experimentation.The corpus combines constrained orthogonal and random configuration pools across training datasets and prediction horizons.

E.1 Analysis of Higher-Order Component Interactions

TSCOMP finds that higher-order component interactions are common statistically, but their incremental explanatory power is small compared with main effects.

  • Analysis design: Type III ANOVA with treatment contrasts evaluates 54 feasible pairwise terms and 66 estimable three-way combinations using partial η^2 and FDR correction.Nested F-tests address rank deficiency among three-way combinations.
  • Interaction prevalence: 30 of 54 pairwise and 58 of 66 three-way combinations are significant after FDR correction.Significant synergies include Attention Type × Loss Function and Normalization × Backbone.
  • Variance contribution: Pairwise interactions increase total R^2 by only 5.27%, from 27.29% to 32.56%, while main effects explain 83.8% of explainable variance.The maximum individual interaction effect size is η^2 = 0.043.

E.2 Robustness and Generalization of Preprocessing Dominance

Across scale-sensitive, scale-independent, and volatility-aware evaluations, series preprocessing remains the dominant contributor to forecasting performance and robustness.

  • Metric robustness: Under MSE, Series Preprocessing explains 66.6% of variance, including 63.0% from Series Normalization, versus 8.0% for Network Architecture.Under MASE, preprocessing remains dominant at 58.7%, while architecture falls to 5.2%.
  • Metric robustness: The preprocessing-to-architecture importance ratio rises from 8.3 under MSE to 11.3 under MASE.This comparison tests whether preprocessing dominance depends on metric scale.
  • Robustness measure: The Performance-to-Volatility Ratio uses mean performance divided by its standard deviation across dataset–horizon scenarios to assess robust transferability.The ratio rewards consistently high-ranking configurations while penalizing fluctuations.
  • Robustness results: Series Preprocessing contributes 55.5% to overall robustness, while the other three pipeline stages contribute comparably at approximately 14–16%.This result is reported under the volatility-aware robustness analysis.
  • Interpretation: Metric-invariant and scenario-robust analyses suggest preprocessing dominance is inherent to MTSF rather than an evaluation artifact.The paper links this dominance to handling non-stationarity and distribution shift.
  • Backbone robustness: MLPs rank first in 164 of 168 evaluation scenarios, and adding RNN or Transformer backbones yields no significant performance improvement over the MLP-only corpus.The expanded backbone comparison supports restricting automated construction to MLP configurations.

E.4 Experimental Cost Analysis

TSCOMP front-loads substantial corpus-building computation to provide lightweight zero-shot model recommendations and efficient deployment on new forecasting tasks.

  • Cost structure: Building the algorithm corpus requires significant offline computation, but this investment is decoupled from practical deployment.The paper compares offline preparation, online processing, and predictive performance on ETTh1 with prediction length 96 using 8 A800 GPUs.
  • Online efficiency: Zero-shot recommendations instantly identify lightweight MLP configurations, reducing model-fitting and inference costs on new datasets.The meta-learning strategy avoids repeating exhaustive local searches during deployment.
  • Compared variants: TSCOMP-standard includes RAG, Series Sampling, and Series Decomposition, whereas TSCOMP-fast excludes these components.The distinction defines the two framework variants used in the cost comparison.

E.5 Comprehensive Results of TSCOMP Against State-of-the-Art Methods

The appendix provides comprehensive long- and short-term forecasting comparisons, while also analyzing the importance of dimensions and pipeline stages across architectures.

  • Tables 17 and 18 provide complete performance comparisons for long-term and short-term forecasting tasks, respectively.Table 17 covers horizons 96, 192, 336, and 720; Table 18 covers the M4 short-term task.
  • Dimension importance and pipeline-stage importance are analyzed across different architectures in Figures 7 and 8.

E.7 Detailed Component Analysis

The detailed component analysis visualizes performance distributions and dataset adaptability across preprocessing, encoding, architectural, augmentation, configuration, and optimization components.

  • Overall component analysis: Ridgeline plots and radar charts visualize component performance distributions across datasets and architectures, together with dataset adaptability.
  • Series preprocessing: The analysis covers series normalization, series decomposition, and series sampling/mixing.
  • Encoding strategies: Channel independence, timestamp embeddings, and series tokenization are examined as representation and encoding components.
  • Network and augmentation components: Network backbones, feature attention mechanisms, and retrieval augmented generation are analyzed across model architectures.
  • Optimization and configuration: Sequence length configurations and loss functions are evaluated as additional experimental components.
  • Architecture comparison: The component visualizations compare performance across architectures including LLM, MLP, RNN, TSFM, and Transformer categories.
Loading 2605.26562v1…