PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

Shuang Liang, Chaochuan Hou, Xu Yao, Shiping Wang, Hailiang Huang, Songqiao Han, Minqi Jiang

arXiv:2605.26562v1cs.LG

TL;DR

Holistic benchmarks of multivariate time-series forecasting models leave component-level contributions underanalyzed. TSCOMP systematically benchmarks forecasting components and uses the resulting corpus for data-adaptive model construction, which consistently outperforms state-of-the-art methods.

  • Problem

    Existing benchmarks evaluate multivariate time-series forecasting models holistically, leaving the contributions of their internal mechanisms underanalyzed.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    TSCOMP deconstructs forecasting pipelines into components and uses constrained orthogonal experiments to analyze their effectiveness, interactions, and adaptability across backbones and data characteristics.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across diverse datasets and component combinations, data-adaptive selection and assembly consistently achieves superior performance compared with state-of-the-art methods.

    Source: §3.1 · p. 4

  • Takeaways & Limitations

    The resulting performance corpus supports zero-shot automated model construction on unseen datasets without exhaustive experimentation.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

While previous research in multivariate time series forecasting has focused on developing complex holistic models, this work advocates for a shift toward a granular, component-level understanding of their impacts. We propose TSCOMP, the first large-scale benchmark that systematically deconstructs deep forecasting methods into their core, fine-grained components--spanning series preprocessing, encoding strategies, network architectures including specific and large time-series models, and optimization methods. Using constrained orthogonal experimental design and extensive evaluations, we conduct multi-view analyses that reveal component effectiveness across different backbones, data characteristics, and their interactions. Beyond providing insights, this benchmark establishes a fine-grained performance corpus comprising over 20,000 model-dataset evaluations, which supports the learning of automated component selection, enabling zero-shot model construction on new datasets. Our experiments demonstrate that the corpus-driven approach, despite its simplicity, consistently outperforms state-of-the-art methods, validating the soundness of our evaluation design and confirming that systematic component selection surpasses manually designed complex architectures. All code and the performance corpus are publicly available at https://github.com/SUFE-AILAB/TSCOMP.

1 Introduction

TSCOMP argues that understanding deep multivariate time-series forecasting requires evaluating model components within their pipelines, rather than treating models as indivisible systems. It combines hierarchical benchmarking and multi-view analysis with a corpus for dataset-adaptive model construction.

1 Introduction

  • Holistic benchmarks obscure the contributions of internal mechanisms, making effective designs harder to identify and combine.The paper notes that no single model family dominates across all scenarios, motivating finer-grained investigation.Source: §1 · p. 2
  • TSCOMP hierarchically deconstructs deep forecasting methods into pipelines, component dimensions, and individual components, using constrained orthogonal evaluation to isolate their contributions.The benchmark is presented as the first large-scale benchmark to systematically deconstruct deep multivariate forecasting methods.Source: §1 · pp. 2–3
  • Its multi-view analysis examines component effectiveness across backbones, data domains and characteristics, and interactions among components.The analyses include specific models and emerging large language and time-series foundation models.Source: §1 · p. 3
  • TSCOMP uses its fine-grained performance corpus to adaptively assemble components for forecasting scenarios, consistently outperforming state-of-the-art methods.The corpus supports automated construction of methods tailored to different datasets.Source: §1 · pp. 2–3

2 Related Work

Prior MTSF research spans diverse forecasting architectures, benchmarks, and automation strategies, while TSCOMP frames its contribution as evaluating and selecting components across the forecasting pipeline.

2.1 Deep Learning-based MTSF

  • MTSF has evolved from statistical methods through deep architectures to foundation models, and TSCOMP decomposes these approaches into pipeline components.The approaches include Transformers [40] [59], MLP-based models such as DLinear [64], and foundation models such as GPT4TS [71], Timer [37], and Time-MOE [47].Source: §2.1 · p. 3
  • Preprocessing, multi-scale modeling, channel treatment, tokenization, and dependency mechanisms form modular design dimensions that TSCOMP organizes into a component framework.Prior approaches address non-stationarity with normalization [14] or decomposition [34] [64], and balance channel-independent [40] against channel-dependent modeling [33].Source: §2.1 · p. 3

2.2 Benchmarks for Time Series Forecasting

  • Existing benchmarks evaluate models largely as monoliths, while prior fine-grained studies have limited scope; TSCOMP is presented as the first large-scale benchmark covering the entire MTSF pipeline.TSlib, BasicTS [45], TFB [41], and OpenLTM [37] cover broad model or dataset evaluations, while other work focuses on augmentation [55] or Transformer components [56].Source: §2.2 · p. 3

2.3 AutoML for Time Series Forecasting

  • Prior automated MTSF methods use model-level ensembles [46] or meta-learning, while TimeFuse [38] adaptively fuses forecasters at the sample level; TSCOMP automates component selection.The passage characterizes TSCOMP’s broader component search space as enabling pipelines beyond fixed architectures.Source: §2.3 · p. 3

3 TSCOMP: Benchmarking and Automating Deconstructed Components in Deep MTSF

TSCOMP provides a hierarchical, component-level benchmark for deep multivariate time-series forecasting and uses its performance corpus to automate data-adaptive model construction.

3.1 Overview

  • The component-level corpus enables data-adaptive pipeline selection that consistently outperforms SOTA methods.The overview attributes this result to systematic insights into component effectiveness across varying data characteristics.Source: §3.1 · p. 4

3.2.1 Problem Definition for MTSF. 𝐿

  • For multivariate forecasting, TSCOMP directly predicts all future steps to avoid error accumulation when T>1.Source: §3.2.1 · p. 4

3.2 Hierarchical Deconstruction

  • Because model innovations are coupled with other components, TSCOMP deconstructs forecasting methods to isolate component contributions.The passage frames this modular analysis as a way to identify core elements driving forecasting improvements.Source: §3.2.2 · p. 4

3.2.2 Design Space Construction.

  • The design space spans four pipeline stages, 11 dimensions, and 49 components, organized from workflow stages to concrete implementations.Its stages are preprocessing, encoding, network architecture, and optimization; components are extracted from model papers and source code.Source: §3.2.2 · p. 4

3.3.2 Experimental Protocol.

  • The benchmark standardizes datasets, metrics, and training configurations, using MSE, standardized MSE, and GLMM, ANOVA, and Cohen’s d analyses.It evaluates 13 standard long-term forecasting benchmarks and reports MAE, SMAPE, and MASE results in the appendix.Source: §3.3 · p. 4, §3.3.2 · pp. 4–5

3.3.1 Constrained Orthogonal Experimental Design.

  • Constrained orthogonal sampling covers valid pairwise component interactions, reducing an otherwise intractable space to approximately 136 models per horizon.The design excludes invalid combinations and uses a greedy strategy; exhaustive higher-order coverage is impractical, while single-component analysis misses interactions.Source: §3.3.1 · p. 4

3.4 Automated Model Construction

  • The performance corpus supports zero-shot model construction: a two-layer MLP predicts configuration rankings from dataset meta-features, enabling top-k selection without neural-network training on the new dataset.The method converts benchmark MSE values to normalized rankings and extracts meta-features from the new dataset’s training split.Source: §3.4 · p. 5

4 Experiments

Experiments identify preprocessing and encoding as major drivers of forecasting performance, show that component effectiveness varies by architecture and data characteristics, and demonstrate that corpus-ranked pipelines can support competitive zero-shot forecasting.

4.1.1 Component-Level.

  • RevIN and Stationary yield strong MSE reductions, Channel Independence and Inverted or Orthogonal Encoding outperform point-wise encoding, and HUBER and MAE outperform MSE.Series Decomposition increases MSE on average but has mixed effects; a simple MLP with Sparse Feature Attention performs well, while MSE performs poorly without Series Normalization.Source: §4.1.1 · pp. 5–6, Figure 3 · p. 6

4.1.2 Dimension-Level.

  • Series Normalization explains 63.0% of performance variance; across pipeline stages, preprocessing explains 66.6%, encoding 18.3%, architecture 8.0%, and optimization 7.1%.Preprocessing’s dominance persists across MAE, RMSE, and MASE and across scenarios, while encoding contributes more than architecture.Source: Table 2 · p. 6, §4.1.2 · p. 6, §4.1.3 · p. 6

4.2.1 MLP-Based Architectures.

  • MLPs rely heavily on Series Normalization and benefit from Channel Independence, while RNNs gain nearly twice the global-average benefit from channel isolation but are harmed by Moving Average decomposition.Orthogonal Encoding can help MLPs compensate for limited sequential processing; longer sequence lengths exacerbate RNN gradient vanishing and error accumulation.Source: §4.2.1 · pp. 6–7, §4.2.2 · p. 7

4.2.2 RNN-Based Architectures.

  • Transformers benefit from Orthogonal Encoding and advanced losses but are harmed by decomposition; LLMs favor Moving Average and Multi-scale Mixing, while TSFMs are highly sensitive to tokenization.For TSFMs, patching degrades performance despite its pretraining use, whereas Channel Independence helps; LLMs are comparatively insensitive to Channel Independence.Source: §4.2.3 · p. 7, §4.2.4 · p. 7

4.2.5 Intra-Backbone and Pipeline-Level Analysis.

  • Within-family MLP and RNN variants do not beat their baselines, whereas Frequency Enhanced Attention improves Transformer forecasting; pipeline priorities also differ across model families.MLPs rely on preprocessing, TSFMs emphasize encoding, and Transformers and LLM-based methods rely heavily on architecture design; other LLM and TSFM variants do not surpass GPT4TS in full-shot scenarios.Source: Table 4 · p. 7, §4.2.5 · p. 7

4.4.2 Selection Quality Analysis.

  • 98% of the meta-predictor’s top-five recommendations fall within the top quartile and over 99% within the top half, substantially exceeding random selection.This supports the claim that the benchmark corpus contains learnable patterns useful for selecting pipelines on unseen datasets.Source: Figure 5 · p. 8, §4.4.2 · p. 8

4.3 Data-Specific Analysis.

  • Component preferences vary with data: longer datasets favor MLPs, distribution shift favors Stationary Norm, and correlated variables can make Channel Independence harmful.Auto-correlation and Destationary Attention improve performance in datasets with their corresponding characteristics; RevIN fails under high distribution shift, and large models struggle with longer datasets. Channel strategy should therefore match dataset properties.Source: Figure 4 · p. 7, Table 5 · p. 7, §4.3 · p. 8

4.4.3 Performance Comparison with SOTA.

  • TSCOMP dominates M4 short-term forecasting, achieves state-of-the-art results on 10 of 14 long-term tasks, beats TimeFuse [38] by up to 10.4% in MSE, and surpasses GPT4TS [71].These results use an ensemble of the top-five selected combinations; TSCOMP recommends components zero-shot and fits a lightweight MLP, avoiding the full-shot fine-tuning overhead of large models.Source: Table 6 · p. 8, §4.4.3 · pp. 8–9, Table 7 · p. 8

5 Conclusions and Future Work

TSCOMP decomposes multivariate time-series forecasting pipelines into fine-grained components to analyze their performance and automate customized model construction. The paper reports that TSCOMP-constructed models significantly outperform current MTSF state-of-the-art solutions and outlines an LLM-agent workflow for keeping the performance corpus current.

5 Conclusions and Future Work

  • TSCOMP-constructed MTSF models significantly outperform current state-of-the-art solutions by adaptively customizing models to distinct data characteristics, even without exhaustively covering all state-of-the-art components.The authors release the benchmark code, results, and performance corpus for the MTSF community.Source: §5 · p. 9
  • TSCOMP systematically decomposes MTSF pipelines into component dimensions and design choices, enabling component-level performance analysis and automated construction of customized forecasting models.The framework is presented as a way to advance beyond holistic evaluations.Source: §5 · p. 9
  • Future work will use an LLM-agent-powered workflow to continuously build a knowledge base from emerging papers, decompose new architectures into reusable components, and enable self-evolving model synthesis.The workflow is intended to prevent the performance corpus from becoming an outdated static snapshot.Source: §5 · p. 9

A Datasets … D.2 Meta-Features and Meta-Predictors

The section specifies the evaluation datasets and metrics, then describes TSCOMP’s four-stage, 11-dimension component decomposition. It also frames dataset characterization as a challenge for automated model selection.

A Datasets

  • TSCOMP evaluates 13 long-term forecasting benchmarks plus M4 for short-term tasks, using horizon sets tailored to standard, limited-sample, and M4 datasets.Standard long-term horizons are {96, 192, 336, 720}; limited-sample horizons are {24, 36, 48, 60}; M4 horizons are {6, 8, 13, 14, 18, 48}.Source: A Datasets · pp. 10–11

B Metrics Mathematical Formula

  • Long-term forecasting is evaluated with MSE and MAE, while short-term forecasting uses SMAPE, MASE, and OWA [58].The metric definitions use the data periodicity and compare ground truth with predictions over future time points and dimensions.Source: B Metrics Mathematical Formula · p. 11

C System Configuration

  • All experiments used four NVIDIA A100 GPUs with 80GB and eight 40GB of memory, with experiments run in parallel to reduce total time.Source: C System Configuration · p. 11

D Details of TSCOMP

  • The appendix details TSCOMP’s deconstructed components, extracted meta-features, and trained meta-predictors.Source: D Details of TSCOMP · p. 11

D.1 More Details of Deconstructed Components in TSCOMP.

  • TSCOMP decomposes forecasting design into 11 dimensions across preprocessing, encoding, architecture, and optimization, with architectural constraints filtering invalid component combinations.Preprocessing covers normalization, decomposition, and sampling or mixing; encoding covers channel independence or dependence, tokenization, and timestamp embeddings.Source: §D.1 · p. 11
  • Architectural dimensions include backbones, feature attention, and retrieval-augmented generation, with LLMs and TSFMs also integrated as backbones; optimization covers sequence length and specialized loss functions.The listed backbones include MLPs, RNNs, TCNs, and Transformers with varied attention mechanisms and Mamba hybrids.Source: §D.1 · p. 11

D.2 Meta-Features and Meta-Predictors

  • Handcrafted statistics such as skewness, kurtosis, and entropy capture static marginal distributions but may miss conditional dependencies between historical observations and future targets.Source: §D.2.1 · p. 11

D.2.1 Details of Meta-Features. · D.2.2 Details of Meta-Predictor. · E Additional Experimental Results

TSCOMP encodes dataset dynamics with TabPFN-probed meta-features and learns to rank component configurations from benchmark results for zero-shot selection on new forecasting tasks.

D.2.1 Details of Meta-Features.

  • TabPFN-probed proxy classification maps historical windows to discretized next-step targets, yielding pooled embeddings that capture temporal transition laws rather than static statistics.The method samples channel-time instances, buckets continuous next-step targets into K labels, and aggregates TabPFN representations to form dataset-level meta-features.Source: §D.2.1 · p. 12
  • Meta-feature distances are negatively correlated with model-performance rank consistency, and the representation strategy surpasses TSFEL and TimeFuse [38].The authors interpret the negative correlation as evidence that nearby datasets in meta-feature space have more similar model performances.Source: §D.2.1 · pp. 12–13, Table 10 · p. 13
  • TabPFN-derived meta-features have a normal-like distribution, providing greater numerical stability during meta-predictor optimization than often-skewed handcrafted statistics.Source: §D.2.1 · p. 13

D.2.2 Details of Meta-Predictor.

  • TSCOMP customizes forecasting models through constraint-validated fine-grained component selection, learning configuration-to-performance mappings for new tasks.Source: §D.2.2 · p. 13
  • The performance corpus contains approximately 20,760 entries from constrained-orthogonal and random configuration pools evaluated across four horizons and 13 training datasets.The random pool was evaluated on a subset of seven training datasets; MSE results are converted to normalized rankings so the predictor learns relative model quality across datasets.Source: §D.2.2 · p. 13
  • The meta-predictor represents component choices with learnable embeddings, combines them with dataset meta-features, and uses Pearson loss to learn relative performance rankings.Source: §D.2.2 · p. 13
  • At inference, a two-layer MLP ranks candidate configurations from a new dataset’s meta-features without model training, enabling immediate top-k selection instead of exhaustive local experimentation.The predictor is pretrained on benchmarking results from training datasets for deployment on new forecasting tasks.Source: §D.2.2 · p. 13

E.1 Analysis of Higher-Order Component Interactions

Higher-order interactions are statistically prevalent, but their contribution to performance variance is modest relative to main effects. The analysis therefore supports using main effects as a pragmatic proxy for component ranking and automated search.

E.1 Analysis of Higher-Order Component Interactions

  • The study used Type III ANOVA with treatment contrasts to model 54 feasible pairwise terms and 66 estimable three-way combinations, applying partial η^2 with FDR correction.Nested F-tests were used for three-way combinations because of rank deficiency.Source: §E.1 · p. 13
  • FDR correction finds 30/54 pairwise and 58/66 three-way combinations significant, including Attention Type × Loss Function and Normalization × Backbone synergies.The analysis identifies these specific component pairings as statistically significant in model dynamics.Source: §E.1 · p. 13
  • Pairwise interactions raise R^2 by 5.27%, from 27.29% to 32.56%, while main effects explain 83.8% of explainable variance and individual interaction η^2 peaks at 0.043.The authors describe main effects as a robust, pragmatic proxy that avoids modeling all n-way interactions.Source: §E.1 · p. 13, Table 11 · p. 13

E.2 Robustness and Generalization of Preprocessing Dominance · E.3 Justification for Backbone Corpus Design

Preprocessing remains the leading contributor across scale-sensitive and scale-independent metrics and across scenarios, while expanded backbone experiments find no significant gains over MLP-only configurations. The findings support preprocessing’s importance and show that the automation framework identifies optimal configurations across backbone variety.

E.2 Robustness and Generalization of Preprocessing Dominance

  • Under MSE, Series Preprocessing explained 66.6% of variance (Series Normalization 63.0%) versus Network Architecture’s 8.0%; under MASE, contributions were 58.7% and 5.2%.The preprocessing-to-architecture importance ratio rose from 8.3 under MSE to 11.3 under MASE.Source: Table 13 · p. 14, §E.2 · p. 14
  • Preprocessing dominance persisted across the evaluated scale-sensitive and scale-independent metrics, supporting its interpretation as an inherent MTSF property rather than an evaluation artifact.The analysis included MAE, RMSE, and MASE; the passage reports preprocessing ranked first across all evaluated metrics.Source: §E.2 · pp. 13–14
  • The Performance-to-Volatility Ratio uses mean performance score divided by its standard deviation across 52 scenarios to reward consistently high-ranking configurations.Within each dataset–horizon scenario, combinations are ranked by MSE and converted into performance scores.Source: §E.2 · p. 14
  • Under the robustness dimension, Series Preprocessing was the most critical pipeline stage, contributing 55.5% to overall model robustness; the other three stages had comparable importance of approximately 14–16%.Source: Table 14 · p. 14, §E.2 · p. 14
  • The robustness and metric analyses associate preprocessing’s dominance with handling non-stationarity and distribution shift being more fundamental than architectural complexity.The passages characterize the conclusion as supported by metric-invariant and scenario-robust evaluations.Source: §E.2 · p. 14

E.3 Justification for Backbone Corpus Design

  • Despite negligible performance differences as search complexity increases, the automation framework consistently identifies optimal configurations across backbone variety, making its meta-learning methodology architecture-agnostic.The identified optimal configurations were predominantly MLP-based.Source: §E.3 · p. 14
  • MLP backbones ranked first in 164 of 168 evaluation scenarios, and adding RNN and Transformer architectures yielded no significant performance improvements over the MLP-only corpus.The scenarios covered 7 datasets, 4 prediction lengths, 3 Top-K settings, and both mean and median MSE values.Source: §E.3 · p. 14, Table 15 · p. 15

E.4 Experimental Cost Analysis

TSCOMP’s corpus-building cost is a one-time offline investment, while zero-shot recommendations enable efficient deployment on new datasets. On ETTh1, TSCOMP-fast is reported to be nearly 7× faster than AutoGluon while achieving superior accuracy.

E.4 Experimental Cost Analysis

  • TSCOMP’s offline corpus-building cost is front-loaded and decoupled from deployment, while zero-shot meta-learning recommends lightweight MLP configurations for rapid fitting and inference.AutoGluon incurs iterative search on each new task, whereas TimeFuse relies on subsequence ensembling.Source: §E.4 · p. 15
  • Nearly 7× faster than AutoGluon, TSCOMP-fast achieves superior accuracy in the ETTh1 cost comparison.The evaluation uses prediction length 96 and 8 A800 GPUs; Table 16 compares offline preparation time, online processing time, and predictive performance.Source: Table 16 · p. 15, §E.4 · p. 15
  • TSCOMP-fast excludes RAG, Series Sampling, and Series Decomposition, unlike the complete TSCOMP-standard framework.The comparison also specifies different time and per-run limits for Auto-sklearn-standard and Auto-sklearn-fast.Source: §E.4 · p. 15

E.5 Comprehensive Results of TSCOMP Against State-of-the-Art Methods · E.6 Dimension Importance Analysis

The appendix reports comprehensive long- and short-term forecasting comparisons that validate TSCOMP’s competitive performance across temporal prediction scenarios. It also analyzes dimension importance and pipeline stages across architectures.

E.5 Comprehensive Results of TSCOMP Against State-of-the-Art Methods

  • TSCOMP’s comprehensive long- and short-term forecasting evaluations validate its competitive performance across diverse temporal prediction scenarios.Table 17 covers long-term forecasting across prediction horizons, and Table 18 presents short-term results.Source: §E.5 · p. 15

E.6 Dimension Importance Analysis

  • The dimension-importance analysis examines Effect Range and pipeline stages across architectures.The analysis is presented in Fig. 7 and Fig. 8.Source: §E.6 · p. 15

E.7 Detailed Component Analysis … E.7.4 Network Optimization.

The appendix presents component-wise performance distributions and dataset-adaptability analyses across components and architectures, alongside full long- and short-term forecasting results.

E.7 Detailed Component Analysis

  • The component analysis uses ridgeline plots and radar charts to visualize performance across datasets and architectures.The plots show performance distributions and dataset adaptability for each component.Source: §E.7 · p. 15
  • Series normalization, decomposition, and sampling/mixing are examined through performance-distribution and dataset-adaptability visualizations.These analyses are shown in Figures 9–14.Source: §E.7.1 · p. 15

E.7.1 Series Preprocessing.

  • The preprocessing analysis examines channel independence, timestamp embeddings, and series tokenization.Each component is accompanied by performance-distribution and dataset-adaptability analyses.Source: §E.7.2 · p. 15

E.7.2 Series Encoding.

  • The encoding analysis covers network backbones, feature attention mechanisms, and retrieval augmented generation.The components are examined through performance-distribution and dataset-adaptability visualizations.Source: §E.7.3 · p. 15

E.7.3 Network Architecture.

  • The network-architecture analysis evaluates sequence-length configurations and loss functions.It uses performance-distribution and dataset-adaptability visualizations.Source: §E.7.4 · p. 15

E.7.4 Network Optimization.

  • Tables 17 and 18 report full long-term results across horizons 96, 192, 336, and 720 and short-term M4 results, highlighting the first and second best values.Lower MSE and MAE indicate superior long-term accuracy; lower OWA, SMAPE, and MASE indicate superior short-term accuracy.Source: Table 17 · p. 16, Table 18 · p. 17
  • Dimension-importance and pipeline-stage-importance plots visualize performance distributions across model architectures.These are presented as effect-range plots and pipeline-importance plots, respectively.Source: Figure 7 · p. 17, Figure 8 · p. 17

Related papers

Loading 2605.26562v1…