PaperlayerView exact revision on arXiv

Source-linked AI summary

Channel Dependence, Limited Lookback Windows, and the Simplicity of Datasets: How Biased is Time Series Forecasting?

Ibram Abdelmalak, Kiran Madhusudhanan, Jungmin Choi, Christian Kloetergens, Vijaya Krishna Yalavarit, Maximilian Stubbemann, Lars Schmidt-Thieme

arXiv:2502.09683v3cs.LG

TL;DR

LTSF studies often fix the lookback window despite evidence that its optimum varies by model and dataset, complicating fair comparisons. The paper evaluates lookback choices and channel dependence using performance metrics, Granger causality, and ODE datasets. It finds that CD models win on highly coupled ODE datasets, whereas standard datasets favor CI models.

  • Problem

    Lookback windows are often fixed arbitrarily, although their optima vary across models and datasets, and standard benchmarks show limited channel dependence.

    Source: §1 · pp. 2–3

  • Method

    The study tunes lookback windows and combines forecasting metrics with Granger causality analysis and ODE datasets to assess how channel dependence affects multivariate forecasting.

    Source: §1 · pp. 2–3

  • Results

    Optimal lookback varies by model and dataset; CD variants consistently outperform CI counterparts on highly coupled ODE datasets, while standard datasets’ simplicity favors CI models.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Tune lookback windows for fair comparisons, favor CI models on standard datasets, and use statistical tests of channel correlations to guide CI/CD selection.

    Source: §7 · pp. 10–11

    The findings’ broader applicability remains uncertain for highly stochastic real-world domains, and observed CI/CD dynamics are conditioned on current model families.

    Source: §7 · p. 11

Abstract

from arXiv · show

In Long-term Time Series Forecasting (LTSF), the lookback window is a critical hyperparameter often set arbitrarily, undermining the validity of model evaluations. We argue that the lookback window must be tuned on a per-task basis to ensure fair comparisons. Our empirical results show that failing to do so can invert performance rankings, particularly when comparing univariate and multivariate methods. Experiments on standard benchmarks reposition Channel-Independent (CI) models, such as PatchTST, as state-of-the-art methods. However, we reveal this superior performance is largely an artifact of weak inter-channel correlations and simplicity of patterns within these specific datasets. Using Granger causality analysis and ODE datasets (with implicit channel correlations), we demonstrate that the true strength of multivariate Channel-Dependent (CD) models emerges on datasets with strong, inherent cross-channel dependencies, where they significantly outperform CI models. We conclude with four key recommendations for improving TSF research: (i) consider the lookback window as a key hyperparameter to tune, (ii) for standard datasets, examining CI architectures is advantageous, (iii) leverage statistical analysis of datasets to guide the choice between CI and CD architectures, and (iv) prefer CD models in scenarios with limited data.

1 Introduction

The introduction argues that fair LTSF comparisons require model- and dataset-specific lookback tuning. It also finds that channel dependence in the data shapes whether CI or CD models perform better.

1 Introduction

  • Because lookback is often fixed arbitrarily at 96 despite model-specific optima, fixed-window comparisons can obscure state-of-the-art performance.The study tunes all hyperparameters for each lookback window to support reliable evaluation.Source: §1 · p. 2
  • Optimal lookback depends on dataset and architecture: CI models typically favor 336–512, while CD models often thrive at 96.Crossformer-CD beats TimeMixer-CD at L = 96 but falls behind at L = 336.Source: §1 · p. 2
  • Standard multivariate benchmarks have very limited channel dependence, and the study combines performance metrics with Granger causality analysis and complex ODE datasets to examine its relevance.The analysis finds that multivariate-modeling benefits depend on available history and the degree of causal influence among channels.Source: §1 · p. 3
  • CD variants consistently outperform CI counterparts on highly coupled ODE datasets, while standard datasets’ simplicity favors CI models.Table 1 reports win frequencies across six ODE and seven standard datasets.Source: §1 · p. 3, Table 1 · p. 3

2 Related Work

Prior LTSF comparisons often favor Channel-Independent models on benchmarks with limited channel dependence, while potentially masking Channel-Dependent models’ capacity to capture spatial dependencies. The study addresses lookback-window bias and evaluates models on ODE-based data.

2 Related Work

  • Common benchmarks’ predominantly univariate nature can mask the potential of Channel-Dependent models to capture complex spatial dependencies.Recent approaches include TimeFilter’s graph filtration and DUET’s hierarchical clustering.Source: §2 · p. 4
  • ODE collections such as Physiome and chaotic ODE benchmarks provide mathematically defined channel correlations for testing multivariate forecasting.Standard benchmarks often show limited channel dependence.Source: §2 · p. 4
  • The study systematically tunes lookback windows for all models and evaluates them on ODE-based datasets to address lookback bias and benchmark coupling.Earlier benchmark efforts advocate broad evaluation but do not address these compounding issues.Source: §2 · p. 4

3 Problem Statement

The forecasting task is formulated using windowed input and target segments sampled from a time series. A model maps each input window to a forecast, with training framed as minimizing expected loss over the windowed pairs.

3 Problem Statement

  • A sliding window divides the series into an input segment of length L and a target segment of length H, forming pairs (X, Y) ∈ R^C×L × R^C×H.X contains the lookback observations and Y the following forecast-horizon observations.Source: §3 · p. 4
  • The forecasting model maps X from R^C×L to a prediction in R^C×H and is trained on many windowed pairs sampled from the full sequence.The training dataset is assumed to be drawn i.i.d. from an underlying distribution.Source: §3 · p. 4
  • Training minimizes expected loss, typically MSE, while treating windowed pairs as realizations of a stationary process for supervised optimization.This formulation lets the model focus on temporal and inter-channel dependencies within each local window.Source: §3 · p. 4

4 Background

The background frames lookback length as a model-specific design choice and introduces channel independence and Granger causality. Granger causality compares restricted and unrestricted residual errors to assess whether another channel adds predictive information.

4 Background

  • Most LTSF models use a fixed lookback length, commonly L = 96, but the lookback window should be treated as a model-specific hyperparameter.Fixed lengths are often used for implementation convenience or benchmarking conventions.Source: §4 · p. 5
  • A model is channel-independent when one univariate predictor f maps each channel’s history to that channel’s forecast.The background notes that CI models often suffice on widely used benchmarks.Source: §4 · p. 5
  • Granger causality tests whether channel c2 adds predictive information for c1 beyond c1’s own history by comparing restricted and unrestricted residual sums of squares.The unrestricted model incorporates c2; the restricted model follows the CI assumption.Source: §4 · p. 5
  • The lagged values of c1 and c2 enter the residual models through learned coefficients, and a high F-statistic indicates that c2 Granger-causes c1.The F-statistic measures the significance of the error reduction.Source: §4 · p. 5

5 General Experimental Setup

The experimental setup compares multiple forecasting architectures on standard benchmarks and chaotic ODE datasets. Lookback windows are tuned for each model–dataset–horizon, and Granger-causality statistics characterize channel dependence.

5.1 Models

  • The evaluated baselines include MLP- and attention-based models and DLinear, with selected models tested in CI or CD variants.DLinear-CI and iTransformer-CD are used to avoid architectural overlap with TSMixer.Source: §5.1 · pp. 5–6

5.2 Datasets

  • The dataset evaluation contrasts widely used Weather, Electricity, Traffic, and ETT benchmarks with a chaotic ODE collection designed for challenging CD forecasting.The ODE benchmark exhibits recurrent patterns across channels, including synchronized angular velocities in the Double Pendulum example.Source: §5.2 · p. 6
  • ODE datasets are selected for complex trajectories and direct derivation from underlying equations, which provide mathematical correlations between variables.High Largest Lyapunov exponent values are used to select complex datasets where one channel alone cannot easily solve forecasting.Source: §5.2 · pp. 6–7

5.3 Hyperparameter Tuning

  • Bayesian optimization uses 20 trials per model–dataset–horizon, tuning L across {96, 192, 336, 512, 720} and selecting by validation MSE.Testing uses drop_last=False so trailing points are not discarded.Source: §5.3 · p. 7

5.4 Granger Causality

  • Granger analysis assesses stationarity, iteratively differences where necessary, and removes one channel from pairs with Pearson correlation above 0.95.The reported measures are average F-score and the percentage of pairs rejecting the null hypothesis.Source: §5.4 · p. 7

6 Results

Results show that model rankings depend on lookback-window tuning and dataset structure: CI models lead on standard benchmarks, while CD models win more often on coupled ODE data. The correlation and window analyses help explain these contrasts.

6.1 Evaluation of Standard Datasets

  • PatchTST ranks first on three of seven standard datasets and second or third on three others, with an average rank of 2.14, 0.57 ahead of CI TSMixer.It falls outside the top three only on Weather; the passage links this likely to stronger inter-channel dependencies. Table 2 reports mean MSE across four horizons.Source: §6.1 · p. 8, Table 2 · p. 8

6.2 Granger Causality for Evaluation of Channel Correlations

  • Granger tests find stronger inter-channel dependencies in ODE datasets than standard datasets at all tested lags; standard-dataset F-scores sometimes fall below 1.0 at longer lags.As lag increases, correlations generally decline, especially in standard datasets; ODE F-scores remain 88.5 at lag 96 and 35.29 at lag 192.Source: §6.2 · p. 8, Table 3 · p. 9

6.3 Evaluation on the ODE Benchmark

  • CD variants outperform CI models in 18 of 24 ODE experiments, while DLinear ranks last on five of six ODE datasets.DLinear has an average rank of 9.83; the authors argue that architectures must account for intrinsic inter-channel dependencies on ODE-like data.Source: §6.3 · p. 9, Table 4 · p. 9

6.4 Lookback Window Analysis

  • CI models choose 96 in only 11% of best-window cases, whereas CD models favor 96 or 192 in 55% of cases.The authors suggest CI models use longer histories to compensate for lacking cross-channel information, while long windows may add redundant inter-channel noise for CD models.Source: §6.4 · p. 10, Figure 3 · p. 10
  • Hourly ETT datasets typically favor windows around 96, while 15-minute datasets favor windows greater than 336 to cover equivalent temporal spans.The passage relates optimal window length to both temporal span and a model’s capacity to process channel versus temporal information.Source: §6.4 · p. 10

7 Conclusion and Recommendations

The conclusion recommends tuning lookback windows and matching channel modeling to dataset properties. It also identifies limits on the window search, metrics, data domains, and model families considered.

7 Conclusion and Recommendations

  • The authors recommend optimizing lookback windows rather than fixing them at a common value, to support balanced comparisons.They state that fixing the window, commonly at 96, biases results against univariate models and masks performance differences.Source: §7 · p. 10
  • For standard datasets, the authors recommend CI models; for datasets with strong inter-channel correlations, they recommend CD models and statistical pre-assessment.They report CI models generally outperforming CD models on standard datasets by a margin of 21 to 7, while ODE datasets support CD models.Source: §7 · p. 11
  • The evaluation searches only five discrete lookback windows and relies primarily on MSE, which may not capture performance when distributional or scale-invariant metrics are prioritized.The tested windows are 96, 192, 336, 512, and 720.Source: §7 · p. 11
  • Whether the findings transfer from chaotic ODE simulations to highly stochastic real-world domains remains for future work; hybrid architectures may also shift CI/CD performance patterns.Source: §7 · p. 11

A Description of All Datasets

The supplementary material describes the datasets, search and training procedures, channel-causality analysis, model variants, and reporting of detailed results.

A Description of All Datasets

  • The standard benchmarks include ETT at 15-minute and hourly granularities, Electricity and Traffic hourly series, and 10-minute Weather data.Electricity covers 321 households over three years, Traffic 862 sensors over two years, and Weather 21 meteorological features.Source: A Description of All Datasets · p. 13, Table 5 · p. 13

B Hyperparameter Tuning

C.1 Libraries and Hardware

  • Experiments use PyTorch 2.4, Python 3.12.1, Optuna 3.6.1, and NVIDIA RTX 4090 and RTX 3090 GPUs.Source: §C.1 · p. 14

C.2 Training Procedure

  • Models use Adam with MSE loss, a default batch size of 32, and up to 100 epochs, with early stopping after 10 validation epochs without improvement.Source: §C.2 · p. 14

C.3 Granger Causality

  • The Granger analysis tests the first 1,000 timesteps, removes highly redundant channels, checks stationarity, and differences series where needed.The redundancy filter uses Pearson correlation greater than 0.95; tests use the statsmodels package.Source: §C.3 · p. 14

D Derivation of CI/CD Model Variants

  • CI and CD variants are derived by removing or enabling channel mixing or attention, with PatchTST’s CD version attending jointly across flattened patches and channels.Crossformer’s CI version removes its second attention stage; TimeMixer’s CI version removes channel mixing while retaining multi-resolution temporal mixing.Source: D Derivation of CI/CD Model Variants · p. 15

E Detailed Results

  • Detailed results report mean MSE over five random seeds for standard and ODE datasets.The full results are presented in Tables 8 and 9, respectively.Source: E Detailed Results · p. 15

F Detailed Granger Causality Analysis

The detailed analysis contrasts channel dependencies across standard benchmarks, chaotic ODE datasets, and Monash datasets. It concludes that ODE systems retain dependencies across lags, while most analyzed Monash datasets do not show strong lateral dependence.

Standard Benchmarks

  • Electricity and Traffic show moderate inter-channel correlation, concentrated at lag 30; neighboring-channel predictive power diminishes substantially by lag 192.The reported decline is consistent with CI models performing well at longer lookbacks when marginal information from other channels becomes negligible.Source: F Detailed Granger Causality Analysis · p. 16, Table 8 · p. 16

Chaotic ODE Datasets

  • Chaotic ODE datasets retain high F-scores across all tested lags, including lag 192 for Lorenz and Double Pendulum systems.The authors characterize these as persistent dependencies and say the results support CD models for physical or simulated dynamics.Source: F Detailed Granger Causality Analysis · p. 16, Table 9 · p. 17

Monash Forecasting Collection

  • Only six of 40 filtered Monash datasets have an inter-channel F-score above 10, and most scores decline sharply as lag increases.The authors note that many behave more like collections of independent univariate series than integrated multivariate systems.Source: F Detailed Granger Causality Analysis · p. 17

Summary

  • The authors conclude that standard benchmarks can support univariate treatment at larger lags, whereas ODE datasets call for explicit modeling of channel interactions.They suggest prioritizing genuinely integrated datasets, such as chaotic ODEs, in research on multivariate dependencies.Source: F Detailed Granger Causality Analysis · p. 17

G Full Lookback Window Analysis

Lookback-window preferences vary across forecasting horizons and model categories, with larger windows becoming more common at the longest horizon. These patterns support systematic tuning across datasets and horizons.

G Full Lookback Window Analysis

  • Figures 4–7 report the frequency of best-performing lookback windows by dataset and model category across four forecasting horizons.Table 10 reports standard deviations averaged over the four horizons and marks out-of-memory errors.Source: G Full Lookback Window Analysis · p. 17, Figure 4 · p. 18, Table 10 · p. 18
  • At H = 720, datasets and model categories shift toward larger lookback windows, while ETTh1 and ETTh2 favor smaller windows at horizons 192 and 336.The authors relate the H = 720 shift to the greater difficulty of forecasting farther ahead.Source: G Full Lookback Window Analysis · p. 18
  • CI models typically benefit more from larger lookback windows than CD models across horizons, and more models favor larger windows as the horizon reaches 720.The analysis compares window-frequency patterns by dataset and model category for horizons 96, 192, 336, and 720.Source: G Full Lookback Window Analysis · pp. 17–18

H ODE Datasets Sample Visualization

The ODE sample visualizations show complex multivariate dynamics, including changes across associated channels. The cell-cycle equations describe linked variables, and the RevIN ablation caption reports different outcomes on standard and chaotic ODE datasets.

H ODE Datasets Sample Visualization

I Cell Cycle ODE Discussion

  • The cell-cycle equations describe two oscillators based on cyclin concentrations; they couple C1 with M2 and C2 with M1, illustrating how one channel can change trends in others.The equations are attributed to.Source: I Cell Cycle ODE Discussion · p. 23
  • RevIN improves performance on ETTh2 and ETTm1 but can be detrimental on Hopfield and LorenzCoupled.Table 13 presents the ablation at forecasting horizon L = 96.Source: Table 13 · p. 24

J Ablation Study on RevIN

This section examines how RevIN’s effect varies by dataset type for Crossformer variants and PatchTST, and documents forecasting models’ dataset- and horizon-specific hyperparameter settings.

J Ablation Study on RevIN

  • The ablation added RevIN to Crossformer’s CI and CD versions and evaluated them on two standard and two chaotic-benchmark datasets, with PatchTST as a CI baseline.Crossformer was originally proposed without RevIN.Source: J Ablation Study on RevIN · p. 24
  • RevIN improves both Crossformer CI and CD variants on standard datasets, but Crossformer without RevIN outperforms normalized versions on chaotic ODE datasets.The authors suggest complex, non-stationary ODE patterns may not benefit from RevIN’s normalization-denormalization technique.Source: J Ablation Study on RevIN · p. 24
  • Prior literature reports that RevIN generally enhances standard-dataset performance, but the trend was not universal for PatchTST in these tests.Across different dataset types, RevIN’s effect was not consistent.Source: J Ablation Study on RevIN · p. 24
  • Hyperparameter tables cover Hopfield, Lorenz, LorenzCoupled, electricity, weather, and Traffic datasets.The supplied passages identify these tables as settings records, not RevIN ablation results.Source: Table 21 · p. 41, Table 22 · p. 43, Table 23 · p. 46
Loading 2502.09683v3…