PaperlayerView exact revision on arXiv

Source-linked AI summary

MixBench-TS: A Multivariate Time Series Forecasting Benchmark Where Channel Mixing Pays Off

Ibram Abdelmalak, Mischa Putzke, Jungmin Choi, Tom Hanika, Vijaya Krishna Yalavarthi, Lars Schmidt-Thieme

arXiv:2609.32656v1cs.AIcs.LG

TL;DR

MTSF dataset evaluations rarely characterize the lagged, nonlinear, and joint coupling that channel-mixing models assume and rely on. The paper tests four coupling measures on synthetic data with known planted couplings, profiles datasets with MIlag and CDg, and proposes MixBench-TS. Only MIlag and CDg recover every planted coupling in the six synthetic datasets with known planted couplings.

  • Problem

    MTSF dataset evaluations rarely characterize the lagged, nonlinear, and joint coupling that channel-mixing models assume and rely on.

    Source: §1 · pp. 1–2

  • Method

    The paper tests four coupling measures on synthetic data with known planted couplings, profiles datasets with MIlag and CDg, and proposes MixBench-TS.

    Source: §1 · p. 2

  • Results

    Only MIlag and CDg recover every planted coupling in the six synthetic datasets with known planted couplings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend MixBench-TS for evaluating new channel-dependent models and profiling new datasets with MIlag and CDg.

    Source: §1 · p. 2, §8 · p. 9

    CDg uses only TSMixer, MIlag is pairwise and captures joint coupling only indirectly, and the synthetic suite has four channels and fixed couplings.

    Source: §8 · p. 9

Abstract

from arXiv · show

Multivariate Time Series Forecasting (MTSF) models that mix information across channels assume that the past of one channel carries information about the future of another. Yet they are evaluated on a small fixed set of standard datasets whose cross-channel structure is rarely examined. We ask two questions: "How can we reliably measure lagged, non-linear, and joint coupling in MTSF datasets?" and "Do the standard datasets actually have such coupling?" To answer the first, we test four candidate measures on synthetic datasets with planted ground-truth coupling: Granger Causality (GC), Transfer Entropy (TE), lagged Mutual Information (MI), and the CD gain, a model-based measure we introduce that compares a channel-dependent (CD) model to its channel-independent (CI) variant. Only lagged MI and the CD gain recover every planted coupling. For the second question, the answer is a definite no, as the standard datasets have a median of only 23% lagged-coupled channel pairs and a median CD gain of -4.9%, compared to 78% and +4.6% on chaotic ODE systems. We therefore propose MixBench-TS, a benchmark of 10 real-world datasets with a median of 55.5% lagged-coupled pairs and a median CD gain of +1.7%. Across six state-of-the-art models tuned under one protocol, CI models win on 10/10 (MSE) and 8/10 (MAE) standard datasets, but on only 3/10 and 2/10 MixBench-TS datasets. We recommend using our benchmark for evaluating new CD models. Moreover, we propose profiling new datasets with lagged MI and the CD gain before using them to evaluate multivariate models. Code and data are available at https://anonymous.4open.science/r/mixbench-ts-B027.

1 INTRODUCTION

Channel-dependent forecasting relies on lagged cross-channel information, but prior dataset characterizations did not assess the coupling it requires. The paper tests coupling measures, profiles standard datasets, and introduces MixBench-TS to evaluate channel-mixing models on more strongly coupled data.

1 INTRODUCTION

  • Prior dataset studies characterize domain coverage and temporal properties, but not the lagged coupling channel-dependent models rely on.The closest prior effort uses Granger Causality, which cannot capture nonlinear lagged coupling, and considers only standard and ODE datasets.Source: §1 · p. 1
  • The introduction frames two questions: how to measure lagged, nonlinear, and joint coupling, and whether standard datasets contain it.The paper also evaluates six state-of-the-art models under a unified tuning protocol covering lookback windows.Source: §1 · pp. 1–2
  • Only MIlag and CDg recover every planted coupling among the four measures tested on synthetic datasets.GC misses nonlinear and joint coupling, while TE places contemporaneous coupling at a wrong positive lag.Source: §1 · p. 2, Table 1 · p. 2
  • On all 10 standard datasets, TSMixerCI beats full channel-mixing TSMixer (median CDg −4.9%), whereas channel mixing helps on every one of six chaotic-ODE datasets (median CDg +4.6%).Standard datasets have a median of 23% lagged-coupled pairs, versus 78% for the chaotic-ODE datasets.Source: §1 · p. 2
  • MixBench-TS adds 10 rarely used real-world datasets selected for positive CDg and substantially stronger lagged coupling than standard datasets.The datasets span environmental monitoring, energy, hydrology, marine sensing, and health.Source: §1 · p. 2
  • The paper introduces CDg, a model-based measure comparing a channel-dependent forecaster with its channel-independent variant.It is validated alongside three other measures on synthetic data with known planted coupling.Source: §1 · p. 2

2 RELATED WORK

The related work distinguishes channel-independent models from models that mix channels, and reviews benchmark and evaluation studies. The paper extends this literature by examining nonlinear and lagged coupling under a rigorous evaluation protocol.

2 RELATED WORK

  • Channel-independent models forecast each channel in isolation, while channel-dependent models mix information across channels.The reviewed channel-dependent approaches include attention, MLPs, linear, CNN, frequency-domain, graph-filtering, and channel-clustering methods.Source: §2 · p. 3
  • Prior benchmarks extend MTSF data across domains and temporal properties, while evaluation studies examine model performance, dataset choice, tuning, and horizons.The paper situates its work alongside studies of when cross-channel modeling pays off, including analyses of distribution shift, dataset heterogeneity, and linear coupling.Source: §2 · p. 3
  • The paper builds on prior evaluation practices while extending coupling analysis to nonlinear and lagged relationships.The closest cited work uses Granger Causality to study linear channel coupling in standard datasets.Source: §2 · p. 3

3 PROBLEM FORMULATION

The paper formulates each dataset as equally spaced multivariate observations and defines forecasting from a lookback window to a future horizon. Data are split chronologically, with samples formed using stride-one sliding windows.

3 PROBLEM FORMULATION

  • A dataset is a multivariate series with C channels observed at T equally spaced steps.The series is denoted Z = (z1, ..., zT) ∈ R^(T × C).Source: §3 · p. 3
  • A forecasting model maps the last L steps across C channels to the next H steps across those channels.The model is written fθ: R^(L × C) → R^(H × C).Source: §3 · p. 3
  • Datasets are split chronologically into training, validation, and test segments, and samples use stride-one sliding windows.Forecast targets do not cross split boundaries; standard datasets commonly use horizons of 96, 192, 336, or 720 steps.Source: §3 · p. 3

4 CHANNEL DEPENDENCY MEASUREMENT

The section measures cross-channel coupling with statistical pairwise measures and a model-based CD gain. MIlag compares lagged with contemporaneous coupling, while CDg tests whether channel mixing improves forecasting.

Granger Causality (GC)

  • GC estimates the degrees-of-freedom-corrected partial R2 gain from adding a source’s past to a target’s own-past regression.Pairs are jointly made stationary before restricted and multivariate OLS regressions; GC is the median over pairs and is near zero under the no-causality null.Source: §4 · p. 4

Transfer Entropy (TE)

  • Transfer Entropy measures directed nonlinear information from a source’s past to a target’s present, conditioned on the target’s recent past.It sweeps positive lags, selects the statistically significant lag with the largest effective value, maps the result to the GC R2 scale, and takes the median over pairs.Source: §4 · p. 4

Lagged Mutual Information (MIlag)

  • MIlag summarizes the share of channel pairs whose lagged mutual information exceeds their contemporaneous mutual information.It uses rank-transformed raw series and a KSG estimator; positive-lag MI is directional, while lag-0 MI is a symmetric common-driver measure.Source: §4 · p. 4

CD Gain (CDg)

  • CDg measures the relative test-MSE reduction from channel mixing, comparing TSMixer with TSMixerCI across forecast horizons and three seeds.A positive value indicates channel mixing can exploit cross-channel information; a non-positive value indicates no benefit or worse forecasting performance.Source: §4 · p. 5

5 EVALUATION OF CHANNEL DEPENDENCY MEASUREMENTS

On six synthetic datasets with planted coupling, only MIlag and CDg recover every planted coupling. The suite varies coupling type and provides a known-ground-truth test of the measures.

5.1 SYNTHETIC DATASETS CURATION

  • The planted cases include contemporaneous linear, lagged linear, lagged nonlinear, multiplicative-joint, and additive-mixed coupling.Figures 2 and 3 depict the planted coupling structures and source imprints at their specified lags.Source: §5.1 · p. 5, Figure 2 · p. 6, Figure 3 · p. 6
  • The six synthetic datasets cover no coupling, single-source pairwise coupling, and joint coupling, including a target that depends on the product of two sources.Each dataset has four channels and 20,000 timesteps; the independent dataset establishes the null values of the measures.Source: §5.1 · p. 5

5.2 EVALUATION OF MEASUREMENTS ON SYNTHETIC DATASETS

  • Only MIlag and CDg retrieve the planted coupling on all six datasets; GC misses nonlinear and joint cases, while TE mislocates the contemporaneous coupling at a positive lag.MIlag also evaluates lag 0, and CDg makes no assumption about functional form and evaluates all channels jointly.Source: §5.2 · pp. 5–6

6 CHANNEL DEPENDENCY MEASUREMENT OF STANDARD DATASETS

The standard datasets show weak lagged coupling compared with the chaotic ODE reference, and TSMixerCI outperforms TSMixer on all 10 standard datasets. The measures therefore indicate little exploitable cross-channel coupling in the standard benchmark suite.

6 CHANNEL DEPENDENCY MEASUREMENT OF STANDARD DATASETS

  • 23% of channel pairs are lagged-coupled at the median in standard datasets, versus 78% in the ODE reference; median CDg is −4.9% versus +4.6%.TSMixerCI outperforms TSMixer on all 10 standard datasets.Source: §6 · pp. 6–7, Table 3 · p. 7
  • Only Weather (52%) and Electricity (61%) exceed 50% lagged-coupled pairs among standard datasets; no ODE dataset falls below 63%.The median shares are 23% for standard datasets and 78% for ODE datasets.Source: §6 · p. 6
  • The comparison reports MIlag and CDg for standard datasets and the chaotic ODE benchmark, with CDg measuring test-MSE reduction of TSMixer over TSMixerCI.MIlag is the share of channel pairs coupled more strongly at a lag than at lag 0.Source: Table 3 · p. 7
  • TSMixerCI outperforms TSMixer on all 10 standard datasets, while the ODE datasets have a median CDg of +4.6%.The section concludes that discarding other channels does not hurt forecasting on the standard datasets and consistently improves it.Source: §6 · p. 7, Table 3 · p. 7

7 THE MIXBENCH-TS BENCHMARK

MixBench-TS comprises 10 real-world datasets selected for positive CDg and evaluates six CI and CD models under a shared tuning protocol. Channel-dependent models win more often on these datasets than on the standard suite, where CI models dominate.

7.1 DATASETS

  • MixBench-TS contains 10 real-world datasets screened for CDg > 0, spanning environmental monitoring, energy, hydrology, marine sensing, and health.Seven datasets with MIlag > 50% and CDg ≥+1.1% form MixBench-TS-Core.Source: §7.1 · p. 8

7.2 EXPERIMENTAL PROTOCOL

  • Six state-of-the-art CI and CD models were tuned on every dataset for 20 trials, with lookback windows tuned and test MSE and MAE averaged over horizons and three seeds.Models were selected by validation MSE after training with MSE loss.Source: §7.2 · p. 8

7.3 RESULTS

  • CI models win on only 3 of 10 MixBench-TS datasets by MSE and 2 of 10 by MAE, compared with 10 of 10 and 8 of 10 standard datasets.MixBench-TS has a median of 55.5% lagged-coupled pairs and median CDg of +1.7%, versus 23% and −4.9% on standard datasets.Source: §7.3 · pp. 8–9, Table 5 · p. 9
  • On ZafNoo, AirQuality, and HouseholdPower, channel mixing improves TSMixer by +0.6 to +1.8%, yet DLinear has the lowest MSE, suggesting architecture differences outweigh channel-mixing gains.Source: §7.3 · p. 9

8 CONCLUSIONS AND FUTURE WORK

The paper concludes that standard MTSF datasets lack exploitable cross-channel coupling and proposes MixBench-TS as a benchmark with stronger coupling. It recommends screening further datasets with MIlag and CDg, while noting limitations in both measures and the synthetic suite.

8 CONCLUSIONS AND FUTURE WORK

  • Standard datasets have a median of 23% lagged-coupled pairs, while MixBench-TS has 55.5%; CI models win on only 3 MixBench-TS datasets versus all 10 standard datasets.The conclusion recommends benchmarks that contain multivariate structure so channel-mixing benefits can be observed rather than assumed.Source: §8 · p. 9
  • CDg uses only TSMixer, MIlag is pairwise and captures joint coupling indirectly, and the synthetic suite has four channels with fixed couplings.MIlag also uses at most 40 channels on wide datasets; the synthetic suite omits regime changes and hidden common drivers.Source: §8 · p. 9
  • The authors propose extending CDg to further CD models, mapping coupling by channel and time window, and broadening MixBench-TS to additional dataset types.Named extensions include short-and-wide, spatio-temporal, and irregularly sampled datasets.Source: §8 · p. 9

AI USE STATEMENT

The authors used generative AI to polish writing, draft portions of sections, and search for related work; they reviewed and validated the final content.

AI USE STATEMENT

  • Generative AI supported writing, section drafting, and related-work searches, with retrieved references verified and the final content reviewed and validated by the authors.The authors state that they take full responsibility for the final version.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The appendix documents data sources, forecasting settings, and reproducibility details for the benchmark and its dataset suites.

REPRODUCIBILITY STATEMENT

  • The repository includes experiment code, a profiling toolkit, a synthetic-data generator, and instructions for obtaining datasets and running experiments.The paper also specifies where dataset splits, horizons, lookbacks, coupling measures, generation parameters, model settings, and search ranges are documented.Source: REPRODUCIBILITY STATEMENT · p. 10

A DATASETS - MORE DETAILS

  • Datasets use chronological splits, except Wearable, which is split by subject; channel standardization uses training-set statistics and metrics are reported on that scale.Forecast targets remain inside their assigned split, though validation and test lookbacks may reach into the preceding split.Source: A DATASETS - MORE DETAILS · p. 14
  • The dataset details cover standard forecasting series, six chaotic ODE systems, spatio-temporal traffic data, and real-world coupled datasets across several domains.The coupled examples include air quality, tree sap flow, river discharge, coastal water quality, ocean currents, wearable sensing, and household electricity.Source: A DATASETS - MORE DETAILS · pp. 14–15

B SPATIO-TEMPORAL DATASETS

This appendix reports the separate traffic-dataset results and details the data, coupling measures, and tuning procedures used in the benchmark.

B SPATIO-TEMPORAL DATASETS

  • 53.5% of pairs are lagged-coupled across METR-LA and PEMS-BAY, versus 55.5% for MixBench-TS and 23% for standard datasets; CDg is mixed across the two traffic datasets.Channel mixing helps on PEMS-BAY (+6.4%) but not METR-LA (−1.5%); a CD model wins MSE on PEMS-BAY, while a CI model wins on METR-LA.Source: Table 7 · p. 16, B SPATIO-TEMPORAL DATASETS · p. 16

C PROBLEM FORMULATION - DETAILED

  • The problem formulation builds stride-1 supervised windows from lookback and forecast intervals, with forecast targets kept within their assigned data split.Lookbacks may reach into the preceding split, but targets do not cross split boundaries.Source: C PROBLEM FORMULATION - DETAILED · p. 16

D STATISTICAL PROFILING - IMPLEMENTATION DETAILS

  • Pairwise profiling analyzes ordered channel pairs and randomly subsamples wide datasets to 40 channels because the computation scales with pair count.The subset is drawn uniformly without replacement using a fixed seed.Source: D STATISTICAL PROFILING -… · p. 17

D.1 GRANGER CAUSALITY

  • Granger Causality uses differenced series and a VAR order set to the smallest candidate lookback; its partial R2 summarizes linear directed predictability across channel pairs.Seasonal and regular differencing address nonstationarity, and the reported dataset value is the median across directed pairs.Source: §D.1 · p. 17

D.2 TRANSFER ENTROPY (TE)

  • Transfer Entropy estimates conditional information between a lagged source and target using a KSG estimator, then tests significance against locally permuted surrogate data.The lag sweep is bounded by the smallest candidate lookback, and the reported effective TE is zero when no lag survives false-discovery-rate correction.Source: §D.2 · pp. 17–18

D.3 MUTUAL INFORMATION (MI)

  • Lagged MI uses rank-transformed raw series and compares contemporaneous MI with effective MI at positive lags on a fixed geometric lag grid.The fraction of pairs whose peak-lag MI exceeds lag-0 MI is used to measure lagged coupling.Source: §D.3 · pp. 18–19

E HYPERPARAMETER RANGES

  • Each model, dataset, and horizon is tuned separately with 20 Optuna trials selected by validation MSE under a shared training protocol.The lookback is tuned jointly with other hyperparameters; TSMixerCI shares TSMixer’s search space with channel mixing disabled.Source: E HYPERPARAMETER RANGES · p. 20

F STATISTICAL PROFILING - MORE RESULTS

The section finds that GC and TE do not reliably distinguish coupling suites or track channel-mixing payoff, unlike lagged MI.

F STATISTICAL PROFILING - MORE RESULTS

  • Traffic has the highest real-world GC (0.067) and TE (0.053) but a CDg of −9.2%, while Bavaria-Donau has the largest coupled-dataset CDg (+11.1%) despite low GC and TE.Blinking-Rotlet likewise combines the largest ODE CDg (+16.0%) with GC and TE values close to zero.Source: F STATISTICAL PROFILING - MORE RESULTS · p. 21
  • Across 26 datasets, GC and TE have Spearman correlations with CDg of only 0.04 and 0.08, respectively, versus 0.78 for MIlag.Figure 4 plots median GC and TE against TSMixer CDg; compare their relationships with MIlag in Figure 1.Source: F STATISTICAL PROFILING - MORE RESULTS · p. 21, Figure 4 · p. 22

G PARAMETER SENSITIVITY

The appendix evaluates channel-subsampling sensitivity, catalogs per-horizon results, and describes the synthetic coupling suite.

G PARAMETER SENSITIVITY

  • Doubling the sampled subset from 40 to 80 channels changes medians by at most 9%, while halving it to 20 produces less stable PEMS-BAY GC estimates.At 20 channels, the PEMS-BAY GC median deviates by +20% at seed 2021 and −17% at seed 7; the table compares subset sizes and sampling seeds.Source: Table 12 · p. 22, G PARAMETER SENSITIVITY · pp. 22–23
  • Despite subset and seed deviations, the cross-dataset ordering remains stable; Electricity stays the highest-MIlag standard dataset and both wide datasets remain above 50% lagged-coupled pairs.The reported MIlag ranges are 61–68% for Electricity and 64–74% for PEMS-BAY.Source: G PARAMETER SENSITIVITY · p. 23

I PER-HORIZON RESULTS

J SYNTHETIC DATA GENERATION DESCRIPTION

  • The synthetic suite uses four-channel series with independent source bases and plants no coupling, pairwise coupling, or joint coupling into target c0.Planted forms include contemporaneous linear, lagged linear, lagged nonlinear, and multiplicative joint coupling; contributions are standardized before weighting so edge weights set variance shares.Source: J SYNTHETIC DATA GENERATION DESCRIPTION · pp. 23, 25

Related papers

Loading 2609.32656v1…