MTS-Mixers: Multivariate Time Series Forecasting via Factorized Temporal and Channel Mixing

1. Introduction

• The introduction identifies unclear attention contributions, reliance on positional or date-specific encodings, and additional operations that can make sparse-attention methods slow in practice.Source: §1 · pp. 1–2
• On ETTh1, Fourier replacement maintained Transformer forecasting performance, removing cross-attention improved it, and separate temporal and channel modeling brought further improvement.The figure compares modifications and Fourier-Net variants at a 96-96 horizon setting.Source: §1 · p. 2, Figure 2 · p. 2
• Time-series data can be low-rank, with redundant temporal information across sampling rates and consistent information across channels.Figure 3 illustrates temporal redundancy with Exchange-rate series and channel redundancy with three consumers' electricity consumption.Source: §1 · p. 2, Figure 3 · p. 2
• MTS-Mixers factorize temporal and channel mixing to capture their dependencies, leverage low-rank structure, and improve prediction accuracy and efficiency.Source: §1 · p. 2
• MTS-Mixers achieves state-of-the-art forecasting on several public real-world datasets, reducing MSE by 15.4% and MAE by 12.9% on average.Source: §1 · p. 3

2. Related Work

• Time-series forecasting work addressed Transformer costs with sparse attention and non-autoregressive decoding, and enhanced temporal modeling with trend-seasonality decomposition and frequency transformations.Vanilla Transformer applications to long-sequence forecasting had failed because self-attention scales quadratically with input length.Source: §2 · p. 3
• Synthesizer, MLP-Mixer, FNet, and Poolformer replaced attention with other aggregation operations for competitive performance in machine translation or computer vision.Source: §2 · p. 3
• Unlike those works, this paper examines what attention-like modules learn in time-series forecasting and proposes factorized temporal and channel mixing for low-rank data.Source: §2 · p. 3

3.1. Problem definition

• Forecasting maps a historical series of n steps and c channels to the next m steps across those same c channels.The historical and forecast sequences are consecutive.Source: §3.1 · p. 3

3.2. Rethinking the mechanism of attention in Transformer-based forecasting models

• Transformer forecasting uses encoder-side token-level temporal similarity, channel-wise feature learning, and a decoder query to produce forecasts from the input history.A projection layer maps the decoder output to the final forecast; R2 describes the relationship between historical inputs and predicted outputs.Source: §3.2 · p. 3
• Removing self- or cross-attention can maintain or improve forecasting, while separately modeling temporal and channel dependencies further enhances results.The section interprets attention-like modules as capturing temporal dependencies across channels and cross-attention as bridging input and output sequences.Source: §3.2 · p. 3

4. MTS-Mixers

• MTS-Mixers separately captures temporal and channel dependencies to learn a map from the historical input to the forecast output.The input embedding module is optional.Source: §4 · p. 3
• The framework’s temporal and channel modules can use attention, random matrices, or factorized MLPs to capture dependencies.The architecture also includes a linear projection that outputs the forecasting results.Source: Figure 4 · p. 4, §4 · p. 4
• The attention-based variant uses self-attention for temporal dependency and a feed-forward network for channel interaction, then directly projects learned features to the prediction sequence.It removes the vanilla Transformer decoder.Source: §4 · p. 4
• The random-matrix variant multiplies temporal and channel dependency matrices with the input and a projection, whose initialization is controllable.The matrices encode temporal dependency, channel dependency, and projection, respectively.Source: §4 · p. 4
• Temporal mixing downsamples a series into interleaved subsequences, processes each with a temporal feature extractor, and merges them in original order.The described extractors include MLP or attention.Source: §4 · p. 4
• Channel mixing reduces channel-dimension noise through factorization, with U and V representing factorized channel interaction and m<c.Truncated SVD and NMF are examples of methods for obtaining the channel representation.Source: §4 · p. 5
• For baseline comparisons, the paper uses MLP-based MTS-Mixers with factorized temporal and channel mixing, and allows variants to be combined additively when needed.The supplied table caption specifies forecasting results across datasets and prediction horizons but does not report result values.Source: §4 · p. 5, Table 1 · p. 5

5.1. Experimental setup

• Experiments cover economics, energy, traffic, weather, and infectious-disease forecasting, comparing MTS-Mixers with Transformer-based and non-Transformer baselines.The ETT data use a 6:2:2 train-validation-test split, while other datasets use 7:1:2.Source: §5.1 · pp. 5–6

5.2. Main results

• Under the stated protocol, MTS-Mixers reports averaged MSE reductions versus previous state-of-the-art results of 18.1% on ECL, 10.8% on Traffic, 17.0% on PeMS04, 7.2% on Weather, and 39.0% on ILI, with comparable Exchange results.Source: §5.2 · p. 6

5.3. Ablation studies

• Factorized MLP performs best among the tested MTS-Mixers variants, while factorized attention outperforms prior Transformer-based models and factorized random matrices outperform attention.The tested variants include MLP, attention, and random-matrix interaction capture; all gain from temporal and channel factorization.Source: §5.3 · p. 7
• Adding channel factorization consistently improves FEDformer and SCINet, which use frequency-domain feature selection and recursive downsampling, respectively.The experiments apply channel factorization to these methods to address channel redundancy.Source: §5.3 · p. 7
• Temporal factorization improves MLP- and attention-based MTS-Mixers, although the best subsequence count varies by dataset; larger counts particularly benefit attention.The tested counts are s ∈ {1, 2, 3, 4, 6, 8, 12}.Source: §5.3 · p. 7, Figure 5 · p. 7
• Sharing temporal-extractor parameters across downsampled subsequences substantially reduces parameter quantity, with a trade-off in forecasting performance.The comparison is reported for the ETTm1 96-96 setting.Source: §5.3 · pp. 7–8, Table 4 · p. 8
• Four channel-denoising strategies improve prediction accuracy, supporting channel factorization; the tested settings include retaining 10% of the maximum singular value and dropping 10% of channels.The truncated-SVD and channel-drop settings are stated explicitly.Source: §5.3 · p. 8, Table 5 · p. 8

5.4. Model analysis

• Vanilla Transformer attention heads often capture little useful information, whereas MTS-Mixers variants learn similar periodic mapping patterns; MTS-Mixers also runs more efficiently in training and inference.The pattern analysis uses one encoder layer per model on ECL; runtime is measured over three runs of one epoch.Source: §5.4 · p. 8, Figure 6 · p. 8, Table 3 · p. 8

• Experiments indicate attention is unnecessary for capturing temporal dependencies, while redundancy in time-series data affects forecasting performance.The conclusion also identifies the input-to-output sequence mapping as a possible key revealed by model-pattern analysis.Source: §6 · pp. 8–9
• Temporal and channel factorization leverage time-series low-rank structure, and MTS-Mixers achieves state-of-the-art results with higher efficiency on several real-world datasets.Source: §6 · p. 9

• MTS-Mixers achieves averaged MSE reductions of 1.8% on ETTh1, 4.7% on ETTm1, 3.7% on ETTh2, and 8.2% on ETTm2 versus the second-best results.The benchmarks include hourly ETTh1 and ETTh2 and minutely ETTm1 and ETTm2, with prediction lengths 96, 192, 336, and 720.Source: A. Full Benchmark on the ETT Datasets · p. 11, Table 6 · p. 11

B. Univariate Time Series Forecasting

• MTS-Mixers yields averaged MSE reductions of 1.3% on Exchange and 1.6% on ETTm2 in univariate benchmarks, with channel factorization removed.Source: Table 7 · p. 11, B. Univariate Time Series Forecasting · p. 12

C.1. Reproduction Details for MTS-Mixers

• Temporal factorization equidistantly downsamples the original series into s interleaved subsequences, with s selected by grid search from {1, 2, 3, 4, 6, 8, 12}.The appendix illustrates the operation with two subsequences.Source: §C.1 · p. 12, Figure 7 · p. 12
• Channel factorization uses a channel MLP with a low-dimensional factorization X_C = UV, where U ∈ R^{n×m}, V ∈ R^{m×c}, and m < c.The appendix grid-searches m ∈ {0, 16, 64}; m = 0 means no interaction among channels.Source: §C.1 · p. 12

C.2. Details on Benchmark Tasks and Baselines

• Ten time-series datasets follow the preprocessing protocol in Zhou et al. (2021), with training random seed 1024.The passage also points to dataset-specific learning rates and batch sizes in Table 8.Source: §C.2 · p. 12, Table 8 · p. 12
• FEDformer, DLinear, SCINet, Autoformer, and Informer results are based on the authors’ reproductions.Their hyperparameters follow the suggestions in each original paper.Source: §C.2 · p. 12
• Table 8 lists the initial learning rate and batch size used for each dataset.Source: Table 8 · p. 12, §C.2 · p. 12

D. MTS-Mixers Weights Visualization on Different Datasets

• The visual analysis compares attention, random, and identity mixing across datasets, with Figure 8 showing one layer of the stacked block.Source: Figure 8 · p. 13, D. MTS-Mixers Weights Visualization on… · p. 13

E. From vanilla Transformer to Attention-based MTS-Mixers

• The attention-based MTS-Mixer removes the Transformer decoder, replaces tokenization with direct temporal self-attention, decouples temporal and channel interaction, and applies temporal factorization.The authors report that these modifications are effective on ECL under the 96-96 setting.Source: E. From vanilla Transformer to… · p. 13, Table 9 · p. 14

F. The Impact of Positional Encoding

• Attention-based MTS-Mixer outperforms other Transformer-based methods, but remains behind matrix-based and MLP-based MTS-Mixers.The authors argue that positional encoding distorts temporal information; adding ordering information degrades matrix-based and MLP-based variants, while attention-based MTS-Mixer requires it.Source: F. The Impact of Positional Encoding · p. 13, Figure 9 · p. 14

G. Forecasting Showcases

• The supplied showcase labels identify ground truth and prediction and include ticks from 0 to 400.Source: G. Forecasting Showcases · p. 14
• The showcases include forecasts from 96 input steps to 96 or 336 steps on both ECL and PeMS04.Source: G. Forecasting Showcases · p. 14