PaperlayerView exact revision on arXiv

Source-linked AI summary

TSMixer: An All-MLP Architecture for Time Series Forecasting

Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O. Arik, Tomas Pfister

arXiv:2303.06053v5cs.LGcs.AI

TL;DR

The paper investigates whether linear-model ideas can support multivariate forecasting when common benchmarks often favor univariate models. It proposes TSMixer, which mixes information across time and features with stacked MLPs, and reports parity with univariate models on common benchmarks plus superior performance on the challenging M5 retail benchmark.

  • Problem

    The paper examines whether cross-variate information benefits forecasting and whether multivariate models can match univariate models when that information is not beneficial.

  • Method

    TSMixer stacks MLPs that alternately mix across time and feature dimensions, retaining temporal linear-model capacity while using cross-variate and auxiliary information.

  • Results

    TSMixer is on par with univariate models on common long-term benchmarks and achieves state-of-the-art performance on a large-scale industrial application where cross-variate information is crucial.

  • Takeaways & Limitations

    The results indicate that multivariate modeling can remain competitive on biased common benchmarks and be especially valuable in applications containing crucial cross-variate information.

  • Takeaways & Limitations

    The analysis focuses on univariate temporal linear models under smoothness or periodicity assumptions and leaves more complex, non-smooth cases for future work.

Abstract

from arXiv · show

Real-world time-series datasets are often multivariate with complex dynamics. To capture this complexity, high capacity architectures like recurrent- or attention-based sequential deep learning models have become popular. However, recent work demonstrates that simple univariate linear models can outperform such deep learning models on several commonly used academic benchmarks. Extending them, in this paper, we investigate the capabilities of linear models for time-series forecasting and present Time-Series Mixer (TSMixer), a novel architecture designed by stacking multi-layer perceptrons (MLPs). TSMixer is based on mixing operations along both the time and feature dimensions to extract information efficiently. On popular academic benchmarks, the simple-to-implement TSMixer is comparable to specialized state-of-the-art models that leverage the inductive biases of specific benchmarks. On the challenging and large scale M5 benchmark, a real-world retail dataset, TSMixer demonstrates superior performance compared to the state-of-the-art alternatives. Our results underline the importance of efficiently utilizing cross-variate and auxiliary information for improving the performance of time series forecasting. We present various analyses to shed light into the capabilities of TSMixer. The design paradigms utilized in TSMixer are expected to open new horizons for deep learning-based time series forecasting. The implementation is available at https://github.com/google-research/google-research/tree/master/tsmixer

1 Introduction

The paper asks whether cross-variate information benefits forecasting and develops TSMixer to combine temporal and cross-variate modeling while retaining linear-model capacity. Experiments report parity with strong univariate models on common benchmarks and advantages on M5, where cross-variate information is important.

  • Motivation: Common benchmarks can favor univariate models because multivariate models may overfit when covariates are weakly correlated with the target.This motivates testing whether cross-variate information consistently helps and whether multivariate models can match univariate performance.
  • Motivation: The analysis finds temporal linear models effective for learning temporal patterns under common assumptions, motivating capacity increases from linear models.The authors first analyze temporal linear models before adding nonlinearities and cross-variate layers.
  • Method: TSMixer alternates time-mixing and feature-mixing MLPs to capture temporal patterns and cross-variate information efficiently.Its residual design retains temporal linear-model capacity while exploiting cross-variate information.
  • Results: On common long-term forecasting datasets, TSMixer matches state-of-the-art univariate models and significantly outperforms other multivariate models.The ablation study also finds cross-variate information less beneficial on these datasets.
  • Results: On the challenging M5 retail benchmark, cross-variate information improves performance, and TSMixer leverages it effectively.The paper also extends TSMixer to static and future time-varying auxiliary features.
  • Implications: The paper highlights a potential risk of evaluating multivariate models only on common long-term forecasting benchmarks.Its contribution frames dataset characteristics as important when assessing the value of cross-variate information.

2 Related Work

Related work covers forecasting models organized by whether they use univariate data, multivariate data, or auxiliary information. TSMixer is positioned as a multivariate architecture that is also extended to auxiliary features, while addressing benchmark-dependent conclusions about cross-variate modeling.

  • Forecasting settings: Forecasting work is grouped into univariate forecasting, multivariate forecasting, and multivariate forecasting with auxiliary information.The categories distinguish which historical, cross-variate, static, or future information models use.
  • Multivariate forecasting: Transformer-based multivariate models are widely used because they model long and complex sequential data and relationships between covariates.Several variants target efficiency and accuracy improvements.
  • Multivariate forecasting: Some recent models treat multivariate data as separate univariate sequences and report strong accuracy on common long-term benchmarks.The paper argues this pattern mainly reflects dataset bias and may not generalize to some real-world applications.
  • Auxiliary information: Auxiliary-information models incorporate static features and future time-varying features using state-space, recurrent, and attention-based architectures.Examples include location information and upcoming promotions.
  • Mixer architectures: TSMixer resembles MLP Mixer architectures previously applied to vision, text, speech, network traffic, and point clouds.The paper adapts the mixer design paradigm to time-series forecasting.

3 Linear Models for Time Series Forecasting

The paper analyzes why temporal linear models can capture important time dependencies, especially under periodicity or smoothness assumptions. It then motivates extending these models to address cross-variate information while noting limits in highly volatile, non-smooth settings.

  • Linear models have time-step-dependent weights that can learn temporal dependencies in univariate series, unlike recurrent or attention-based architectures.
  • For periodic sequences, a linear model can perfectly predict future values when the lookback window exceeds the period.
  • A lookback window L ≥ P + 1 yields an error bound for signals combining a periodic component with a Lipschitz-smooth component.
  • For smooth non-periodic patterns, prediction error remains bounded with an adequate lookback window.
  • The univariate analysis is limited because real-world series may be volatile, non-periodic, and non-smooth; analysis beyond Lipschitz cases is left for future work.

4 TSMixer Architecture

TSMixer alternates MLP-based time-mixing and feature-mixing operations to model temporal patterns and cross-variate information efficiently. Its residual design, auxiliary-feature extension, and simple implementation support competitive benchmark performance.

  • TSMixer alternates MLPs across time and feature dimensions, combining temporal-pattern modeling with cross-variate information.
  • The alternating design keeps parameter growth at O(L + C) rather than O(LC), enabling long lookback windows with limited model size.
  • Time-mixing MLPs model temporal patterns, while feature-mixing MLPs shared across time steps leverage covariate information.
  • Temporal projection maps the input length L to the target forecast length T while learning temporal patterns.
  • Residual connections support deeper architectures and allow the model to ignore unnecessary time- and feature-mixing operations.
  • 2D normalization maintains scale across features and time steps, distinguishing TSMixer from MLP-Mixer's normalization approach.
  • TSMixer is relatively simple to implement and remains competitive with state-of-the-art models on representative benchmarks.
  • For heterogeneous inputs, TSMixer aligns features with different shapes before concatenation and collective time- and feature-mixing.

5 Experiments

Experiments evaluate TSMixer across long-term forecasting benchmarks and the large-scale M5 retail dataset, testing temporal, cross-variate, auxiliary-feature, lookback, and computational behavior. TSMixer remains competitive with univariate models on common benchmarks while leveraging cross-variate and auxiliary information effectively on M5.

  • Datasets and setup: The evaluation covers seven multivariate long-term forecasting benchmarks and M5, a 30,490-series Walmart retail dataset with static and time-varying features.The long-term datasets span weather, electricity, and traffic; M5 is designed to test cross-variate and auxiliary information.
  • Long-term forecasting: TMix-Only benefits from stacking and performs comparably to PatchTST, indicating that time-mixing can match more complex attention mechanisms without cross-variate information.TMix-Only is the univariate time-mixing variant of TSMixer.
  • Long-term forecasting: TSMixer matches TMix-Only and PatchTST on long-term benchmarks while significantly outperforming multivariate baselines, which perform worse than univariate models.These results suggest cross-variate information is less useful in the commonly used datasets evaluated here.
  • Lookback window length: Increasing the lookback window from L = 96 to L = 336 significantly improves linear-model performance, with performance appearing to approach a plateau afterward.The same lookback-window experiments are conducted for TSMixer and linear models.
  • Large-scale demand forecasting: On M5 without auxiliary information, multivariate models substantially outperform univariate models, while PatchTST performs significantly worse than multivariate TSMixer and FEDformer.TSMixer also substantially outperforms FEDformer in this setting.
  • Large-scale demand forecasting: With auxiliary features, TSMixer outperforms TFT and DeepAR by a significant margin, demonstrating effective modeling of cross-variate information and auxiliary features.TSMixer-Ext aligns different feature types before concatenating them and applying mixing layers jointly.
  • Computational cost: TSMixer has a smaller model size than RNN- and Transformer-based models and achieves much faster inference while maintaining similar training time to multivariate models.Its inference speed is reported as almost the same as simple linear models.

6 Conclusions

The paper presents TSMixer as a simple MLP-based alternative to RNNs and attention mechanisms for time-series forecasting. Across diverse tasks, it reports effectiveness on both multivariate long-term forecasting and large-scale retail demand forecasting, while identifying interpretability and scalability as future directions.

  • Conclusion: TSMixer uses stacked MLPs instead of commonly used RNNs and attention mechanisms to pursue superior generalization with a simple architecture.The architecture mixes information across time and feature dimensions.
  • Conclusion: Across real-world forecasting tasks, TSMixer is effective on multivariate long-term benchmarks and large-scale retail demand forecasting.The conclusion covers both benchmark and retail-demand settings.
  • Conclusion: TSMixer is the only multivariate model reported to achieve performance similar to univariate models on the long-term forecasting benchmarks.The paper identifies further interpretability and scalability studies as potential future work.

A Proof of Theorem 3.1

Theorem 3.1 considers a periodic signal plus a Lipschitz-smooth component and establishes a linear forecasting model whose error is bounded when the lookback window covers at least one period plus one step. The proof constructs the predictor through periodic indexing and bounds the residual smooth-component differences.

  • Theorem statement: Under x(t) = g(t) + f(t), with g periodic of period P and f Lipschitz with constant K, a linear model exists when L ≥ P + 1.The theorem assumes the decomposition into periodic and smooth components.
  • Theorem statement: The resulting forecast satisfies |y_i − ŷ_i| ≤ K(i + min(i, P)) for every i = 1, . . . , T.This is the theorem’s stated mean absolute error bound.
  • Proof construction: The proof expresses the future value using periodic repetition of g and a linear combination of historical values indexed by i mod P.The construction uses the historical window x ∈ R^L and periodic indices.
  • Error decomposition: The prediction error is decomposed into differences between smooth-component values at future and periodic-reference positions.The decomposition is written first in terms of x and then in terms of f and g.
  • Error bound: Applying the triangle inequality and the Lipschitz condition bounds each smooth-component difference by K times its time-index separation.The proof then combines the two bounds to obtain the theorem’s error guarantee.

B Implementation Details

The implementation uses global, local, and model-level normalization, with choices differing between long-term benchmarks and M5. TSMixer-Ext additionally uses modified mixing layers for M5, including post-normalization and layer normalization.

  • Normalization: Global normalization standardizes each variate independently for long-term forecasting experiments, but is not applied to M5 because it has one target series, sales.The standardized data is used for training and evaluation on long-term datasets.
  • Normalization: Local normalization uses reversible instance normalization for long-term datasets and mean scaling with output rescaling for M5 sales.M5 products are independently scaled by their means.
  • Normalization: Model-level normalization uses batch normalization on long-term datasets and layer normalization on M5.The choice follows the differing experimental settings.
  • M5 modifications: For M5, TSMixer-Ext uses post-normalization instead of pre-normalization and layer normalization instead of batch normalization.The stated reasons are avoiding NaNs at large input scales and obtaining stable statistics without much larger batches.
  • Architecture: TSMixer-Ext includes mixing layers designed to handle auxiliary information.The figure identifies the mixing layers in TSMixer-Ext.

B.3 Formulae of TSMixer architecture

TSMixer combines temporal projection, time mixing, feature mixing, and conditional feature mixing into mixer layers for forecasting with historical and auxiliary information.

  • B.3.1 TSMixer Components: Dimension-changing layers use A → B subscripts to indicate mappings between input and output sizes.This notation distinguishes transformations whose output dimensionality differs from their input dimensionality.
  • B.3.1 TSMixer Components: TSMixer comprises temporal projection, time mixing, feature mixing, conditional feature mixing, and their mixer-layer compositions.Mixer layers compose Time Mixing with Feature Mixing, while Conditional Mixer Layers replace Feature Mixing with Conditional Feature Mixing.
  • B.3.1 TSMixer Components: Temporal Projection applies a shared linear layer to each column, mapping the lookback length L to the prediction length T.The projection operates on each time-series column using weights W1 ∈ R^(L×T) and bias b1 ∈ R^T.
  • B.3.1 TSMixer Components: Time Mixing transforms temporal features across all columns using deep-learning layers, normalization, dropout, activation, and residual connections.Normalization is applied across the entire time-feature matrix rather than row-by-row, and the block captures temporal dependencies.
  • B.3.1 TSMixer Components: Feature Mixing is a two-layer residual MLP applied independently to each input row and shared across rows to model feature transformations.When the feature size changes from C to H, TSMixer linearly transforms the residual term using the corresponding dimension-changing weights.
  • B.3.1 TSMixer Components: Conditional Feature Mixing incorporates static features S to transform hidden features conditionally for each sequence.Static features are expanded across the time dimension, concatenated with the sequence representation, and processed through conditional feature mixing.
  • B.3.2 Basic TSMixer for Multivariate Time Series Forecasting: The basic multivariate TSMixer applies mixer blocks to historical targets, forms a latent representation, and projects it to the prediction length T.Its feature-mixing hidden layers can use size H, and pre-normalization is used in residual blocks to keep input scale.
  • B.3.3 Extended TSMixer for Time Series Forecasting with Auxiliary Information: The extended architecture combines past targets and features, known future features, and static features through conditional mixer blocks.The final representation is linearly projected either to real-valued predictions or probability-distribution parameters such as a negative binomial distribution.

C.1 Long-term time series forecasting datasets

The long-term forecasting experiments use publicly available ETTm2, Weather, Electricity, and Traffic data under settings aligned with recent work.

  • C.1 Long-term time series forecasting datasets: Experiments use publicly available ETTm2, Weather, Electricity, and Traffic datasets pre-processed by Wu et al. (2021).The study follows experimental settings used in recent papers.

C.2 M5 dataset

The M5 experiments use Kaggle data, established feature-engineering resources, and controlled training and hyperparameter procedures.

  • C.2 M5 dataset: The M5 dataset is obtained from Kaggle, with competition details referenced through its participants guide.Basic feature engineering follows an example GluonTS script and a third-place competition solution.
  • C.2 M5 dataset: The M5 experiments use static and time-varying features listed in Table 7.The feature categories are documented as inputs used in the experiments.
  • C.2 M5 dataset: Models are trained for up to 300 epochs with early stopping after 30 epochs without validation-loss improvement.The implementation uses GluonTS models and custom implementations or modifications for the evaluated architectures.
  • C.2 M5 dataset: Lookback-window experiments evaluate L = {96, 336, 512, 720} with prediction lengths T = {96, 192, 336, 720}.These experiments are conducted on ETTm2, Weather, Electricity, and Traffic, with results shown in Fig. 7.

F Alternatives to MLPs

The alternatives study compares full MLPs and CNNs with TSMixer, emphasizing how alternative time-feature mixing affects computation, window size, and generalization.

  • F Alternatives to MLPs: Full MLPs vectorize inputs and outputs, while CNN alternatives use a one-dimensional convolution followed by a linear layer.The evaluation is conducted on the ETTm2 and Weather datasets.
  • F Alternatives to MLPs: Full MLPs have the highest computation cost and perform worse than both TSMixer and CNNs in the reported evaluation.CNN performance is similar to TSMixer on Weather but significantly worse on ETTm2.
  • F Alternatives to MLPs: TSMixer alternates time mixing and feature mixing, whereas full MLPs and CNNs mix both dimensions simultaneously in each linear operation.The alternative design supports large lookback windows while maintaining a reasonable parameter count.
  • F Alternatives to MLPs: Conventional MLP and CNN parameter counts grow faster than TSMixer’s as the lookback window L increases, potentially increasing overfitting risk.The comparison is framed as a parameter-growth and generalization difference between the architectures.
  • F Alternatives to MLPs: Figure 7 reports the effects of lookback-window size on TSMixer.The figure is identified as the visualization for this sensitivity analysis.
Loading 2303.06053v5…