PaperlayerView exact revision on arXiv

Source-linked AI summary

TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting

Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, Jayant Kalagnanam

arXiv:2306.09364v4cs.LGcs.AI

TL;DR

Long-term multivariate forecasting needs models that capture temporal and channel structure without the high memory and computing costs of Transformers. TSMixer uses a patched MLP-Mixer backbone with online reconciliation heads, hybrid channel modeling, and gating, and reports stronger benchmark performance with lower resource usage.

  • Problem

    Transformer-based long-term forecasting offers long-sequence modeling but has high memory and computing requirements, while existing channel strategies do not explicitly and reliably model channel interactions.

  • Method

    TSMixer is a patching-based architecture using an MLP-only backbone with online reconciliation heads, hybrid channel modeling, and gated attention for multivariate forecasting and representation learning.

  • Results

    TSMixer outperforms DLinear by 8%, surpasses PatchTST by 1-2%, and reduces PatchTST training time and memory usage by 2-3X.

  • Takeaways & Limitations

    TSMixer's modular design supports supervised and masked self-supervised learning and is presented as a potential building block for time-series foundation models.

  • Takeaways & Limitations

    The paper defines its reconciliation mechanism around patch aggregation and cross-channel correlation, performed online during training rather than as standard hierarchical forecasting reconciliation.

Abstract

from arXiv · show

Transformers have gained popularity in time series forecasting for their ability to capture long-sequence interactions. However, their high memory and computing requirements pose a critical bottleneck for long-term forecasting. To address this, we propose TSMixer, a lightweight neural architecture exclusively composed of multi-layer perceptron (MLP) modules for multivariate forecasting and representation learning on patched time series. Inspired by MLP-Mixer's success in computer vision, we adapt it for time series, addressing challenges and introducing validated components for enhanced accuracy. This includes a novel design paradigm of attaching online reconciliation heads to the MLP-Mixer backbone, for explicitly modeling the time-series properties such as hierarchy and channel-correlations. We also propose a novel Hybrid channel modeling and infusion of a simple gating approach to effectively handle noisy channel interactions and generalization across diverse datasets. By incorporating these lightweight components, we significantly enhance the learning capability of simple MLP structures, outperforming complex Transformer models with minimal computing usage. Moreover, TSMixer's modular design enables compatibility with both supervised and masked self-supervised learning methods, making it a promising building block for time-series Foundation Models. TSMixer outperforms state-of-the-art MLP and Transformer models in forecasting by a considerable margin of 8-60%. It also outperforms the latest strong benchmarks of Patch-Transformer models (by 1-2%) with a significant reduction in memory and runtime (2-3X). The source code of our model is officially released as PatchTSMixer in the HuggingFace. Model: https://huggingface.co/docs/transformers/main/en/model_doc/patchtsmixer Examples: https://github.com/ibm/tsfm/#notebooks-links

1 INTRODUCTION

TSMixer adapts patch-based MLP-Mixer modeling to multivariate time-series forecasting and representation learning, adding heads and channel strategies for time-series structure. It reports stronger benchmark performance with substantially lower compute usage.

  • TSMixer is a patching-based architecture composed solely of MLP modules for multivariate forecasting and representation learning.
  • Its online reconciliation heads exploit hierarchical patch aggregation and cross-channel correlation to tune forecasts.
  • Hybrid channel modeling augments a channel-independent backbone with a cross-channel reconciliation head to handle noisy channel interactions and improve generalization across datasets.
  • A simple Gated Attention mechanism focuses the model on important features while supporting long-sequence interaction modeling without complex multi-head self-attention blocks.
  • TSMixer's modular design supports supervised and masked self-supervised learning, making it a potential building block for time-series foundation models.
  • TSMixer outperforms DLinear by 8%, surpasses PatchTST by 1-2%, and reduces PatchTST training time and memory usage by 2-3X.

2 RELATED WORK

Related work spans Transformer-based, patch-based, and MLP-based approaches for time-series forecasting, with MLP-Mixer methods adapting feature mixing beyond self-attention.

  • Transformer-based time-series models face quadratic time and memory complexity when processing long input sequences.
  • PatchTST and CrossFormer patch time series before Transformer processing to learn representations across patches.PatchTST uses channel independence, whereas CrossFormer uses channel mixing.
  • MLP-Mixer transforms inputs through permutations that mix features within and across patches without self-attention blocks.In vision, MLP-Mixer attains performance similar to CNNs and Transformers.
  • gMLP and ResMLP extend MLP-based architectures with spatial gating and residual connections, respectively.
  • Time-series MLP models include LightTS, DLinear, and MLP4Rec, using downsampling, decomposition with linear modeling, or MLP-based architectures.DLinear decomposes time series into trend and seasonality components and is reported to outperform Transformer-based state-of-the-art models.

3 METHODOLOGY

TSMixer processes patched multivariate time series with lightweight MLP-based backbones, prediction heads, and optional online reconciliation objectives. Its components address patch efficiency, channel structure, temporal modeling, and hierarchical aggregation.

  • Training workflow: TSMixer accepts moving-window history batches, applies normalization, patching, and permutation, then feeds the result into its backbone.Patching reduces model input tokens by a factor of s, improving runtime relative to point-wise Transformer approaches.
  • Training workflow: TSMixer supports supervised prediction and masked self-supervised pretraining followed by supervised fine-tuning.The supervised workflow minimizes forecast error, with customized objectives when reconciliation heads are activated.
  • TSMixer backbone: The model offers vanilla, channel-independent, and inter-channel backbones, with channel-independent mixing sharing weights across channels to reduce parameters.The channel-independent backbone also supports self-supervised modeling across datasets with different channel counts.
  • TSMixer backbone: Mixer layers learn correlations across patches, hidden features within patches, and channels using stacked MLP-Mixer operations.The patch embedding layer transforms each input patch before the mixer layers process these directions.
  • Forecast online reconciliation: Online reconciliation heads tune base forecasts using cross-channel dependencies and hierarchical patch aggregation during training.The cross-channel head models local forecast-horizon dependencies, while the hierarchical head jointly targets granular and patch-aggregated accuracy.
  • Forecast online reconciliation: The hybrid channel-independent backbone with a cross-channel reconciliation head provides stable improvements and supports backbone training across datasets with varying channel counts.The prediction head models task- and data-dependent channel correlations, while residual connections protect against noisy interactions.

4 EXPERIMENTS

Experiments evaluate TSMixer across established forecasting benchmarks, component variants, computational costs, self-supervised strategies, and learned patch representations. Results show strong accuracy and efficiency, while ablations support the proposed channel and reconciliation designs.

  • 4.1 Experimental Setup: The experiments use seven public multivariate datasets and compare standard Transformers, patch Transformers, MLPs, non-Transformers, and self-supervised models.Model variants combine Vanilla, Channel Independent, or Inter Channel backbones with gated attention and reconciliation heads.
  • 4.2 Supervised Multivariate Forecasting: TSMixer eliminates self-attention while retaining patching, producing approximately 3X reductions in MACs and parameters and 2X reductions in epoch time and peak memory.These reductions are reported for CI-TSMixer(G,H) relative to PatchTST; adding cross-channel reconciliation can substantially increase parameters on high-channel datasets.
  • 4.3 Component & Design Choice Analysis: Adding channel independence, gated attention, and hierarchy reconciliation improves TSMixer over the vanilla model by 11.5% across three datasets and 19.3% on average across all datasets.The combined gated-attention and hierarchy enhancements produce more stable improvements than either enhancement alone.
  • 4.3 Component & Design Choice Analysis: The hybrid channel approach improves CI-TSMixer by 13.5% overall after adding a cross-channel reconciliation head.The head’s context length must be selected according to the underlying data characteristics, and the hybrid approach outperforms the inter-channel backbone variant in the reported comparison.
  • 4.4 Forecasting via Representation Learning: Self-supervised evaluation examines three pretraining data strategies across Electricity, Traffic, and Weather, while patch-neighborhood analysis finds similar-shaped time series near one another in representation space.CI-TSMixer-Best also beats self-supervised PatchTST by 2% with reduced time and memory.

5 CONCLUSIONS AND FUTURE DIRECTIONS

The paper concludes that TSMixer is a purely MLP-based architecture with time-series-specific enhancements for forecasting and representation learning. Future work targets additional downstream tasks, stronger cross-dataset transfer, and newer Mixer variants.

  • 5 CONCLUSIONS AND FUTURE DIRECTIONS: TSMixer combines a channel-independent backbone with reconciliation heads and gated attention to improve MLP-based multivariate time-series modeling.The architecture is presented for forecasting and representation learning, with reduced compute resources relative to popular benchmarks.
  • 5 CONCLUSIONS AND FUTURE DIRECTIONS: Future work will extend TSMixer to classification and anomaly detection, improve transfer learning across datasets, and investigate newer Mixer variants.The proposed variants include Swin, Shift, and other newer Mixer architectures.

A.1 Datasets

The evaluation uses seven popular multivariate datasets for forecasting and representation learning. These datasets cover meteorological, traffic, electricity-consumption, and transformer-temperature measurements.

  • A.1 Datasets: Seven popular multivariate datasets are used for forecasting and representation learning, following established literature and public benchmark availability.The study uses the same data-loading parameters as PatchTST, including train, validation, and test split ratios.
  • A.1 Datasets: Weather contains 21 meteorological indicators, Traffic records freeway occupancy, Electricity measures 321 customers’ hourly consumption, and ETT records transformer sensors at different resolutions.

A.2 Supplementary Details

The implementation uses adaptive learning-rate selection, early stopping, PyTorch distributed training, and multi-node deployment support.

  • A.2 Supplementary Details: TSMixer uses optimal learning-rate selection, patience-10 early stopping, PyTorch, and distributed multi-GPU training.The library supports PyTorch DDP and deployment across multiple Kubernetes nodes for large-scale forecasting.

A.3 Supplementary Figures

The supplementary material provides figures and detailed benchmarking analyses for TSMixer, including model-head workflows, channel-mixing variants, and representation quality. It also compares TSMixer with several baseline models in supervised and self-supervised forecasting settings.

  • Supplementary figures: Figures 6–8 cover MLP and gated attention, model heads for two workflows, and correlations between patch time series and associated embeddings.Nearby patch representations are reported to correlate highly with similarly shaped and patterned patch time series.
  • Benchmarking: Appendix benchmarks compare TSMixer with LightTS, S4, CrossFormer, and TS2Vec, whose results are reported separately from the primary PatchTST and DLinear comparisons.Tables 8–10 present these additional supervised benchmark comparisons, with baseline results taken from prior work.
  • Detailed analyses: Detailed analyses examine CI-TSMixer-Best against PatchTST and DLinear for supervised and self-supervised multivariate forecasting.CI-TSMixer-Best selects the better of CI-TSMixer(G,H) and CI-TSMixer(G,H,CC) on each considered dataset.
  • Component analysis: CI-TSMixer improves over V-TSMixer by 18%, while adding gated attention and hierarchy reconciliation yields a total 19.3% improvement.Using both gated attention and hierarchy reconciliation together produces stable improvements over using either component alone; a cross-channel reconciliation head yields a total 19% improvement in the channel-mixing analysis.
Loading 2306.09364v4…