PaperlayerView exact revision on arXiv

Source-linked AI summary

iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, Mingsheng Long

arXiv:2310.06625v4cs.LG

TL;DR

Transformer forecasters face performance and efficiency challenges on multivariate time series because temporal tokens fuse heterogeneous variates and can lose useful correlations. iTransformer keeps native Transformer components but applies them to independently embedded variate tokens, using attention for multivariate correlations and feed-forward networks for series representations. It achieves state-of-the-art performance on real-world benchmarks and consistently improves several Transformer variants, while its attention-based design assumes attention is applicable to multivariate correlations.

  • Problem

    Temporal-token Transformers face performance degradation and computation growth on longer lookback windows, while fusing heterogeneous variates can hinder variate-centric representations and meaningful attention.

  • Method

    iTransformer embeds each time series independently as a variate token, applies attention across variates, and uses feed-forward networks for series representations without modifying native Transformer components.

  • Results

    iTransformer achieves state-of-the-art performance on real-world forecasting benchmarks and averages 38.9% promotion on Transformer, 36.1% on Reformer, 28.5% on Informer, 16.8% on Flowformer, and 32.2% on Flashformer.

  • Takeaways & Limitations

    The inverted architecture provides a fundamental forecasting backbone with framework generality across Transformer variants and improved use of multivariate correlations.

  • Takeaways & Limitations

    The architecture presupposes that attention is applicable for multivariate correlation.

Abstract

from arXiv · show

The recent boom of linear forecasting models questions the ongoing passion for architectural modifications of Transformer-based forecasters. These forecasters leverage Transformers to model the global dependencies over temporal tokens of time series, with each token formed by multiple variates of the same timestamp. However, Transformers are challenged in forecasting series with larger lookback windows due to performance degradation and computation explosion. Besides, the embedding for each temporal token fuses multiple variates that represent potential delayed events and distinct physical measurements, which may fail in learning variate-centric representations and result in meaningless attention maps. In this work, we reflect on the competent duties of Transformer components and repurpose the Transformer architecture without any modification to the basic components. We propose iTransformer that simply applies the attention and feed-forward network on the inverted dimensions. Specifically, the time points of individual series are embedded into variate tokens which are utilized by the attention mechanism to capture multivariate correlations; meanwhile, the feed-forward network is applied for each variate token to learn nonlinear representations. The iTransformer model achieves state-of-the-art on challenging real-world datasets, which further empowers the Transformer family with promoted performance, generalization ability across different variates, and better utilization of arbitrary lookback windows, making it a nice alternative as the fundamental backbone of time series forecasting. Code is available at this repository: https://github.com/thuml/iTransformer.

1 INTRODUCTION

The paper argues that Transformer forecasters are poorly configured for multivariate time series because temporal tokens fuse heterogeneous variates and have limited receptive fields. iTransformer inverts the tokenization, using variate-centric representations and native Transformer components to capture multivariate correlations and achieve state-of-the-art forecasting.

  • Transformer forecasters embed heterogeneous variates from each timestamp into indistinguishable temporal tokens, while simple linear models can outperform them in performance and efficiency.
  • Timestamp-level tokens can erase multivariate correlations, expose only local temporal context, and combine time-unaligned events.
  • iTransformer embeds each complete variate series independently as a variate token, aggregating global series representations for attention-based multivariate correlation modeling.
  • The model uses attention for multivariate correlations and layer normalization and feed-forward networks for series representations without modifying native Transformer modules.
  • iTransformer achieves comprehensive state-of-the-art performance on real-world forecasting benchmarks and addresses identified Transformer forecasting pain points.

2 RELATED WORK

Prior Transformer forecasters modify components, architectures, or time-series processing to address forecasting challenges. iTransformer instead retains native components and applies them on inverted dimensions, defining a distinct architectural direction.

  • Transformer-based forecasters are categorized into four groups according to whether they modify Transformer components and architecture.
  • The first category adapts components, especially attention, to model temporal dependencies and reduce complexity on long sequences.
  • The second category uses time-series processing strategies such as stationarization, channel independence, and patching, which bring consistently improved performance.
  • iTransformer belongs to the fourth category because it modifies neither native Transformer components nor their basic operations, but applies them on inverted dimensions.

3 ITRANSFORMER

iTransformer inverts the usual Transformer dimensions: whole variate series become tokens, attention models variate correlations, and shared feed-forward layers learn series representations. It retains the encoder-style Transformer arrangement while reconsidering normalization and component roles for multivariate forecasting.

  • Architecture: The model uses an encoder-only Transformer architecture with embedding, Transformer blocks, and projection, while allowing efficient attention variants as plugins.Its attention input can vary in token number between training and inference, supporting flexibility as the variate count grows.
  • Layer normalization: Layer normalization is reconsidered on the inverted dimensions to reduce discrepancies among variates rather than fusing potentially delayed or noncausal timestamp measurements.The conventional normalization of same-timestamp multivariate representations can introduce interaction noise when those measurements do not represent the same event.
  • Structure overview: iTransformer independently embeds each variate’s whole time series into a token, instead of combining multiple variates from one timestamp.The resulting token matrix contains N variate tokens of dimension D.
  • Feed-forward network: The shared feed-forward network independently processes each variate token to extract nonlinear, series-level representations.The paper connects this shared operation with learning intrinsic properties such as amplitude, periodicity, and frequency spectra across distinct time series.
  • Design rationale: The paper reports that this division of labor promotes performance with enlarged lookback series and generalization to unseen variates.The reported benefits are attributed to combining linear-layer representation learning with attention-based multivariate correlation modeling.
  • Self-attention: Self-attention operates across variate tokens to capture and expose multivariate correlations through an N×N attention score map.After feature normalization, score entries reflect pairwise variate relationships, so highly correlated variates receive greater weighting in representation interaction.

4 EXPERIMENTS

Experiments evaluate iTransformer across real-world forecasting benchmarks, Transformer variants, unseen variates, longer lookback windows, and component configurations. The results report strong forecasting performance, consistent promotion of Transformer variants, variate generalization, improved use of enlarged lookback windows, and evidence supporting the inverted allocation of Transformer components.

  • Experimental Setup: iTransformer is evaluated on various time series forecasting applications using seven real-world datasets and ten established forecasting baselines.The baselines include Transformer-based, linear-based, and TCN-based methods.
  • Forecasting Results: iTransformer is particularly effective for high-dimensional time series and outperforms PatchTST in many PEMS cases.The authors attribute this difference to aggregating whole-series variations rather than relying on patching that may lose locality during rapid fluctuations.
  • iTransformers Generality: 38.9% promotion on Transformer, 36.1% on Reformer, 28.5% on Informer, 16.8% on Flowformer, and 32.2% on Flashformer are reported on average.The inverted framework consistently improves the evaluated Transformer architectures, while efficient attention addresses computation related to numerous variates.
  • iTransformers Generality: Training on 20% of variates enables iTransformers to forecast all variates without fine-tuning and generally with smaller increases than Channel Independence.The experiment averages results across five variate partitions; iTransformers also predict all variates directly at inference.
  • iTransformers Generality: With lookback lengths T ∈{48, 96, 192, 336, 720} and fixed S = 96, the inverted framework improves performance on enlarged windows.The analysis supports applying MLPs on the temporal dimension while attention operates on inverted dimensions.
  • Model Analysis: Ablations and representation analyses indicate that attention on variates and feed-forward networks on temporal dimensions generally provide the best design.The reported CKA analysis shows a clear division between Transformer variants and iTransformers, supporting more appropriate series representations after inversion.

5 CONCLUSION AND FUTURE WORK

The paper concludes that iTransformer inverts Transformer structure without modifying native components, using attention for multivariate correlations and feed-forward networks for series representations. It reports state-of-the-art performance, framework generality, and future interest in large-scale pre-training and additional time series tasks.

  • Conclusion and Future Work: iTransformer inverts Transformer structure without modifying native modules, assigning attention to multivariate correlations and feed-forward networks to series representations.The conclusion describes independent series as variate tokens.
  • Conclusion and Future Work: The model achieves state-of-the-art performance and exhibits framework generality supported by analysis.Future work includes large-scale pre-training and additional time series analysis tasks.

6 ETHICS STATEMENT

The paper states that its scope is limited to time series forecasting and reports no potential ethical risk.

  • Ethics Statement: The work focuses only on time series forecasting and identifies no potential ethical risk.

7 REPRODUCIBILITY STATEMENT

The paper states that the model architecture is formalized with equations and that implementation details are provided in the Appendix. It also states that the code will be made public after acceptance.

  • Reproducibility Statement: The main text formalizes the model architecture with equations, while the Appendix contains datasets, metrics, model, and experiment configurations.
  • Reproducibility Statement: The authors state that the code will be made public once the paper is accepted.

A.1 DATASET DESCRIPTIONS

The experiments use public forecasting benchmarks and real-world Market datasets under chronological splits and varied lookback and prediction horizons. Implementation fixes the main training configuration while varying model dimensions and seeds for evaluation.

  • Datasets: Seven real-world datasets evaluate iTransformer, including ETT, Exchange, Weather, and other forecasting benchmarks.The supplied passage begins the dataset list and identifies ETT's four subsets and Exchange's eight-country exchange-rate panel.
  • Datasets: The Market collection records minute-sampled Alipay transaction server loads across six transaction-domain subsets, with 285–759 variates.Its observations span January 30th to April 9th, 2023.
  • Protocols: Chronological train-validation-test splits prevent data leakage, while standard datasets use lookback 96 and prediction lengths {96, 192, 336, 720}.PEMS uses prediction lengths {12, 24, 36, 48}; Market uses lookback 144 and prediction lengths {12, 24, 72, 144}.
  • Implementation: The architecture transposes X, embeds series into variate tokens, applies inverted Transformer blocks with self-attention, FFN, and LayerNorm, then returns predictions.The algorithm specifies input length T, prediction length S, variate count N, token dimension D, and block count L.
  • Implementation: Experiments use Adam, L2 loss, batch size 32, 10 epochs, 2–4 inverted blocks, and representation dimensions 256 or 512.The implementation runs on a single NVIDIA P100 16GB GPU and tests three initial learning rates.

B ABLATION STUDIES

Ablations indicate that iTransformer’s inverted assignment of attention and feed-forward components is generally superior, especially as variate counts increase. Sensitivity analyses identify learning rate as the main tuning concern for large multivariate datasets.

  • Component ablations: iTransformer generally performs best by using self-attention for multivariate correlations and FFN for series representations.The ablation contrasts this arrangement with designs that replace or remove Transformer components.
  • Component ablations: Vanilla Transformer arrangements and designs applying both attentions show relatively poor results, suggesting temporal-token attention is unsuitable in these settings.The passage presents this as one possible reason for degraded performance on lagged time series.
  • Component ablations: With many variates, explicit variate correlation is increasingly important because vanilla embedding obscures heterogeneous measurements and variate independence.Attention on variates improves performance on challenging multivariate datasets.
  • Component ablations: FFN on both dimensions can perform fairly well when variate counts are small, such as Weather with 21 variates.The passage contrasts this case with challenging datasets containing more variates.
  • Hyperparameter sensitivity: Learning rate requires careful selection for large-variate ECL and Traffic, while larger block counts and hidden dimensions are not consistently preferred.The sensitivity study uses lookback T = 96 and forecast window S = 96.

D MODEL EFFICIENCY

Efficiency depends on the number of variates because inverted attention scales with variate tokens, while its learned maps provide interpretable multivariate-correlation structure. The model is faster than other Transformers on Weather and can be accelerated further for high-dimensional data.

  • Efficiency: iTransformer exceeds other Transformers in efficiency on Weather with 21 variates, while on Traffic with 862 variates it trains faster but has similar memory footprints.The comparison uses 96 lookback time steps and representative Weather and Traffic datasets.
  • Efficiency: O(N^2) attention makes efficiency depend on token count: vanilla Transformer uses N = 96 temporal tokens, whereas iTransformer uses N = 862 variate tokens on Traffic.This token-count difference can make vanilla Transformer more efficient in that case despite iTransformer’s stronger forecasting performance.
  • Efficiency: Linear-complexity attention or training on 20% of variates while forecasting all variates can bring iTransformer’s speed and memory footprint near linear models.The two strategies can also be combined.
  • Interpretability: Inverted self-attention maps resemble raw lookback correlations in shallow layers and become more transformed in deeper layers.The visualization compares learned pre-Softmax maps with Pearson correlations on Solar-Energy cases.
  • Interpretability: Market attention maps partition variates by application, with within-application similarities closer than correlations across different groups.Each variate represents a service interface, and adjacent variates are grouped by application category.

E.2 VISUALIZATION OF PREDICTION RESULTS

Prediction visualizations across representative datasets show iTransformer tracking future-series variations more precisely than the compared models. The authors connect this advantage to representing variates independently despite delays, offsets, and heterogeneous measurements.

  • Prediction visualizations: Across four representative datasets, iTransformer predicts future-series variations most precisely among the showcased models.The comparison includes PatchTST, DLinear, Crossformer, Autoformer, and Transformer.
  • Traffic structure: Traffic series exhibit strong correlations alongside phase offsets caused by systematic time lags across sensors in different highway areas.The example includes synchronized sensors and delayed or advanced relationships between sensor series.
  • Traffic structure: Attention over temporal tokens can degrade substantially when timestamp measurements contain inherent delays rather than representing the same event semantically.The passage links this observation to the poor performance of temporal-attention ablation designs on Traffic.
  • Design rationale: Distinct measurements such as temperature and rainfall create additional risks for timestamp-token embedding in multivariate forecasting.The paper proposes embedding each whole series as a variate token as a more robust alternative for delayed and inconsistent measurements.
  • Overall comparison: Comparisons across all datasets show consistent improvements from applying attention and feed-forward networks on inverted dimensions.The authors present this as an instructive direction for Transformer-based forecasting and report that the framework also promotes several Transformer variants.

F.3 FULL RESULTS OF VARIATE GENERALIZATION

The variate-generalization evaluation trains on only 20% of variates and tests forecasting across all variates, alongside broader benchmark results. iTransformer is reported to learn transferable representations and offers several avenues for further architectural improvement.

  • Variate generalization: Models are trained using only 20% of each dataset’s variates and then forecast all variates without fine-tuning.The evaluation divides variates into five folders and averages results across the five training folds.
  • Baselines: The evaluation compares iTransformers with CI-Transformers, whose channel-independent inference predicts variates one by one and can be time-consuming.
  • Variate generalization: iTransformers can be naturally trained with 20% of variates and forecast all variates, demonstrating transferable representations.
  • Benchmark coverage: The paper reports full forecasting results for PEMS, long-term forecasting benchmarks, and the Market transaction-forecasting dataset across prediction lengths.The tables report averages across prediction lengths, with input length 96 for the PEMS and long-term forecasting settings.
  • Scope and extensions: The inverted architecture is described as particularly suitable for high-dimensional series with numerous variates and complicated correlations, while leaving distribution-shift improvements for future work.The discussion identifies efficient attention, structural temporal modeling, fine-grained variate tokenization, and improved embeddings as open directions.
Loading 2310.06625v4…