PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Transformers Effective for Time Series Forecasting?

Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu

arXiv:2205.13504v3cs.AIcs.LG

TL;DR

Long-term time series forecasting requires modeling temporal relations, raising questions about whether permutation-invariant Transformer attention is suitable. The paper introduces simple one-layer linear baselines and finds they outperform existing Transformer-based models across nine benchmarks, while serving as competitive baselines for future research.

  • Problem

    The paper questions whether permutation-invariant Transformer self-attention can adequately preserve the temporal relations central to long-term time series forecasting.

  • Method

    The authors introduce LTSF-Linear, one-layer linear models that directly regress historical time series to forecast future values.

  • Results

    LTSF-Linear outperformed existing complex Transformer-based models in all nine benchmark cases.

  • Takeaways & Limitations

    The findings suggest Transformer-based temporal modeling capabilities are exaggerated on existing long-term forecasting benchmarks, while LTSF-Linear provides a competitive baseline.

  • Takeaways & Limitations

    LTSF-Linear has limited model capacity and is hard to capture temporal dynamics caused by change points.

Abstract

from arXiv · show

Recently, there has been a surge of Transformer-based solutions for the long-term time series forecasting (LTSF) task. Despite the growing performance over the past few years, we question the validity of this line of research in this work. Specifically, Transformers is arguably the most successful solution to extract the semantic correlations among the elements in a long sequence. However, in time series modeling, we are to extract the temporal relations in an ordered set of continuous points. While employing positional encoding and using tokens to embed sub-series in Transformers facilitate preserving some ordering information, the nature of the \emph{permutation-invariant} self-attention mechanism inevitably results in temporal information loss. To validate our claim, we introduce a set of embarrassingly simple one-layer linear models named LTSF-Linear for comparison. Experimental results on nine real-life datasets show that LTSF-Linear surprisingly outperforms existing sophisticated Transformer-based LTSF models in all cases, and often by a large margin. Moreover, we conduct comprehensive empirical studies to explore the impacts of various design elements of LTSF models on their temporal relation extraction capability. We hope this surprising finding opens up new research directions for the LTSF task. We also advocate revisiting the validity of Transformer-based solutions for other time series analysis tasks (e.g., anomaly detection) in the future. Code is available at: \url{https://github.com/cure-lab/LTSF-Linear}.

1. Introduction

The introduction challenges the effectiveness of Transformer-based models for long-term time series forecasting, arguing that self-attention loses temporal information despite positional encoding. It proposes LTSF-Linear as a simple direct-forecasting baseline and supports the challenge with benchmark comparisons and empirical studies.

  • Motivation: Self-attention is permutation-invariant and can inevitably lose temporal information even when positional encoding preserves some ordering.The authors characterize self-attention as “anti-order” for extracting temporal relations.
  • Motivation: Existing Transformer-based LTSF comparisons may overstate their advantage because non-Transformer baselines use autoregressive or iterated multi-step forecasting with error accumulation.The study therefore evaluates Transformer-based models against direct multi-step forecasting strategies.
  • Method: LTSF-Linear uses a one-layer linear model to regress historical series and forecast future values directly, providing a simple baseline for LTSF.The authors hypothesize that long-term forecasting is feasible mainly for series with relatively clear trends and periodicity.
  • Evaluation: The study compares LTSF-Linear with existing Transformer-based LTSF solutions on nine benchmarks and examines long-input modeling, order sensitivity, positional encoding, sub-series embedding, and efficiency.These experiments are intended to test models’ temporal relation extraction capabilities and inform future research.
  • Conclusion: The authors conclude that Transformers’ temporal modeling capabilities are exaggerated on existing LTSF benchmarks, while LTSF-Linear achieves better prediction accuracy but remains a simple baseline.They also advocate revisiting Transformer validity for other time series analysis tasks.

2. Preliminaries: TSF Problem Formulation

The time-series forecasting problem predicts future values of multiple variates from historical observations. For multiple horizons, forecasting can be iterative or direct, with their relative advantages depending on forecast length and single-step accuracy.

  • TSF Problem Formulation: Given historical observations of C variates, time-series forecasting predicts their values at T future time steps.The historical and forecast series are represented as multivariate observations indexed by variate and time step.
  • TSF Problem Formulation: Iterated multi-step forecasting learns a single-step forecaster and applies it repeatedly, whereas direct multi-step forecasting optimizes the multi-step objective jointly.These are the two standard approaches when T > 1.
  • TSF Problem Formulation: IMS predictions have smaller variance but accumulate errors, while DMS is more accurate when unbiased single-step modeling is difficult or T is large.IMS is preferable with a highly accurate single-step forecaster and relatively small T.

3. Transformer-Based LTSF Solutions

Transformer-based LTSF research adapts the vanilla architecture to address efficiency, decoding, and time-series-specific modeling challenges. However, its semantic-correlation premise and permutation-invariant self-attention create fundamental concerns for extracting temporal relations from raw numerical sequences.

  • Transformer-based models have attracted substantial research interest for long-term time series forecasting because multi-head self-attention performs strongly in natural language processing and computer vision.
  • Vanilla Transformers face quadratic time/memory complexity and autoregressive decoder error accumulation when applied to LTSF.
  • Time series decomposition: Autoformer introduces seasonal-trend decomposition behind each neural block, using moving averages to extract trend-cyclical components before modeling the residual sequence.
  • Recent variants improve efficiency through sparse or structured attention, including LogTrans at O(LlogL), Pyraformer at O(L), Informer at O(LlogL), and FEDformer at O(L).LogTrans uses Logsparse masking, Pyraformer pyramidal attention, Informer ProbSparse attention with distilling, and FEDformer Fourier- and wavelet-enhanced blocks.
  • Transformer self-attention is permutation-invariant, so temporal-relation modeling depends largely on positional encodings despite time series containing few point-wise semantic correlations.

4. An Embarrassingly Simple Baseline

The authors introduce LTSF-Linear, a simple direct multi-step temporal linear baseline, to test whether prior Transformer-LTSF gains mainly arise from their DMS strategy. Its variants, DLinear and NLinear, use decomposition and normalization to address trends and distribution shifts.

  • Motivation: The baseline tests the hypothesis that prior Transformer-LTSF improvements largely result from their direct multi-step forecasting strategy rather than the Transformer architecture itself.Existing non-Transformer baselines use iterative multi-step forecasting techniques known to suffer error accumulation.
  • LTSF-Linear: LTSF-Linear directly regresses historical time series onto future predictions through a weighted temporal sum, using a linear layer W ∈ R^(T×L).The formulation is X̂_i = W X_i for each variate.
  • LTSF-Linear: The model shares weights across variates and does not model spatial correlations.X̂_i and X_i denote the prediction and input for the ith variate.
  • Variants: LTSF-Linear comprises Vanilla Linear, a one-layer model, plus DLinear and NLinear variants designed for different domains and preprocessing needs.The variants target time series from domains including finance, traffic, and energy.
  • DLinear: DLinear decomposes inputs into trend and remainder components, applies separate one-layer linear layers, and sums their outputs to improve performance when trends are clear.Its decomposition uses a moving average kernel and explicitly handles the trend component.
  • NLinear: NLinear subtracts the sequence’s last value before linear mapping and adds it back afterward, providing simple normalization for distribution shifts.The subtraction and addition are intended to boost performance when the dataset exhibits a distribution shift.

5. Experiments

Experiments across nine multivariate real-world benchmarks show that simple LTSF-Linear models generally outperform sophisticated Transformer-based forecasters, while additional analyses expose weaknesses in temporal relation extraction and practical efficiency. The results also identify dataset-dependent exceptions and mechanisms affecting performance.

  • Experimental setup: Nine multivariate real-world datasets are evaluated using MSE and MAE, with five Transformer-based methods and the naive Repeat baseline.The datasets include ETT variants, Traffic, Electricity, Weather, ILI, and Exchange-Rate.
  • Quantitative results: 20% ∼50% improvements: LTSF-Linear surpasses FEDformer in most multivariate forecasting cases, despite not modeling correlations among variates.NLinear and DLinear also show advantages in handling distribution shift.
  • Quantitative results: FEDformer is competitive on ETTh1, whereas existing complex Transformers are generally ineffective across the nine benchmarks and LTSF-Linear is a powerful baseline.Frequency processing supplies time-series inductive bias and benefits temporal feature extraction on ETTh1.
  • Quantitative results: 45%: Repeat outperforms all Transformer-based methods on Exchange-Rate, because Transformer trend errors can overfit sudden-change noise.Repeat performs worse on long-term seasonal datasets such as Electricity and Traffic.
  • Look-back window analysis: Larger look-back windows boost all LTSF-Linear models, while Transformer performance deteriorates or remains stable, indicating limited temporal information extraction.The study tests L ∈ {24, 48, 72, 96, 120, 144, 168, 192, 336, 504, 672, 720} for T=720; additional results support the conclusion in almost all cases.
  • Temporal relation extraction: 73.28% and 56.91%: On ETTh1, FEDformer and Autoformer’s average performance drops under whole-sequence shuffling, showing sensitivity to lost order information.Their inductive biases help extract temporal information when temporal patterns are clear, while Transformers can still suffer temporal information loss despite embeddings.

6. Conclusion and Future Work

The paper questions the effectiveness of Transformer-based solutions for long-term time series forecasting through surprising comparisons with the simple, interpretable LTSF-Linear baseline. Future work should address LTSF-Linear’s limited capacity, including its difficulty capturing change-point dynamics, through new models, data processing, and benchmarks.

  • Conclusion: The study questions the effectiveness of favored Transformer-based solutions for long-term time series forecasting using LTSF-Linear as a simple DMS forecasting baseline.The authors emphasize that their contribution lies in raising an important question, showing surprising comparisons, and examining Transformer limitations rather than proposing a linear model.
  • Future work: LTSF-Linear provides a simple, competitive, and strongly interpretable baseline for future long-term forecasting research.Its limited model capacity means it is intended as a baseline rather than a complete solution.
  • Future work: The one-layer linear network is hard to capture temporal dynamics caused by change points.This limitation motivates further investigation rather than invalidating the baseline’s usefulness.
  • Future work: New model designs, data processing methods, and benchmarks are needed to tackle the challenging long-term time series forecasting problem.The authors identify these directions as having substantial potential for future research.

A. Related Work: Non-Transformer-Based TSF Solutions

Non-Transformer time-series forecasting evolved from expertise-intensive statistical models to data-driven machine learning and deep-learning approaches. Among DNN methods, RNNs summarize history in memory states, while CNNs capture local temporal features and support distinct forecasting strategies.

  • Statistical and machine-learning methods: Statistical forecasting methods such as ARIMA, exponential smoothing, and structural models have been used since the 1970s but generally require substantial domain expertise.These approaches are presented as long-standing solutions for time-series forecasting across many applications.
  • Statistical and machine-learning methods: GBRT learns temporal dynamics in a data-driven manner, yet still requires manual feature engineering and model design.Deep neural networks are introduced as an alternative enabled by abundant data and powerful representation learning.
  • DNN architectures: RNN-based methods recursively update internal memory states to compactly summarize past information for forecasting.RNNs are identified as one of the two popular non-Transformer DNN architectures applied to time-series forecasting.
  • DNN architectures: CNN-based methods use convolutional filters to capture local temporal features, with forecasting implemented through either IMS or DMS techniques.For CNN-based methods, the distinction depends on whether the decoder operates autoregressively; RNN-based methods belong to IMS forecasting.

B. Experimental Details … C.1. Comparison of Univariate Forecasting

The experiments use nine real-life datasets and compare LTSF-Linear with Transformer-based forecasting methods under varied settings. On univariate ETT forecasting, LTSF-Linear—especially NLinear—consistently outperforms Transformer methods, while last-value normalization helps address distribution shifts.

  • B. Experimental Details: The main experiments use nine widely used datasets for long-term time series forecasting.The datasets are introduced in the experimental details.
  • B.1. Data Descriptions: ETT contains two hourly and two 15-minute datasets, each with seven oil and load features from July 2016 to July 2018.The datasets cover electricity transformer temperature measurements.
  • B.1. Data Descriptions: Traffic records hourly San Francisco freeway road occupancy rates, while Electricity records hourly consumption from 321 clients.Traffic spans 2015–2016, and Electricity spans 2012–2014.
  • B.1. Data Descriptions: Exchange-Rate contains daily exchange rates for eight countries, and Weather contains 21 indicators recorded every 10 minutes in Germany during 2020.Exchange-Rate spans 1990–2016.
  • B.1. Data Descriptions: ILI contains weekly U.S. influenza-like illness data from the Centers for Disease Control and Prevention spanning 2002–2021.It measures the ratio of patients seen with influenza-like illness to the number of patients.
  • B.2. Implementation Details: Autoformer, Informer, and vanilla Transformer implementations come from Autoformer, whereas FEDformer and Pyraformer implementations come from their respective repositories.Existing Transformer-based methods use their default hyperparameters; DLinear uses a moving-average kernel size of 25.
  • C. Additional Comparison with Transformers: The study compares LTSF-Linear and LTSF-Transformer on four ETT datasets under various look-back window sizes across all benchmarks.It also examines examples where existing Transformers fail to exploit large look-back windows.
  • C.1. Comparison of Univariate Forecasting: LTSF-Linear, especially NLinear, consistently outperforms Transformer-based methods by a large margin in most cases on univariate ETT forecasting.ETTh1 and ETTh2 exhibit serious training–test distribution shifts, which last-value normalization can greatly relieve.

C.2. Comparison under Different Look-back Windows … D.2. The Features of LTSF-Linear

Across varying look-back windows, LTSF-Linear improves substantially on most datasets while Transformer performance often fluctuates or worsens. The model also offers short dependency paths, efficiency, interpretability, and ease of use, while NLinear addresses testing-distribution shifts through prediction shifting.

  • C.2. Comparison under Different Look-back Windows: Figure 6 compares five LTSF-Transformers with LTSF-Linear across increasing look-back windows on hourly datasets, using forecasting steps {24, 720}.The hourly look-back windows range from 24 to 720 points, representing 1 to 30 days.
  • C.2. Comparison under Different Look-back Windows: As look-back windows increase, LTSF-Linear is significantly boosted on most datasets, whereas Transformer performance often fluctuates or worsens.The passage gives ETTm1 and Traffic as examples of improved LTSF-Linear performance.
  • C.2. Comparison under Different Look-back Windows: Exchange-Rate does not improve with longer look-back windows, which the authors attribute to financial data’s low information-to-noise ratio.This limitation is reported for the Exchange-Rate results in Figure 6(m) and (n).
  • D.1. Motivation of NLinear: NLinear shifts predictions by subtraction and addition toward the true-value distribution to mitigate large errors caused by testing distribution shifts.The shift addresses cases where normalized test data has a nonzero mean or predictions fall outside the true-value distribution.
  • D.2. The Features of LTSF-Linear: LTSF-Linear has an O(1) maximum signal traversing path length, enabling capture of both short-range and long-range temporal relations.The passage links shorter traversing paths with better dependency capture.
  • D.2. The Features of LTSF-Linear: Because LTSF-Linear has at most two linear layers, it uses less memory and fewer parameters and has faster inference than existing Transformers.The comparison is referenced to Table 8 in the main paper.
  • D.2. The Features of LTSF-Linear: Weights from LTSF-Linear’s seasonality and trend branches can be visualized after training to provide insights into predicted values.This visualization is presented as an interpretability benefit.
  • D.2. The Features of LTSF-Linear: LTSF-Linear is easy to obtain without tuning model hyper-parameters.The passage identifies this as an easy-to-use characteristic.

D.3. Interpretability of LTSF-Linear

LTSF-Linear’s weights directly expose how forecasting works and reveal characteristics of the underlying data. DLinear visualizations show recency effects, while Traffic weights identify daily and weekly periodicities.

  • D.3. Interpretability of LTSF-Linear: LTSF-Linear’s linear-layer weights directly reveal how the model works and can expose characteristics of the forecasting data.Because the model is linear, its weights can be visualized and interpreted directly.
  • D.3. Interpretability of LTSF-Linear: DLinear visualizations use fixed input length 96 across four forecasting horizons, with weights initialized uniformly at 1/L for smoother patterns.The same weight is assigned to every forecasting time step in the look-back window at training start.
  • D.3. Interpretability of LTSF-Linear: On Exchange-Rate, the trend layer assigns greater weights to information closer to the outputs, reflecting larger contributions to predictions.Clear patterns are difficult to observe because financial data lacks periodicity and seasonality.
  • D.3. Interpretability of LTSF-Linear: Traffic weights are highest at the latest time step and at offsets 0, 167, 335, 503, and 671, indicating daily and weekly periodicities.The dataset has 24 time steps per day and 168 time steps per week.
Loading 2205.13504v3…