PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformers in Time Series: A Survey

Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, Liang Sun

arXiv:2202.07125v5cs.LGcs.AIeess.SPstat.ML

TL;DR

Time-series Transformers have proliferated, but their adaptations, applications, strengths, and limitations remain scattered across the literature. This survey organizes the field by network design and application, and finds that seasonal-trend decomposition can boost forecasting performance by 50% to 80%.

  • Problem

    Time-series Transformers face challenges modeling long- and short-range dependencies, seasonality, and long-sequence computational costs, motivating systematic synthesis of the field.

  • Method

    The paper surveys time-series Transformers through a taxonomy of network modifications and application domains, complemented by empirical analyses of robustness, model size, and seasonal-trend decomposition.

  • Results

    Seasonal-trend decomposition significantly boosts Transformer forecasting performance by 50% to 80% across evaluated attention modules.

  • Takeaways & Limitations

    The survey provides a structured account of time-series Transformer developments while identifying seasonal-trend decomposition as a consistent forecasting performance factor.

  • Takeaways & Limitations

    Vanilla Transformers require substantial data to improve generalization and avoid overfitting, while time-series-specific inductive biases can enhance performance.

Abstract

from arXiv · show

Transformers have achieved superior performances in many tasks in natural language processing and computer vision, which also triggered great interest in the time series community. Among multiple advantages of Transformers, the ability to capture long-range dependencies and interactions is especially attractive for time series modeling, leading to exciting progress in various time series applications. In this paper, we systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. In particular, we examine the development of time series Transformers in two perspectives. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers in order to accommodate the challenges in time series analysis. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification. Empirically, we perform robust analysis, model size analysis, and seasonal-trend decomposition analysis to study how Transformers perform in time series. Finally, we discuss and suggest future directions to provide useful research guidance. To the best of our knowledge, this paper is the first work to comprehensively and systematically summarize the recent advances of Transformers for modeling time series data. We hope this survey will ignite further research interests in time series Transformers.

1 Introduction

Transformers’ ability to model long-range dependencies and interactions has driven their adoption in time series, where variants address challenges across forecasting, anomaly detection, and classification. This survey organizes the field by network modifications and application domains, reviews key developments, and discusses future directions.

  • Motivation: Transformers model long-range dependencies and interactions in sequential data, making them appealing for time series modeling.Time series Transformer variants have been proposed to address domain-specific challenges.
  • Applications: Time series Transformers have been applied to forecasting, anomaly detection, and classification.These applications are supported by multiple proposed Transformer variants.
  • Survey scope: The survey proposes a taxonomy spanning network modifications and application domains for time series Transformers.It also introduces vanilla Transformer concepts before summarizing the field’s main developments.
  • Contribution: The authors describe the work as the first comprehensive and systematic review of key developments in Transformers for modeling time series data.The survey concludes by discussing future directions including inductive biases, GNN integration, pre-training, architecture variants, and NAS.

2 Preliminaries of the Transformer

The vanilla Transformer uses stacked encoder-decoder blocks built from self-attention, cross-attention, and position-wise feed-forward modules. It models sequence information without recurrence through positional encodings and combines scaled dot-product attention with multi-head projections, residual connections, and layer normalization.

  • Architecture: The vanilla Transformer follows an encoder-decoder structure with multiple identical blocks, including self-attention and feed-forward networks in the encoder and cross-attention in the decoder.Each decoder block inserts cross-attention between self-attention and the position-wise feed-forward network.
  • Positional Encoding: Unlike LSTM or RNN models, the Transformer has no recurrence and adds positional encodings to input embeddings to represent sequence information.Positional encodings may be absolute, relative, learned, or hybrid.
  • Attention: Transformer attention uses scaled dot-product Query-Key-Value computations and multi-head attention with H different sets of learned projections.Queries and keys have dimension Dk, values have dimension Dv, and query-key/value lengths may differ.
  • Feed-Forward and Normalization: The feed-forward network is fully connected, while deeper Transformer modules place residual connections and layer normalization around each module.The feed-forward parameters include W1, W2, b1, and b2, with dimensions specified by the model formulation.

3 Taxonomy of Transformers in Time Series

The paper organizes existing time series Transformers through a taxonomy spanning network modifications and application domains. It reviews adaptations at both the Transformer module and architecture levels to address challenges in time series modeling.

  • Taxonomy of Transformers in Time Series: The taxonomy categorizes time series Transformers by network modifications and application domains, covering module- and architecture-level changes for time series challenges.The framework is illustrated in Fig. 1 and guides a systematic review of existing time series Transformers.

4 Network Modifications for Time Series

Time-series Transformers modify positional encoding to exploit ordering and timestamp information, while architectural changes address quadratic self-attention costs and multi-resolution structure. These adaptations include fixed, learnable, and timestamp-based encodings, efficient attention mechanisms, and hierarchical designs.

  • Positional Encoding: Time-series Transformers encode input positions because temporal ordering is important, typically injecting positional vectors alongside the input series.Positional-vector construction falls into three main categories.
  • Positional Encoding: Learnable positional embeddings are more flexible and task-adaptive than fixed vanilla encodings, which are hand-crafted and less expressive.Some models learn an embedding vector jointly for each position index and the rest of the model.
  • Positional Encoding: Timestamp encoding incorporates accessible calendar and special timestamps as additional positional information, addressing information that vanilla Transformers hardly leverage.Examples include seconds through years, holidays, and events; Informer uses learnable embeddings for this purpose.
  • Efficient Attention: O(N^2) time and memory complexity makes vanilla self-attention a bottleneck for long sequences, motivating efficient Transformers with sparsity biases and other complexity reductions.Self-attention supports long-term dependencies with fully connected-layer path length but fewer parameters.
  • Hierarchical Architectures: Hierarchical Transformer architectures model time series at multiple resolutions by down-sampling between attention blocks; Informer uses stride-2 max-pooling to halve the series.This architectural renovation accommodates individual Transformer modules for time-series modeling.

5 Applications of Time Series Transformers

The survey organizes time series Transformer applications around forecasting, anomaly detection, and classification. It highlights task-specific adaptations for dependencies, irregular events, anomaly reconstruction, and multivariate classification.

  • Forecasting: Forecasting variants are categorized as module-level or architecture-level, with module-level designs introducing attention, normalization, and token-input inductive biases.Module-level variants comprise the majority of recent work, while architecture-level variants redesign the Transformer beyond its vanilla structure.
  • Forecasting: Forecasting Transformers exploit sparsity, low-rank approximation, normalization changes, decomposition, autocorrelation, and multi-scale structures to reduce complexity or improve performance.Examples include Logsparse attention, Non-stationary Transformer, Autoformer, Triformer, and Scaleformer.
  • Forecasting: DLinear challenges the necessity of Transformers for long-term forecasting, although PatchTST achieves a better numerical result than DLinear in that setting.The passage describes DLinear as a simpler MLP-based model that outperforms some Transformer baselines empirically.
  • Spatio-Temporal Forecasting: Spatio-temporal forecasting models jointly capture temporal and spatial dependencies using Transformer attention together with graph neural or graph convolution modules.Traffic Transformer and Spatial-temporal Transformer apply these components to traffic forecasting.
  • Anomaly Detection and Event Forecasting: Transformer-based anomaly detection models capture temporal dependencies, while TranAD combines Transformer encoder-decoders with adversarial training and records a nearly 11% F1-score drop when they are replaced.Transformer Hawkes models similarly summarize historical events and compute intensity functions, using sinusoidal encodings of event-time intervals for irregular event prediction.
  • Classification: GTN uses separate time-step-wise and channel-wise attention towers with learnable weighted concatenation and achieves state-of-the-art results on 13 multivariate time series classification datasets.The model’s two-tower design merges temporal and channel features through gating.

6 Experimental Evaluation and Discussion

The evaluation studies Transformer behavior on the ETTm2 benchmark through robustness, model-size, and seasonal-trend decomposition experiments. Results indicate that Transformer performance deteriorates with prolonged inputs, motivating designs that better exploit long sequences.

  • Experimental Setup: Experiments focus on popular time series Transformers on the challenging ETTm2 benchmark because statistical, RNN, and CNN baselines performed worse there.The study compares Transformer configurations rather than those weaker baseline families.
  • Robustness Analysis: Robustness is tested by prolonging input sequences to assess prediction power and stability with long-term inputs.The experiment reports forecasting 96 steps on ETTm2 using mean squared error comparisons.
  • Robustness Analysis: Transformer-based models deteriorate quickly as input length increases, limiting their practicality for long-term forecasting.The findings motivate further designs that can fully utilize long sequence inputs.
  • Model Size Analysis: The model-size analysis examines whether increasing Transformer layer count improves time series prediction, as it does in NLP and computer vision.Transformer capacity is commonly controlled by layer number, typically set between 12 and 128 in those fields.
  • Seasonal-Trend Decomposition Analysis: The seasonal-trend analysis evaluates attention modules within a simple moving-average decomposition architecture.The architecture follows the seasonal-trend decomposition approach proposed in Wu et al. (2021).

7 Future Research Opportunities

Future research should adapt Transformers to seasonal, periodic, and trend patterns, high-dimensional relationships, and task-specific pretraining. It should also explore broader architecture designs and automated hyper-parameter configuration.

  • 7 Future Research Opportunities: Transformers should incorporate seasonal, periodic, and trend patterns while addressing their substantial data requirements for generalization and overfitting avoidance.Vanilla Transformers make no assumptions about data patterns and characteristics, whereas seasonal and trend patterns are key features of time series.
  • 7 Future Research Opportunities: Future methods should capture relationships among dimensions in high-dimensional multivariate and spatio-temporal time series, potentially by combining GNNs with Transformers or attention.GNNs naturally model spatial dependencies or relationships among dimensions, and their combination with Transformers or attention has shown significant performance benefits.
  • 7 Future Research Opportunities: Appropriate pre-trained Transformer models remain to be developed for different time series tasks, since existing studies mainly focus on classification.Large-scale pre-trained Transformers have boosted NLP and CV performance, but comparatively few time series studies use pretraining.
  • 7 Future Research Opportunities: Research should investigate architecture-level Transformer designs beyond attention-module modifications, including lightweight models, cross-block connectivity, adaptive computation time, and recurrence.Most developed time series Transformers retain the vanilla architecture and primarily modify attention, while NLP and CV variants offer alternative designs.
  • 7 Future Research Opportunities: NAS could automate Transformer design because manually configuring embedding dimensions and numbers of heads or layers is time-consuming and often suboptimal.Neural architecture search has been used to discover effective deep neural architectures and automate Transformer design in NLP and CV.

8 Conclusion

The paper surveys time series Transformers through a taxonomy of network design and application, evaluates representative methods’ strengths and limitations, and identifies future research directions.

  • The survey organizes time series Transformer methods by network design and application, summarizes representative approaches, evaluates their strengths and limitations, and highlights future directions.
Loading 2202.07125v5…