PaperlayerView exact revision on arXiv

Source-linked AI summary

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, Jayant Kalagnanam

arXiv:2211.14730v2cs.LGcs.AI

TL;DR

Long-term multivariate time-series forecasting and representation learning need efficient Transformer designs that can capture meaningful temporal structure. PatchTST uses subseries-level patching and shared channel-independent processing, outperforming baselines in supervised forecasting while showing strong self-supervised and transfer-learning performance.

  • Problem

    The paper addresses forecasting T future values from multivariate time series given a look-back window of length L.

  • Method

    PatchTST segments each univariate channel into subseries-level patches and processes channels independently with shared Transformer weights.

  • Results

    PatchTST outperforms other baselines in supervised forecasting and shows strong self-supervised representation-learning and transfer-learning performance.

  • Takeaways & Limitations

    Patching retains local semantic information, reduces Transformer complexity, and supports longer look-back windows for time-series forecasting.

  • Takeaways & Limitations

    The model does not yet properly incorporate dependencies between different channels, which the authors identify as an important future step.

Abstract

from arXiv · show

We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.

1 INTRODUCTION

PatchTST addresses time-series forecasting with subseries-level patching and channel-independent Transformer inputs, improving efficiency, longer-context learning, and representation learning. The introduction highlights reduced attention complexity, improved MSE with longer look-back windows, and strong supervised, self-supervised, and transfer-learning performance.

  • Patching: Patching aggregates time steps into subseries-level tokens to preserve local semantic information that point-wise inputs lack.This enhances locality and captures more comprehensive semantic information than point-level representations.
  • Channel-independence: Channel-independence treats each channel as a univariate series while sharing embedding and Transformer weights across series.This contrasts with channel-mixing, which projects all features together into the embedding space.
  • Efficiency: O(N^2) attention time and space complexity is reduced quadratically because patching yields N ≈ L/S instead of N = L.Here, N is the number of input tokens, L is the input sequence length, and S is the patch stride.
  • Longer look-back window: 0.518 to 0.397 MSE is achieved when the look-back window increases from 96 to 336, demonstrating longer-history learning.The introduction notes that simply extending the look-back window increases memory and computational usage.
  • Representation learning: PatchTST demonstrates model effectiveness through supervised forecasting, ablation studies, self-supervised representation learning, and transfer learning.The introduction states that self-supervised and transfer-learning results achieve SOTA performance.

2 RELATED WORK

Prior work applies patching to preserve local semantics across modalities and develops Transformer mechanisms for efficient long-term time-series forecasting. Related research also explores self-supervised time-series representation learning, while many forecasting models still rely on point-wise attention that overlooks patches.

  • Patch in Transformer-based Models: Patching is important when local semantic information matters, appearing in subword tokenization for NLP and image splitting in Vision Transformer.The passage identifies BERT and ViT as examples of patch-like modeling across modalities.
  • Transformer-based Long-term Time Series Forecasting: Transformer-based forecasting methods such as LogTrans and Informer introduce sparse or distilled attention mechanisms for efficient long-term prediction.LogTrans uses convolutional self-attention with LogSparse design, while Informer uses ProbSparse self-attention and distilling techniques.
  • Transformer-based Long-term Time Series Forecasting: Most long-term forecasting models reduce the complexity of standard attention, but many retain point-wise attention and therefore ignore the importance of patches.LogTrans avoids point-wise key-query dot products, yet its values remain based on single time steps.
  • Time Series Representation Learning: Self-supervised learning is an important direction for learning useful time-series representations for downstream tasks, alongside numerous non-Transformer approaches.The passage also notes that Transformers are considered promising candidates for foundation models.

3 PROPOSED METHOD

PatchTST forecasts multivariate time series with a vanilla Transformer encoder by independently processing channel-wise univariate series segmented into patches. Instance normalization and masked pre-training extend the method to distribution-shift mitigation and transferable representation learning.

  • Forecasting setup: PatchTST forecasts T future values from a multivariate look-back window of length L using a vanilla Transformer encoder.The input at each time step has dimension M, and the model predicts (xL+1, ..., xL+T).
  • Channel independence: Each of the M channels is split into an independent univariate series and processed with shared Transformer weights.The forward processes are independent across channels while sharing the same backbone.
  • Self-supervised representation learning: For self-supervised learning, PatchTST uses non-overlapping patches, masks selected patches, removes the prediction head, and reconstructs them with a D × P linear layer.Shared weights cross-learn channel-specific latent representations and permit pre-training data with a different number of series from downstream data.
  • Patching: Patching divides each univariate series into possibly overlapping patches, reducing input tokens from L to approximately L/S.The patch length is P, the stride is S, and padding uses S repetitions of the final value before patching.
  • Transformer encoder: Patch embeddings are projected into a D-dimensional latent space, combined with learnable positional encoding, and passed through multi-head attention, BatchNorm, feed-forward, and residual layers.A flatten layer and linear head produce the channel-wise prediction, while the objective averages MSE across channels.
  • Instance normalization: Instance normalization standardizes each series to zero mean and unit standard deviation before patching, then restores its mean and deviation to predictions.The technique is used to mitigate distribution shift between training and testing data.

4 EXPERIMENTS

Experiments across eight datasets show that PatchTST improves supervised long-term forecasting over Transformer and DLinear baselines. Self-supervised pre-training, transfer learning, and ablations further demonstrate strong representation quality and the importance of patching and channel-independence.

  • Datasets and settings: Experiments cover eight benchmark datasets, using FEDformer, Autoformer, Informer, Pyraformer, LogTrans, and DLinear as baselines.Prediction lengths are T ∈{24, 36, 48, 60} for ILI and T ∈{96, 192, 336, 720} otherwise.
  • Supervised forecasting results: 21.0% lower MSE and 16.7% lower MAE are achieved by PatchTST/64 versus the best Transformer-based results, while PatchTST/42 reduces MSE 20.2% and MAE 16.4%.PatchTST/64 uses 64 patches with L = 512; PatchTST/42 uses 42 patches with L = 336, and both use P = 16 and S = 8.
  • Self-supervised learning: Self-supervised pre-training improves forecasting on large datasets over supervised training from scratch, while linear probing is comparable to full training and better than DLinear.The representation experiments use 512-length inputs, patch size 12, 42 non-overlapping patches, 40% masking, and 100 pre-training epochs.
  • Transfer learning: Pre-training on Electricity transfers to other datasets with slightly worse MSE than same-dataset pre-training, yet remains better than other models in forecasting performance.Transfer experiments retrain the linear head or entire model for fewer epochs, reducing computational time.
  • Representation comparisons and ablations: Comparisons on ETTh1 find PatchTST competitive with state-of-the-art self-supervised methods, while ablations show that both patching and channel-independence improve forecasting performance.The ablation study also evaluates running time and memory consumption, and the representation comparison reports improvements ranging from 34.5% to 48.8% across prediction lengths.

5 CONCLUSION AND FUTURE WORK

The paper concludes that patching and channel independence form an effective Transformer design for forecasting, supervised learning, self-supervised representation learning, and transfer learning. It identifies cross-channel dependency modeling and broader use in foundation models as key future directions.

  • Conclusion: Patching and channel independence improve Transformer-based time series forecasting by preserving local semantics and enabling longer look-back windows.The design also outperforms other baselines in supervised learning and shows promise for self-supervised representation learning and transfer learning.
  • Future Work: The model could serve as a basis for future Transformer-based forecasting and as a building block for time series foundation models.Patching is a simple, effective operator that can be transferred to other models.
  • Future Work: Future work should properly model cross-channel dependencies while further exploiting channel independence.The paper identifies incorporating correlations between different channels as an important next step.

A APPENDIX · A.1 EXPERIMENTAL DETAILS · A.1.1 DATASETS

The appendix describes eight multivariate forecasting and representation-learning datasets, including Weather, Traffic, Electricity, and ILI, while treating Exchange-rate separately because financial time series differ in predictability. Exchange-rate forecasting is contextualized against the random-walk benchmark for efficient markets.

  • A.1.1 DATASETS: The study uses 8 popular multivariate datasets for forecasting and representation learning.These datasets are provided in Wu et al. (2021).
  • A.1.1 DATASETS: Weather3 contains 21 meteorological indicators collected in Germany.Examples include humidity and air temperature.
  • A.1.1 DATASETS: Traffic4 records road occupancy rates from sensors on San Francisco freeways.
  • A.1.1 DATASETS: Electricity5 describes hourly electricity consumption from 321 customers.
  • A.1.1 DATASETS: ILI6 collects patient counts and influenza-like illness ratios weekly.
  • A.1.1 DATASETS: Exchange-rate8 contains daily exchange rates for 8 countries.It is mentioned additionally in the original paper.
  • A.1.1 DATASETS: Financial datasets differ from other time-series datasets in properties such as predictability.The passage notes that, under market efficiency, the best prediction for x_t is x_t−1.
  • A.1.1 DATASETS: A random walk without drift is described as the toughest benchmark for exchange-rate forecasting.

A.1.2 DETAILS OF BASELINE SETTINGS

Baseline models use different default look-back windows because Transformer-based models can overfit with long histories, whereas DLinear tends to underfit. The smaller ILI dataset uses separate windows and broader tuning for selected Transformer baselines.

  • Default look-back windows: Transformer-based baselines use L = 96, while DLinear uses L = 336 by default.The difference reflects their differing overfitting and underfitting tendencies and may underestimate baseline performance.
  • Default look-back windows: Transformer-based baselines can overfit with long look-back windows, whereas DLinear tends to underfit.The passage notes that this mismatch in default windows could lead to under-estimation of the baselines.
  • ILI dataset settings: For ILI, Transformer-based models and DLinear use default look-back windows of L = 36 and L = 104, respectively.FEDformer, Autoformer, and Informer are evaluated across six look-back windows, with the best results selected.

A.1.3 BASELINES FROM TRADITIONAL MODELS

Traditional time-series forecasting includes models such as ARIMA, LSTM, TCN, and DeepAR. The passage states that these pre-Transformer models were less effective than Transformer-based models for long-term forecasting.

  • Traditional and pre-Transformer models: ARIMA is identified as a famous traditional time-series model developed before Transformer-based methods.The passage attributes ARIMA to Box and Jenkins (1970).
  • Traditional and pre-Transformer models: Before Transformers, deep-learning models for sequence modeling and forecasting included LSTM, TCN, and DeepAR.The passage cites LSTM, TCN, and DeepAR as examples of models proposed before Transformers appeared.
  • Traditional and pre-Transformer models: The passage states that these traditional and pre-Transformer models were less effective than Transformer-based models for long-term forecasting.This comparison is presented as motivation for Transformer-based forecasting models.

A.1.4 MODEL PARAMETERS

PatchTST uses a default three-layer Transformer encoder with 16 heads and a 128-dimensional latent space, while smaller datasets use reduced parameter sizes. Its feed-forward block expands representations to 256 dimensions before projecting them back to 128.

  • Default configuration: By default, PatchTST uses 3 encoder layers, H = 16 attention heads, and latent dimension D = 128.These are the model’s default parameter settings.
  • Feed-forward network: The feed-forward network uses two linear layers with GELU activation, mapping D = 128 to F = 256 and back to D = 128.The first layer expands the hidden representation, and the second projects it back to the latent dimension.
  • Small-dataset configuration: For the very small ILI, ETTh1, and ETTh2 datasets, PatchTST reduces the configuration to H = 4, D = 16, and F = 128.The reduced parameter size is used to mitigate an issue described in the passage’s truncated text.

A.1.5 IMPLEMENTATION DETAILS … A.7.2 PERFORMANCE OF CHANNEL-INDEPENDENCE ON OTHER MODELS

The appendix details PatchTST’s efficient implementation and evaluates its forecasting behavior across visualization, ablations, self-supervised transfer, robustness, and channel-independence analyses. Across these studies, patching and channel-independence provide strong, robust performance while remaining compatible with standard Transformer implementations and other forecasting models.

  • A.1.5 IMPLEMENTATION DETAILS: PatchTST reshapes patched multivariate batches from B × M × P × N to (B · M) × P × N, allowing processing by any standard Transformer implementation.The implementation supports parallel channel processing without requiring a special operator.
  • A.2 VISUALIZATION: PatchTST provides the best visual forecasting in the reported long-term examples, predicting 192 steps for Weather and Electricity and 60 steps for ILI.The appendix states that its forecasts are best in both scale and bias and capture trends closest to ground truth.
  • A.4.2 VARYING LOOK-BACK WINDOW / A.4.3 PATCHING AND CHANNEL-INDEPENDENCE / A.4.4 INSTANCE NORMALIZATION: Increasing the look-back window generally improves PatchTST/42, while the full ablation finds that combining patching and channel-independence achieves the best results, especially on larger datasets.The ablation compares P+CI, CI-only, patching-only, and the original TST model; instance normalization improves forecasting slightly on two PatchTST variants.
  • A.5.1 FULL BENCHMARK OF MULTIVARIATE FORECASTING / A.5.2 FULL BENCHMARK OF TRANSFER LEARNING: Self-supervised PatchTST is benchmarked on multivariate forecasting and transferred after Electricity pre-training to six other datasets, often fine-tuning with fewer series than used during pre-training.The transfer benchmark is presented as evidence of PatchTST’s transfer-learning capability.
  • A.6.1 RESULTS WITH DIFFERENT RANDOM SEEDS / A.6.2 RESULTS WITH DIFFERENT MODEL PARAMETERS: Across five random seeds, supervised PatchTST shows considerably small score variances, while self-supervised runs have insignificant variance especially on large datasets.The supervised evaluation uses seeds 2019, 2020, 2021, 2022, and 2023; model-parameter experiments vary Transformer layers L = {3, 4, 5} and dimensions D = {128, 256}.
  • A.7 CHANNEL-INDEPENDENCE ANALYSIS / A.7.1 CHANNEL-INDEPENDENCE VS CHANNEL-MIXING / A.7.2 PERFORMANCE OF CHANNEL-INDEPENDENCE ON OTHER MODELS: Channel-independence lets each series learn separate attention patterns, can reduce overfitting, and generally improves Informer, Autoformer, and FEDformer forecasting performance.The appendix attributes channel-mixing’s weaker performance to greater data requirements and overfitting, while channel-independent variants still do not outperform PatchTST.
Loading 2211.14730v2…