PaperlayerView exact revision on arXiv

Source-linked AI summary

One Fits All:Power General Time Series Analysis by Pretrained LM

Tian Zhou, PeiSong Niu, Xue Wang, Liang Sun, Rong Jin

arXiv:2302.11939v6cs.LGcs.AI

TL;DR

General time series analysis lacks the large datasets and unified pretrained models available in NLP and CV. The paper transfers pretrained language or vision transformers through a Frozen Pretrained Transformer, retaining self-attention and feedforward layers while fine-tuning interface components. Across major time series tasks, the approach achieves comparable or state-of-the-art performance, while analysis links self-attention to PCA.

  • Problem

    General time series analysis has limited progress toward unified pretrained models, partly because its largest datasets are under 10GB.

  • Method

    The Frozen Pretrained Transformer transfers pretrained language or vision models to time series while freezing self-attention and feedforward layers and fine-tuning selected components.

  • Results

    The same pretrained-transformer approach performs on par or better than state-of-the-art methods across almost all major time series tasks.

  • Takeaways & Limitations

    Self-attention behaves similarly to PCA, providing a proposed explanation for transformer universality across time series and other modalities.

  • Takeaways & Limitations

    Zero-shot performance remains behind N-beat on several datasets, and the analysis of transformer generality remains at an early stage.

Abstract

from arXiv · show

Although we have witnessed great success of pre-trained models in natural language processing (NLP) and computer vision (CV), limited progress has been made for general time series analysis. Unlike NLP and CV where a unified model can be used to perform different tasks, specially designed approach still dominates in each time series analysis task such as classification, anomaly detection, forecasting, and few-shot learning. The main challenge that blocks the development of pre-trained model for time series analysis is the lack of a large amount of data for training. In this work, we address this challenge by leveraging language or CV models, pre-trained from billions of tokens, for time series analysis. Specifically, we refrain from altering the self-attention and feedforward layers of the residual blocks in the pre-trained language or image model. This model, known as the Frozen Pretrained Transformer (FPT), is evaluated through fine-tuning on all major types of tasks involving time series. Our results demonstrate that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks, as illustrated in Figure 1. We also found both theoretically and empirically that the self-attention module behaviors similarly to principle component analysis (PCA), an observation that helps explains how transformer bridges the domain gap and a crucial step towards understanding the universality of a pre-trained transformer.The code is publicly available at https://github.com/DAMO-DI-ML/One_Fits_All.

1 Introduction

Time series analysis spans many applications, but unlike NLP and CV it has seen limited progress toward unified pretrained models, partly because available datasets are small. The paper proposes transferring pretrained models to diverse time series tasks and reports comparable or better performance across them.

  • Time series analysis supports applications including forecasting, imputation, anomaly detection, and classification.
  • Foundation models offer unified frameworks for diverse tasks, but time series analysis has made limited use of them because its largest datasets are under 10GB.
  • The proposed framework uses a pretrained language model across classification, anomaly detection, forecasting, and few-shot or zero-shot learning.
  • The same backbone performs on-par or better than state-of-the-art methods across the main time series analysis tasks.
  • The paper also connects self-attention with PCA to investigate why pretrained transformers can transfer across domains.

2 Related Work

Prior work established pretrained transformers and cross-modality transfer across NLP, vision, language, and speech. However, cross-modality transfer had not been investigated for general time series forecasting or analysis.

  • Pretrained models have demonstrated effective transfer across NLP, computer vision, and vision-and-language tasks.
  • Transformers support cross-domain transfer by tokenizing inputs from different modalities into embeddings.
  • Earlier work transferred frozen language models to other domains and pretrained speech models to time series classification.
  • Before this paper, cross-modality knowledge transfer had not been studied for time series forecasting or general time series analysis.

3 Methodology

The method transfers pretrained transformer components to time series by adapting input and normalization interfaces while freezing most learned transformer blocks. It also uses patching to represent longer histories with fewer tokens.

  • The architecture uses pretrained GPT2 parameters for time series analysis and tests BERT and BEiT to assess transfer across model structures and modalities.
  • Self-attention and feedforward blocks remain frozen, while embedding, normalization, and output components are retrained during fine-tuning.
  • A redesigned input embedding projects time-series data into the dimensions required by the pretrained model using linear probing.
  • Reverse instance normalization normalizes the input using its mean and variance, then adds those statistics back to the output.
  • Patching aggregates adjacent time steps into tokens, increasing the historical horizon while maintaining token length and reducing redundancy.

4 Main Time Series Analysis Tasks

The GPT2-backbone Frozen Pretrained Transformer is evaluated across classification, anomaly detection, imputation, and forecasting settings, including short-, long-term, few-shot, and cross-dataset evaluation. Across these tasks, it generally matches or outperforms established time-series baselines.

  • 4.1 Main Results: GPT2-backbone FPT outperforms other models in most evaluated tasks, spanning forecasting, classification, anomaly detection, imputation, and few-shot or zero-shot forecasting.The evaluation follows experimental settings aligned with TimesNet and compares against representative CNN-, MLP-, and Transformer-based baselines.
  • 4.3 Imputation: GPT2(3) FPT achieves the best performance on most imputation datasets, with a relative 4.1% average MSE reduction versus TimesNet across six benchmarks.On ETTh1 specifically, the reported relative MSE reduction is 11.5%.
  • 4.3 Time Series Classification: GPT2(6) FPT achieves 74.00% average classification accuracy, exceeding TimesNet’s 73.60% and a patch-transformer baseline by 9.0%.Classification results are averaged over 10 multivariate UEA datasets.
  • 4.4 Time Series Anomaly Detection: GPT2(6) FPT obtains an averaged anomaly-detection F1-score of 86.72%, surpassing TimesNet by 1.7%.The comparison uses classical reconstruction error for the baseline models and covers five commonly used datasets.
  • 4.5 Long-term Forecasting: For long-term forecasting, GPT2(6) FPT is comparable to PatchTST and yields a relative 9.3% average MSE reduction versus TimesNet.The evaluation uses eight real-world benchmark datasets and averages results across four prediction lengths.
  • 4.6 Short-term Forecasting: GPT2-backbone FPT is superior to advanced Transformer- and MLP-based models on M4 short-term forecasting and comparable to TimesNet and N-BEATS.The M4 results are weighted averages across datasets with different sampling intervals and prediction lengths from 6 to 48.
  • 4.7 Few-shot Forecasting: With 10% of training timesteps, GPT2(6) FPT achieves the best few-shot forecasting performance, reducing average MSE relative to TimesNet by 33.3% and DLinear by 13.5%.It also surpasses the traditional ETS, ARIMA, and NaiveDrift methods in the reported comparison.
  • 4.8 Zero-shot Forecasting: In cross-dataset forecasting, GPT2(6) FPT consistently outperforms recent Transformer- and MLP-based methods, with relative average metric reductions of 13.1%, 13.6%, and 7.3% versus DLinear, TimesNet, and PatchTST.It is comparable to N-BEATS and outperforms N-BEATS on the ELECTR dataset.

5 Ablations

The ablations examine model depth, freezing, and pre-training, showing that GPT2(6) with partially frozen pre-trained parameters is the strongest supported configuration.

  • Model Selection: GPT2 with 6 layers is a sound choice compared with using full or few layers.The authors report that the results appear in Appendix H.
  • Model Selection: Partially freezing parameters can avoid catastrophic forgetting and enable fine-tuning without overfitting.
  • Effectiveness of Pre-training: GPT2(6) FPT outperforms GPT2(0) FPT and GPT2-random-initialized in the ablation study.The comparison is reported for results shown in Table 7.
  • Effectiveness of Pre-training: GPT2(6) FPT performs better than GPT2unfrozen, demonstrating that partial freezing helps.
  • Effectiveness of Pre-training: Random-initialized GPT2(6) with freezing performs poorly, indicating that pre-trained knowledge is instrumental for time series tasks.

6 Exploring Transfer Learning from others: The Unexceptional Nature of GPT2-based-FPT

Transfer experiments test whether FPT's effectiveness depends specifically on GPT2-based language pre-training. BERT- and image-pretrained BEiT-backbone variants also demonstrate cross-domain knowledge transfer.

  • Transfer Learning: BERT-backbone FPT and image-pretrained BEiT-backbone FPT are evaluated to illustrate cross-domain knowledge transfer.
  • Transfer Learning: The reported transfer ability is not exclusive to GPT2-based pre-trained language models.The paper presents this result as evidence for the generality of pre-trained models across domains.
  • Transfer Learning: Table 8 reports frozen pretrained transformer results on 5% ETTh2 and ETTm2, averaged across four prediction lengths.

7 Training/Inferencing Cost

The cost analysis compares training and inference efficiency with baseline models. GPT-2(3) uses fewer parameters and is more time-efficient than same-dimension baselines, while GPT-2(3) and GPT-2(6) train only small parameter fractions.

  • Computational Cost: The computational-cost comparison evaluates baseline variants with hidden dimensions 32 and 768, mostly using three layers.Costs are measured on an ETTh2 batch of size 128 using a 32G V100 GPU.
  • Computational Cost: GPT-2(3) substantially improves time efficiency and reduces parameter quantity compared with baselines sharing its model dimension.
  • Computational Cost: GPT-2(3) and GPT-2(6) have only 6.12% and 4.60% learnable parameters, respectively, of their overall parameter sizes.

8 Towards Understanding the Universality of Transformer: Connecting Self-Attention with PCA

The analysis connects self-attention with PCA through token-similarity experiments and gradient-based theory. It argues that minimizing the self-attention gradient yields a function closely related to PCA, helping explain transformer universality.

  • Empirical Analysis: Pairwise token similarity is computed from cosine similarities among output features shaped as (b, n, d), producing a (b, n, n) similarity matrix.Similarity values are then counted within intervals for statistical analysis.
  • Empirical Analysis: The analysis is motivated by increasing within-layer token similarity in deeper transformer layers.The authors examine this behavior using layer-wise average token cosine similarity on ETTh2.
  • Empirical Analysis: Mixing pre-trained GPT2 weights with random Gaussian weights reveals low similarity in randomly initialized models and increased similarity in deeper layers as pre-trained weights are restored.The random-initialized GPT2(6) similarity remains around 0.1-0.2 across layers.
  • Theoretical Connection: The theoretical analysis defines self-attention as a function of input patterns and examines its gradient structure to connect it with PCA.
  • Theoretical Connection: Theorem 1 identifies the optimal solution through the leading eigenvectors of X⊤X, while gradient minimization makes self-attention closely related to PCA.

9 Conclusions

The paper develops a foundation model for time series analysis by adapting pretrained NLP or CV models into a unified framework for diverse tasks. Experiments report performance on par with or better than state-of-the-art approaches on almost all tasks, while zero-shot forecasting remains behind N-BEATS on several datasets.

  • Conclusion: The proposed foundation model uses pretrained NLP or CV models to facilitate downstream training and unify diverse time series analysis tasks.The framework covers classification, anomaly detection, imputation, short- and long-term forecasting, and few-shot and zero-shot forecasting.
  • Conclusion: The method performs on par with or better than state-of-the-art approaches on almost all evaluated time series tasks.The evaluation spans the major task types examined in the paper.
  • Conclusion: GPT2(6) can accurately perform few-shot forecasting, whereas TimesNet and DLinear fail in the showcased task.Figure 5 presents examples of imputation, long-term forecasting, and few-shot forecasting.
  • Related work: The paper positions its unified approach against conventional task-specific time series methods and surveys forecasting methods ranging from ARIMA to transformers.The related-work discussion describes forecasting categories and representative transformer approaches, including Informer, Autoformer, FEDformer, and PatchTST.
  • Experimental scope: The experiments use datasets spanning long- and short-term forecasting, few-shot learning, and zero-shot learning, with task-specific error metrics.The paper reports MSE and MAE for several forecasting settings and MAPE, sMAPE, or ND for zero-shot learning.

D.2 Detailed Definition and Results for Few-shot and Long-term Forecasting

The few-shot and long-term forecasting evaluations treat multivariate series as independent univariate sequences and compare GPT2(6) FPT with neural and traditional baselines. GPT2(6) achieves strong comparative performance, with reported stability across runs, while zero-shot results are superior to DLinear in most situations and comparable to N-BEATS.

  • Few-shot forecasting: The experiments treat each multivariate series as multiple independent univariate series and use only 5% or 10% of training timesteps for few-shot forecasting.Training, validation, and test data are separated, while the validation and test portions remain unchanged in the few-shot setting.
  • Long-term forecasting: GPT2(6) FPT achieves comparable performance to PatchTST and DLinear and outperforms other baselines by a large margin on full-data forecasting.Against FEDformer, GPT2(6) FPT yields an overall 18.7% relative MSE reduction.
  • Few-shot forecasting: The GPT2(6) results show small variance across three runs on 5% ETTh2 and ETTm2, indicating stability in that evaluation.Table 15 reports both means and standard deviations for GPT2(6), DLinear, and PatchTST.
  • Few-shot forecasting: GPT2(6) achieves the best performance in the reported few-shot comparison with traditional methods.The comparison is presented for few-shot learning in Table 16.
  • Few-shot forecasting: With instance normalization included for comparison, GPT(6) also performs superior to the evaluated baselines.The additional comparison equips Autoformer and FEDformer with instance normalization.
  • Zero-shot forecasting: In zero-shot forecasting, GPT2(6) outperforms DLinear in most situations and is comparable to the meta-learning-based N-BEATS without using test-data information.The zero-shot evaluation reports results across diverse frequencies in Tables 18–20.
  • Representation analysis: Pretrained GPT2-FPT shows higher within-sample token similarity than a fine-tuned randomly initialized GPT2(6) model, especially in deeper layers.The analysis measures pairwise cosine similarity among token features and compares pretrained, non-pretrained, and PCA-replaced attention variants.
  • Theoretical analysis: Under the stated assumptions, self-attention outputs converge toward the mean value token µWv at order O(n^-1/2).The analysis connects this behavior to identifying invariant structures among adjacent time-series patches, particularly in few-shot forecasting.

E.2 Theorem E.4

Theorem E.4 analyzes learning an MLP mapping from feature maps to forecasting targets when the feature-map covariance is positive definite. Under this condition, stochastic gradient descent reaches a target optimization tolerance in a number of steps proportional to O(σ^-1), while widely spread feature maps are linked to forecasting behavior and learning speed.

  • Theorem E.4: Theorem E.4 considers learning a matrix W that maps feature-map vectors to forecasting targets under a positive-definite covariance condition.The feature maps are assumed to span the feature space, with covariance bounded below by σI for σ > 0.
  • Empirical interpretation: GPT2-FPT produces feature maps that are more widely spread than PatchTST feature maps in the reported t-SNE visualization.Samples chunked from one time series are colored consistently, and the same t-SNE configuration is applied to both models.
  • Theorem E.4: O(σ^-1) optimization steps suffice to reach a fixed tolerance when stochastic gradient descent uses diminishing step sizes.The more explicit rate is t = ˜O(σ^-1ϵ^-1) for reaching an ϵ optimization gap.
  • Theorem E.4: Under the positive-definite condition, the optimization problem is strongly convex with a parameter proportional to σ.This converts learning the final MLP layer into a well-conditioned least-squares regression problem.
  • Empirical interpretation: The paper links diverse pretrained feature maps to faster learning of the final MLP layer in few-shot forecasting.It also discusses wider feature-map spread as potentially increasing decoder memorization capacity for extreme or rare behavior.

F N-gram Explanation for Universality

The paper explains cross-domain universality by viewing patched time series as token sequences and pretrained transformer attention as a sophisticated, potentially universal form of n-gram estimation. It also shows theoretically that self-attention aligns with principal directions of the input, resembling PCA.

  • N-gram interpretation: Patching represents time-series subsequences as tokens, allowing forecasting to be framed as an n-gram estimation problem similar to text analysis.The paper connects sequential prediction with n-order Markov processes and n-gram language models.
  • Transformer interpretation: A zero-layer transformer resembles a bigram model, while an attention-only transformer combines bigram and skip-trigram behavior.These simplified cases motivate the broader n-gram interpretation of transformer sequence modeling.
  • N-gram interpretation: For sequential data whose n-gram frequencies follow a power law, maximum-entropy estimation can yield a solution that is independent of detailed pattern statistics.The analysis groups high- and low-frequency n-grams and concludes that the solution can be universal under skewed distributions.
  • N-gram interpretation: The paper informally interprets transformer attention as performing sophisticated n-gram estimation, providing a proposed explanation for cross-domain knowledge transfer.This account links the generation rule for n-gram distributions to the universality of pretrained language-model performance.
  • Scope of explanation: The paper presents this universality explanation as informal in part and notes that no concrete metric for cross-domain knowledge transfer is currently available.The authors identify quantitative evaluation of transfer between domains as requiring further investigation.
  • Self-attention and PCA: Minimizing the self-attention objective selects WQ and WK aligned with the principal directions of X⊤X, so self-attention behaves similarly to PCA.The theorem states that the optimum contains the largest m eigenvectors of X⊤X, where m is the rank of A.

H.1 Experiment analysis of GPT2-FPT model

Experiments compare GPT2-FPT with unfrozen and randomly initialized variants, examine layer and fine-tuning choices, and test data efficiency and other pretrained backbones. The results support frozen pretrained attention, strong few-shot performance, and comparable performance from BERT- and BEiT-based variants.

  • Pretraining and freezing: GPT2(6) FPT achieves better performance on most datasets than fine-tuning all parameters, with 12.7% and 11.5% relative MSE reductions using 5% and 10% data.These reductions are reported overall across the evaluated datasets.
  • Pretraining and freezing: GPT2(6) FPT achieves better performance on most datasets than random initialization, with 21.2% and 14.3% relative MSE reductions using 5% and 10% data.The result supports the contribution of cross-domain pretrained knowledge in forecasting.
  • Layer selection: GPT2 with 6 layers is selected as the default architecture because pretrained models perform better between 3 and 9 layers while balancing performance and computational efficiency.The experiments also consider overfitting risks from deeper pretrained models.
  • Pretraining and freezing: Freezing a randomly initialized GPT2 does not work when only input and output modules are trained, indicating that pretrained knowledge is important for time-series tasks.This comparison separates the effect of freezing from the effect of pretrained initialization.
  • Data volume: GPT2(6) FPT reaches SOTA performance with 5% and 10% of the training data and remains comparable to PatchTST and DLinear with full samples.Its performance improvement is reported as nearly flat across increasing percentages of ETTh2 data, while end-to-end models catch up with more data.
  • Other pretrained backbones: BERT(6) FPT and BEiT(6) FPT are comparable to PatchTST and remarkably surpass other baselines on 5% ETTh2 and ETTm2.The result extends the cross-domain transfer observation beyond GPT2 to NLP and computer-vision pretrained models.
  • Input length: The study discusses longer inputs as an open practical direction, recommending joint tuning of input length and model parameters rather than fixing a small length.The authors identify extremely long inputs as a future research direction.
Loading 2302.11939v6…