PaperlayerView exact revision on arXiv

Source-linked AI summary

Timer: Generative Pre-trained Transformers Are Large Time Series Models

Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, Mingsheng Long

arXiv:2402.02368v3cs.LGstat.ML

TL;DR

Time-series models can deteriorate in data-scarce applications, while large language models suggest benefits from large-scale pre-training. The paper develops Timer using a 1-billion-point dataset, unified S3 sequences, and GPT-style next-token prediction, then applies it across forecasting, imputation, and anomaly detection. Timer achieves state-of-the-art performance with few samples and demonstrates zero-shot forecasting, scalability, and task generality.

  • Problem

    Deep time-series models can deteriorate in real-world data-scarce scenarios, motivating development of large time series models with broader pre-training capabilities.

  • Method

    The paper builds Timer by curating UTSD, converting heterogeneous series into S3 sequences, and using GPT-style autoregressive next-token prediction for unified task adaptation.

  • Results

    Timer achieves state-of-the-art performance with few samples across forecasting, imputation, and anomaly detection, and exhibits zero-shot forecasting capability.

  • Takeaways & Limitations

    Timer provides an early large time-series model that is generalizable, scalable, and applicable across multiple time-series analysis tasks.

  • Takeaways & Limitations

    UTSD is smaller than datasets claiming tens or hundreds of billions of time points, and Timer does not yet support unified classification or probabilistic forecasting.

Abstract

from arXiv · show

Deep learning has contributed remarkably to the advancement of time series analysis. Still, deep models can encounter performance bottlenecks in real-world data-scarce scenarios, which can be concealed due to the performance saturation with small models on current benchmarks. Meanwhile, large models have demonstrated great powers in these scenarios through large-scale pre-training. Continuous progress has been achieved with the emergence of large language models, exhibiting unprecedented abilities such as few-shot generalization, scalability, and task generality, which are however absent in small deep models. To change the status quo of training scenario-specific small models from scratch, this paper aims at the early development of large time series models (LTSM). During pre-training, we curate large-scale datasets with up to 1 billion time points, unify heterogeneous time series into single-series sequence (S3) format, and develop the GPT-style architecture toward LTSMs. To meet diverse application needs, we convert forecasting, imputation, and anomaly detection of time series into a unified generative task. The outcome of this study is a Time Series Transformer (Timer), which is generative pre-trained by next token prediction and adapted to various downstream tasks with promising capabilities as an LTSM. Code and datasets are available at: https://github.com/thuml/Large-Time-Series-Model.

1. Introduction

Real-world time-series applications can become data-scarce, where deep models deteriorate, motivating large time series models inspired by large language models. Timer addresses this direction through large-scale pre-training, unified sequences, GPT-style generation, and broad downstream adaptation.

  • Motivation: Data scarcity can cause state-of-the-art deep time-series models to deteriorate drastically, even on prevalent benchmarks.Figure 1 measures degradation as relative MSE increase compared with full-sample training.
  • Motivation: Large language models motivate LTSMs through few-shot generalization, cross-domain generalization, task versatility, and scalability from larger models and corpora.The paper relates these capabilities to similarities between time series and language in generative modeling and autoregression.
  • Research gap: Prior time-series pre-training remains limited by insufficient dataset infrastructure, heterogeneous-series treatment, and architectures that constrain scale and transfer.These methods typically operate at small scale and focus primarily on in-dataset transfer.
  • Approach: Timer combines UTSD, the single-series sequence format, and a GPT-style next-token objective to pre-train a large Time Series Transformer.The approach converts multivariate series into unified token sequences and uses autoregressive generation.
  • Contributions: Timer supports forecasting, imputation, and anomaly detection through a unified generative approach, achieving state-of-the-art performance with few samples and zero-shot forecasting capability.The paper also evaluates quantitative performance and quality among concurrent large time-series models.

2. Related Work

Sequence pre-training has advanced through masked modeling, contrastive learning, and generative pre-training, but generative approaches remain comparatively underexplored for time series. Existing large time-series efforts span language-model adaptations and large-scale time-series pre-training, while Timer targets broader task coverage and scale.

  • Unsupervised Pre-training on Sequences: Unsupervised sequence pre-training commonly uses masked modeling, contrastive learning, or generative pre-training across language, images, video, and time series.Time-series examples include masked reconstruction and contrastive representation learning.
  • Unsupervised Pre-training on Sequences: Generative pre-training receives relatively less attention in time series despite its use of token-level supervision and unconstrained multi-step generation.Each token is generated from previous context and independently supervised.
  • Large Time Series Models: Research on large time-series models remains nascent, including language-model representation extraction, numerical-token encoding, and prompting for forecasting.FPT, LLMTime, and Time-LLM represent this category of approaches.
  • Large Time Series Models: Large-scale time-series pre-training includes synthetic, domain-specific, probabilistic, decoder-only, and commercial zero-shot forecasting systems.Examples include ForecastFPN, CloudOps, Lag-Llama, PreDcT, and TimeGPT-1.
  • Large Time Series Models: Timer differs by using 1B real-world time points, supporting tasks beyond forecasting, and establishing a zero-shot forecasting benchmark for concurrent LTSMs.Its dataset follows curated processing rather than simple aggregation.

3. Approach

The approach develops large time series models by combining extensive heterogeneous corpora, a unified single-series representation, and decoder-only autoregressive pre-training. This design supports scalable pre-training and iterative generation for time series analysis.

  • Approach: The paper advocates LTSMs using extensive time series corpora, standardized data formats, and autoregressive next-token prediction with a decoder-only Transformer.This transfers the generative pre-training paradigm to heterogeneous time series.
  • Dataset Construction: UTSD organizes seven time series domains into hierarchical datasets containing up to 1 billion time points for scalability research.The curation preserves pattern diversity, balances domain sizes, and increases complexity across hierarchies.
  • Unified Representation: The S3 format converts heterogeneous multivariate series into single-series sequences with a unified context length while preserving variation patterns.It addresses differences in amplitude, frequency, stationarity, variate number, series length, and dataset purpose.
  • Unified Representation: S3 samples fixed-context windows from a pooled collection of normalized single-variate series, allowing sequences from different periods and datasets.The format extends Channel Independence while avoiding its requirement that samples come from the same dataset.
  • Model Design: Timer tokenizes each S3 sequence into consecutive time-point segments, embeds them, and uses causal attention to generate the next token autoregressively.The decoder-only Transformer independently encodes and decodes token embeddings, with optional timestamp embeddings.
  • Model Design: Generative pre-training provides flexibility for unfixed inference context lengths and iterative multi-step generation, while large-scale pre-training can make autoregressive models competitive with direct predictors.The reported experiments compare autoregressively pre-trained models with direct multi-step predictors.

4. Experiments

Timer is evaluated as a unified generative model for forecasting, imputation, and anomaly detection, with experiments examining few-shot transfer, pre-training benefits, scalability, architecture, and zero-shot forecasting. Across these settings, pre-training and scaling improve performance, especially when downstream data are scarce.

  • Experimental setup: Timer unifies forecasting, imputation, and anomaly detection through a generative scheme and is evaluated against task-specific models without downstream pre-training data leakage.The experiments also assess pre-training benefits, model and data scaling, and zero-shot forecasting across concurrent large time series models.
  • Time Series Forecasting: Timer fine-tuning achieves competitive forecasting with 1% of ETTh1, 5% of Traffic, 3% of PEMS03, and 25% of PEMS04 samples.These results are compared with state-of-the-art small forecasters trained on full samples.
  • Time Series Forecasting: Pre-training matches randomly initialized Timer trained on full samples using only 2% of ETTh1, 5% of ECL, 1% of Weather, and 4% of PEMS03 training samples.With all samples available, prediction error decreases from 0.165 →0.154 on Weather, 0.126 →0.118 on PEMS03, and 0.125 →0.107 on PEMS04.
  • Imputation: 100.0%, 86.4%, and 56.8% of 44 imputation scenarios favor Timer at 5%, 20%, and 100% available data, respectively.Pre-training consistently improves imputation with 5% samples and continues to provide notable improvement at 20% and 100%.
  • Scalability: 40.3% and 36.6% prediction-error reductions arise as model size and data scale increase, yielding 0.138 and 0.123 few-shot errors versus a 0.139 full-sample SOTA baseline.Increasing model parameters from 1M to 4M and from 3M to 50M also produces further forecasting improvements, while data-scale gains are steady but relatively smaller.
  • Model Analysis: After pre-training on UTSD-12G, decoder-only Timer achieves the best performance in most downstream scenarios, while Timer, Moirai, and TimesFM rank highest in zero-shot evaluation.The zero-shot evaluation uses seven datasets absent from pre-training; performance and pre-training scale show only a relatively weak positive correlation.

5. Conclusion and Future Work

The paper presents Timer as a generalizable, scalable, task-general large time series model, supported by unified data and sequence processing. Evaluations report state-of-the-art performance and pre-training benefits in data-scarce settings, while identifying directions for future development.

  • Timer combines a 1-billion-time-point dataset, a unified sequence format for heterogeneous multivariate series, and generative pre-training.
  • Timer achieves state-of-the-art performance in forecasting, imputation, and anomaly detection, with notable pre-training benefits when data are scarce.
  • Analysis validates Timer’s scalability and examines the versatility of autoregressive generation across time series tasks.
  • Quality evaluations identify better zero-shot generalization, probabilistic forecasting, and long-context forecasting as future development pathways.

Impact Statement

The paper provides a scalable, unified time series dataset and analyzes its statistical properties across diverse domains and sampling frequencies. Its stated impact is to support pre-training, benchmarking, and future large-model research.

  • The work releases a unified time series dataset for scalable pre-training and establishing new benchmarks.
  • UTSD spans ten source domains and sampling frequencies from yearly and quarterly intervals to hourly and minutely intervals.
  • Dataset analysis uses stationarity and forecastability to characterize complexity and filter four high-quality UTSD subsets.
  • The length-weighted ADF statistic weights each series’ contribution according to its length when estimating dataset stationarity.
  • Forecastability is derived from Fourier-decomposition entropy, with higher values indicating superior predictability, and is length-weighted across datasets.

A.3. UTSD Composition Analysis

UTSD is organized hierarchically by dataset complexity and pattern diversity. Larger subsets incorporate more challenging periods while preserving broad representation of patterns across datasets.

  • Dataset Complexity: UTSD-1G, UTSD-2G, UTSD-4G, and UTSD-12G are nested hierarchies built with progressively greater complexity and pattern diversity.
  • Dataset Complexity: ADF statistics classify predictive difficulty into Easy, Medium, and Hard levels.
  • Dataset Complexity: As UTSD capacity increases, periods with low forecastability are added, increasing the proportion of challenging tasks.
  • Pattern Diversity: Each hierarchy is designed to represent individual datasets comprehensively while retaining as many patterns as possible.
  • Pattern Diversity: Representative UTSD series are decomposed into trend, seasonal, and residual components for pattern visualization.
  • Pattern Diversity: Downsampling, representative-variate selection, and period resampling reduce trivial temporal variation while preserving distinctive dataset patterns.

A.4. Experiments

The experiments discuss benchmark coverage, domain transfer, and the limitations of existing forecasting datasets. They emphasize the need for more diverse benchmarks and show that transfer benefits depend on domain and downstream data volume.

  • Forecasting benchmarks: Classical forecasting benchmarks are widely used, but some have relatively homogeneous variability and insufficient coverage of non-stationarity and predictability.
  • Forecasting benchmarks: The paper calls for more diverse and comprehensive forecasting benchmarks in future work.
  • Benchmark descriptions: Benchmark descriptions report time points, sampling frequency, ADF statistics, and forecastability for evaluated datasets.
  • Domain transfer: In-domain transfer can improve downstream performance, while increasing downstream samples reduces pre-training’s relative improvement and may produce negative transfer.

B.1. Pre-training

Timer is pre-trained on hierarchical UTSD datasets in the unified S3 format, with increasing data sizes and model parameters used to study scalability.

  • B.1. Pre-training: Timer is pre-trained on UTSD datasets of different sizes and difficulties represented in the unified S3 format.The study increases both pre-training data and model parameters to validate scalability.
  • B.1. Pre-training: The model configurations fix the number of attention heads at 8 while varying other configuration parameters and total parameter counts.Table 6 provides the detailed configurations and corresponding parameter counts.
  • B.1. Pre-training: Pre-training uses AdamW optimization with cosine-annealed learning rates over 10 epochs and a batch size of 8192.The learning rate decays from 5 × 10^-5 to 2 × 10^-6.
  • B.1. Pre-training: UTSD-1G through UTSD-4G use global shuffling, whereas UTSD-12G uses local shuffling because the full dataset cannot fit in memory.The 12GB dataset is divided into three 4GB subsets for storage-based shuffling.

B.2. Downstream Tasks

Timer is adapted through generative formulations for forecasting, imputation, and anomaly detection, including data-scarce and zero-shot settings across diverse time-series datasets.

  • B.2. Downstream Tasks: Downstream experiments cover forecasting, imputation, and anomaly detection using task-specific generative schemes.The downstream configurations are listed in Table 8, with detailed results in Section C.
  • Forecasting: Forecasting evaluation includes ETT, ECL, and Traffic datasets covering power transformers, electricity consumption, and road occupancy.The datasets contain multivariate real-world measurements sampled at different intervals.
  • Forecasting: Forecasting uses autoregressive next-token prediction with seven input segments of length 96 and iterative multi-step inference.The model is fine-tuned by minimizing MSE across the generated tokens.
  • Imputation: Imputation masks contiguous time-series segments rather than individual points, requiring the model to restore successive missing variations.The protocol uses eight segments of length 24 and denoising autoencoding.
  • Anomaly detection: Anomaly detection predicts future segments from observed segments and compares them with received values to identify anomalies online.The UCR Anomaly Archive setting uses one normal training series, making it an extremely data-scarce task.
  • Results: Timer achieves state-of-the-art forecasting performance while large-scale pre-training alleviates degradation as downstream samples decrease.The forecasting results cover six representative real-world datasets.
  • Results: Across all 250 UCR Anomaly Archive datasets, pre-trained Timer detects anomalies with smaller α on most datasets.The evaluation compares Timer trained from scratch with the pre-trained model.

C.4. Scalability

Timer’s scalability is evaluated by increasing model and data size, while zero-shot results show strong but still uneven performance across large time-series models.

  • C.4. Scalability: Increasing Timer’s model size and pre-training data supports scalability of the GPT-style decoder-only architecture toward large time-series models.The evaluation uses PEMS forecasting subsets and reports results in Table 16 and Figures 10–11.
  • C.5. Zero-shot forecasting: Zero-shot forecasting ranks Timer, Moirai, and TimesFM among the leading large time-series models on seven datasets excluded from their pre-training corpora.Chronos improves with more sampled trajectories, while some models fail at multi-step prediction.
  • C.5. Zero-shot forecasting: Zero-shot scaling behavior is not evident on some datasets, indicating that zero-shot large time-series models remain at an early development stage.The cited evaluation also notes failures of multi-step prediction in some models.
  • C.4. Scalability: The evaluation includes visualizations of forecasting, imputation, and anomaly detection, with forecasting and imputation shown at different sample ratios.Anomaly visualizations show the anomaly position and Timer’s generated normal series.
  • C.4. Scalability: The imputation table defines pre-training benefit as the percentage reduction in MSE error and reports comparisons at 5% samples.At 5% samples, pre-trained Timer outperforms TimesNet in all 44 settings.

E. Limitations

Timer is an early large time-series model whose scope remains bounded by dataset scale and unsupported capabilities.

  • E. Limitations: UTSD is not yet large enough to match recent pre-training efforts involving tens or hundreds of billions of time points.The authors advocate continued expansion of high-quality, hierarchical data infrastructure.
  • E. Limitations: Timer does not yet unify time-series classification, support probabilistic forecasting, or specially adapt to multiple variables.These are explicit method-level scope boundaries despite reported generalization, scalability, and task generality.
  • E. Limitations: Better zero-shot generalization, in-context learning, and multi-modality remain future development targets requiring ever-larger pre-training.The paper characterizes the work as an early but important development of large models.

F. Societal Impacts

The paper presents Timer and its curated pre-training dataset as resources for scalable time-series research and data-scarce applications. Reported downstream evaluations and visualizations support the model’s versatility across forecasting, imputation, and anomaly detection.

  • Real-world applications: Timer is presented as a general-purpose model for data-scarce time-series analysis, with potential applications supporting risk prevention and decision-making from limited samples.The paper states that its focus is scientific research and reports no obvious negative social impact.
  • Academic research: The paper releases a curated dataset for scalable pre-training and presents Timer’s generalization and versatility across many tasks.The authors state that generative pre-training and autoregression can inform future research on scalable time-series architectures and pre-training techniques.
  • Real-world applications: Timer’s pre-trained model outperforms TimesNet in 86.4% of 44 imputation settings with 20% samples.The pre-training benefit is measured by the ratio of decreased imputing error in MSE.
  • Real-world applications: Timer’s pre-trained model outperforms TimesNet in 56.8% of 44 imputation settings with 100% samples.The reported pre-training benefit is calculated from decreased imputing error in MSE.
  • Real-world applications: Timer’s anomaly detection visualizations show abnormal intervals located through generated results that deviate from abnormal series.The masked part represents the abnormal position in the visualization.
Loading 2402.02368v3…