Source-linked AI summary
Foundation Models for Time Series Analysis: A Tutorial and Survey
Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, Qingsong Wen
TL;DR
Time-series foundation models have advanced time-series analysis, but prior surveys offered limited methodological explanation of why and how they benefit the field. This survey synthesizes models through data modality, architecture, pre-training, and adaptation, covering diverse time-series types and future directions. Its central contribution is a comprehensive methodology-centric taxonomy intended to clarify the mechanisms underlying TSFM development and use.
Problem
Prior surveys focused mainly on data or pipeline perspectives and lacked a systematic methodological analysis of TSFMs explaining their underlying mechanisms.
Method
The survey classifies and reviews TSFMs by data modality, model architecture, pre-training technique, and adaptation method.
Results
The survey provides a comprehensive and up-to-date review spanning standard, spatial, trajectory, and event time series, together with a methodology-centric taxonomy.
Takeaways & Limitations
The taxonomy consolidates recent TSFM advances and supports understanding of the mechanisms behind their application to time-series analysis.
Takeaways & Limitations
Privacy-preserved fine-tuning is noted as a constraint-aware strategy within TSFM adaptation.
Abstract
from arXiv · showhide
Time series analysis stands as a focal point within the data mining community, serving as a cornerstone for extracting valuable insights crucial to a myriad of real-world applications. Recent advances in Foundation Models (FMs) have fundamentally reshaped the paradigm of model design for time series analysis, boosting various downstream tasks in practice. These innovative approaches often leverage pre-trained or fine-tuned FMs to harness generalized knowledge tailored for time series analysis. This survey aims to furnish a comprehensive and up-to-date overview of FMs for time series analysis. While prior surveys have predominantly focused on either application or pipeline aspects of FMs in time series analysis, they have often lacked an in-depth understanding of the underlying mechanisms that elucidate why and how FMs benefit time series analysis. To address this gap, our survey adopts a methodology-centric classification, delineating various pivotal elements of time-series FMs, including model architectures, pre-training techniques, adaptation methods, and data modalities. Overall, this survey serves to consolidate the latest advancements in FMs pertinent to time series analysis, accentuating their theoretical underpinnings, recent strides in development, and avenues for future exploration.
1 INTRODUCTION
Time Series Foundation Models are presented as a promising framework for generalized time-series understanding and forecasting, but prior surveys have not systematically explained their methodological foundations. This survey addresses that gap through a comprehensive, methodology-centric review spanning data types, architectures, pre-training, adaptation, and future directions.
- Time series analysis supports insights and decisions across finance, healthcare, cloud computing, environmental, energy, and urban-computing applications.
- TSFMs aim to build generalized models for understanding and forecasting time series across diverse domains using large-scale datasets.
- Prior surveys mainly examined data or pipeline perspectives, leaving a methodological analysis of TSFMs notably absent.
- The survey analyzes TSFMs through model architectures, pre-training techniques, adaptation methods, and data modalities.
- The survey covers standard, spatial, trajectory, and event time series while consolidating recent advances and future research opportunities.
2 BACKGROUND
The background defines foundation models and frames their application to time series through four methodological components. It also introduces standard, spatial, trajectory, and event time-series data types.
- Foundation models are deeply pretrained on vast datasets and can be adapted to specific tasks with relatively small amounts of task-specific data.
- The survey studies data modality, architecture, pre-training, and adaptation as the core components of time-series foundation models.
- Time-Series Definitions: A standard time series is a time-ordered sequence X ∈ R^T×D, with D=1 denoting univariate and D>1 multivariate data.
- Time-Series Definitions: Spatial time series combine temporal and spatial dimensions, with sensor relationships represented as graphs or geographically uniform rasters.
- Time-Series Definitions: Trajectories are timestamped locations, whereas event sequences are temporally ordered events paired with occurrence timestamps.
3 TAXONOMY
The proposed taxonomy classifies time-series foundation models from a methodology-centric perspective. It organizes the field by data categories and methodological choices to clarify how and why these models may work.
- The taxonomy uses four hierarchical levels: data category, model architecture, pre-training techniques, and application domain.
- Its methodology-centric view examines architectural designs, pre-training, and adaptation rather than relying only on application-oriented classification.
- Data Categories: Time-series data are divided into standard, spatial, and other categories containing trajectory and event data.
- Model Architectures: Model architectures are grouped into Transformer-based, non-Transformer-based, and diffusion-based models.
4 DATA PERSPECTIVE
The survey reviews time-series foundation models across standard, spatial, and other temporal data from task-oriented and general-purpose perspectives. It compares methodological choices including architectures, modalities, pre-training, and adaptation across forecasting, classification, climate, mobility, and related applications.
- Standard Time Series: Standard time series vary in sampling rates and temporal patterns, motivating models that identify universal patterns across extensive and heterogeneous sources.
- Standard Time Series: Task-oriented standard TSFMs commonly target forecasting or classification using single or multiple modalities and may be trained from scratch or adapted from other domains.
- Standard Time Series: Representative standard TSFMs include Lag-Llama, TimeGPT-1, TTMs, TimesFM, Moirai, and diffusion-based models such as TimeGrad and TransFusion.
- Standard Time Series: Alternative standard-series approaches adapt pretrained language, vision, or acoustic models, including GPT-2-based forecasting, direct LLM inference, and multimodal reprogramming.
- Standard Time Series: General-purpose standard TSFMs use large-scale pretraining for representation or broad analysis, with examples including TS2Vec, SimMTM, Timer, UniTS, OFA, and TEST.
- Spatial Time Series: Spatial time-series research remains relatively early and domain-specific, covering spatiotemporal graphs and rasters for tasks such as forecasting and classification.
- Spatial Time Series: Spatiotemporal graph models include task-oriented transportation systems and emerging domain-agnostic approaches using graph networks, transformers, or diffusion.
- Spatial Time Series: Climate foundation models operate on spatiotemporal rasters, with examples supporting weather forecasting, climate tasks, and fine-tuning across variables and scales.
5 METHODOLOGY PERSPECTIVE
The survey examines time-series foundation models from a methodology perspective, focusing on architecture and pipeline intricacies to explain their efficacy and adaptability.
- The methodology perspective analyzes TSFMs through architecture and pipeline components.The pipeline discussion includes pre-training and adaptation.
5.1 Architecture
TSFM architectures include Transformer-based, non-Transformer-based, and diffusion-based models, with designs targeting temporal, spatial, and broader temporal-data dependencies.
- Transformer-based Models: Transformers use attention to capture global, long-range dependencies and support parallelized processing.Attention uses queries, keys, values, and a scaling factor to moderate dot-product magnitudes.
- Transformer-based Models: Time-series foundation models include encoder-only, encoder-decoder, and decoder-only frameworks, whose suitability varies with dataset scale and modeling goals.Encoder-only models are favored for forecasting on small datasets, while decoder-only models may suit large-scale models because of their generalization and capacity.
- Transformer-based Models: Transformer-based TSFMs model temporal dynamics using pretrained language models or time-series-specific Transformers trained from scratch.Common enhancements include patching, reversible instance normalization, multi-resolution analysis, and decomposition strategies.
- Transformer-based Models: For spatial time series, attention can jointly model spatial-temporal dependencies or combine temporal Transformers with graph-based spatial models.Examples include spatial-temporal embeddings, dynamic graph encoders, graph structure learners, and spatio-temporal GNNs.
- Transformer-based Models: Transformers have been applied beyond conventional series to trajectory and healthcare records, demonstrating versatility across temporal datasets.
- Non-Transformer-based and Diffusion-based Models: Non-Transformer backbones include MLPs, RNNs, and CNNs, while diffusion models learn data distributions by progressively adding and reversing noise.CNN approaches commonly use one-dimensional convolutions; RNN-based models emphasize resource efficiency and linear-complexity processing of long sequences.
- Diffusion-based Models: Diffusion models predict future states by capturing temporal dynamics and can additionally model spatial correlations in spatial time series.
5.2 Pipeline
The TSFM pipeline covers pre-training and adaptation: pre-training acquires generalizable knowledge, while adaptation tailors models to tasks or datasets through four branches.
- Pre-training: Pre-training enables TSFMs to generalize across contexts and adapt to downstream tasks with minimal adaptation.The survey categorizes pre-training mainly by learning objectives and includes supervised, self-supervised, and other mechanisms.
- Pre-training: Fully supervised pre-training learns generalizable representations from labeled datasets and is suited to settings with sufficient historical labels.It is frequently used for domain-specific transportation and climate forecasting applications.
- Pre-training: Generative self-supervised pre-training learns time-series representations through reconstruction or probabilistic modeling.Reconstruction can use masked autoencoding, while probabilistic methods optimize latent representations toward estimated densities for forecast sampling.
- Pre-training: Self-supervised pre-training exploits unlabeled time-series data to provide generic temporal knowledge for downstream fine-tuning.The survey describes it as more generic and realistic than fully supervised pre-training for acquiring TSFMs.
- Pre-training: TSFMs may also use models pretrained on other modalities, including text, image-text sequences, and acoustic signals.
- Adaptation: Adaptation methods comprise direct usage, fine-tuning, prompt engineering, and time-series tokenization.These branches tailor a TSFM to specific tasks or datasets using learned generic temporal knowledge.
- Adaptation: Direct usage applies a pretrained model without further target-dataset fine-tuning, whereas fine-tuning updates the whole model, selected components, or a larger model system.
- Adaptation: Prompt engineering uses handcrafted textual prompts, intermediate embeddings, or trainable parameterized vectors for downstream prediction and feature enhancement.
5.3 Modality
TSFMs use single-modal or multimodal data, with single-modal approaches offering simplicity and strong empirical results while multimodal methods add information for challenging domains.
- Data Modalities: The survey reviews modalities used during TSFM pre-training and adaptation, including standard time series, trajectories, raster data, and text.
- Data Modalities: Most current TSFMs use single-modal data, avoiding modality gaps while achieving strong empirical results in applications such as traffic and climate forecasting.
- Data Modalities: Single-modal methods may not capture the full picture for some challenging finance and healthcare tasks.Multimodal task-oriented foundation models incorporate additional information to enhance model capability.
6 CONCLUSION
The survey reviews foundation models for time series analysis through a methodology-centric taxonomy and consolidates recent advances and future directions.
- The survey provides a comprehensive and updated review of foundation models specifically designed for time series analysis.
- Its taxonomy classifies time-series foundation models by model architecture, pre-training technique, adaptation technique, and data modality.
- The survey aims to clarify the underlying mechanisms through which foundation models are applied to time series.
- It consolidates recent advancements and identifies potential future directions to inspire further work in time series analysis.
A APPENDIX
The appendix outlines future directions for time-series foundation models, emphasizing multimodal learning, efficient architectures, robust pipelines, and privacy preservation.
- Incooporating Multi-modalities: Current time-series foundation models mostly rely on a single modality, motivating multimodal learning across time series, text, and images.The goal is to learn more comprehensive and generalized knowledge for downstream tasks.
- Exploring more Efficient Architectures: Transformer-based foundation models scale quadratically with sequence length, making long-sequence processing computationally expensive and memory-intensive.The survey identifies state-space models such as Mamba as a possible direction for more efficient backbones.
- Developing more Effective Pipelines: Time-series foundation models must address temporal distribution shift and provide stronger interpretability for downstream tasks.The survey characterizes temporal distribution shift as data distributions evolving over time and notes causality among time-series points.
- Protecting Privacy: Training foundation models on diverse sources and modalities raises privacy risks involving sensitive information.Federated learning is suggested as a way to train across decentralized devices or servers without exchanging raw data.