PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Time Series Models: A Comprehensive Survey and Benchmark

Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Chen Wang, Mingsheng Long, Jianmin Wang

arXiv:2407.13278v3cs.LG

TL;DR

Time series modeling must capture complex temporal structure, but existing reviews and methods provide limited coverage across tasks, architectures, and nonlinear dynamics. The paper surveys deep time series models from basic modules to architectures and introduces TSLib for fair multi-task evaluation. Its benchmark finds that model structures are often task-specific, while Transformers show consistent superiority across evaluated tasks; traditional methods remain limited by linearity and stationarity assumptions.

  • Problem

    Existing time series methods and reviews provide limited coverage of nonlinear dynamics and comprehensive task-and-architecture comparisons.

  • Method

    The paper systematically reviews deep time series models by basic modules and architectures, and develops TSLib as a fair benchmark across diverse analysis tasks.

  • Results

    Models with specific structures are apt at distinct analytical tasks, while Transformer-based models show consistent superiority across the benchmark.

  • Takeaways & Limitations

    TSLib and the model-centric review provide practical and research-oriented guidance on the effective scope of deep time series models.

  • Takeaways & Limitations

    Traditional statistical methods remain constrained by assumptions of linearity and stationarity when applied to intricate real-world time series.

Abstract

from arXiv · show

Time series, characterized by a sequence of data points organized in a discrete-time order, are ubiquitous in real-world scenarios. Unlike other data modalities, time series present unique challenges in learning and modeling due to their intricate and dynamic nature, including the entanglement of nonlinear patterns and time-variant trends. Recent years have witnessed remarkable breakthroughs in time series analysis, with techniques shifting from traditional statistical methods to contemporary deep learning models. In this paper, we delve into the design of deep time series models across various analysis tasks and review the existing literature from two perspectives: basic modules and model architectures. Further, we develop and release Time Series Library (TSLib) as a fair benchmark of deep time series models for diverse analysis tasks. TSLib implements 41 prominent models, including both small- and large-scale time series models, covers 30 datasets from different domains, and supports 5 prevalent analysis tasks. Based on TSLib, we evaluate 16 popular deep time series models and 6 advanced time series foundation models. Empirical findings indicate that models with specific structures are apt only at distinct analytical tasks, providing insights for research and adoption of deep time series models. Code and datasets are available at https://github.com/thuml/Time-Series-Library.

1 INTRODUCTION

Time series analysis faces complex temporal structure that limits traditional statistical methods, while existing deep-model literature lacks a comprehensive task-and-architecture overview. This survey addresses that gap through a model-centric review and a fair benchmark.

  • Time series contain sequential dependencies, trends, seasonal patterns, and complicated dynamics whose semantic information derives primarily from temporal variation.
  • Traditional methods rely on linearity and stationarity assumptions, limiting their ability to model nonlinear relationships and long-term dependencies in evolving data.
  • Existing reviews typically focus on either specific analysis tasks or specific deep-learning architectures, leaving no comprehensive overview covering both.
  • The survey reviews deep time series models from basic modules through modern architectures and discusses time series foundation models.
  • TSLib provides a fair benchmark for evaluating deep time series models across diverse analysis tasks and identifying their effective scope.

2 PRELIMINARIES

Time series analysis models temporal dependencies and inter-variable correlations across tasks such as forecasting, imputation, classification, and anomaly detection. Deep learning offers scalable alternatives to assumption-heavy statistical methods, and this survey organizes models around their foundational design challenges.

  • 2.1 Time Series: A time series consists of T observations ordered by time, with each observation containing C variables and no assumed spatial structure across variates.
  • 2.1 Time Series: Temporal dependencies describe connections among distinct time points or sub-series and are central to modeling sequential data.
  • 2.1 Time Series: Variate correlations capture interactions among multiple measurements as they evolve, especially in high-dimensional multivariate time series.
  • 2.2 Time Series Analysis Tasks: Deep learning models provide representation learning and scalability for high-dimensional and nonlinear time series where statistical assumptions restrict applicability.
  • 2.2 Time Series Analysis Tasks: Time series analysis includes forecasting, imputation, classification, and anomaly detection, each addressing distinct downstream applications.
  • 2.2 Time Series Analysis Tasks: The survey adopts a model-centric view focused on modules and architectures for capturing temporal and inter-variable dependencies in large-scale datasets.

3 BASIC MODULES

The survey reviews foundational tools used in deep time series models, including stationarization, decomposition, basis expansion, factorization, and frequency analysis. These modules transform, separate, or represent temporal data to expose patterns, dependencies, and correlations.

  • Role in Deep Models: Classical tools remain influential because modern deep models continue to integrate them as foundational components despite the shift toward deep learning.The survey frames these tools as major components of contemporary deep time series architectures.
  • Stationarization: Stationarization addresses changing distributions by standardizing series and later restoring their original mean and variance statistics.Deep approaches include adaptive normalization, reversible instance normalization, and series-level stationarization.
  • Decomposition: Decomposition separates complex series into more regular components, with average pooling extracting trends and residuals treated as seasonal parts.Seasonal-trend decomposition can distinguish trend, seasonal, cyclical, and irregular components.
  • Basis Expansion: Basis expansion represents time series through predefined functions or learned coefficients, supporting hierarchical decomposition and interpretable nonlinear temporal relationships.N-BEATS uses backcast residuals to remove well-estimated signal portions and sums partial forecasts from successive blocks.
  • Factorization: Factorization-based decomposition targets multivariate series by reducing correlated variables into a more compact latent space.Temporal and spatial regularizers can model dependencies among latent temporal embeddings and factors.
  • Frequency Analysis: Fourier analysis transforms discrete observations into the frequency domain, revealing periodic components and cyclical behavior for deep-model design.FFT and Wavelet Transform provide bridges between temporal and frequency representations.

4 MODEL ARCHITECTURES

Deep time series models are organized by five architectural backbones: MLP-, RNN-, CNN-, GNN-, and Transformer-based models. Each backbone has distinct design principles and strengths for particular time series challenges.

  • Architectural Taxonomy: Five backbone groups structure the survey: MLP-based, RNN-based, CNN-based, GNN-based, and Transformer-based architectures.The categorization is based on architectural backbones and is illustrated chronologically in Figure 2.

4.1 Multilayer Perceptrons

MLP-based time series models range from fully connected residual architectures to simple linear, frequency-domain, and multiscale designs. Their mechanisms target temporal dependencies, cross-series structure, exogenous variables, or dynamical-system behavior.

  • Overview: MLP-based models became popular for time series after autoregressive models demonstrated the value of using past values to model temporal behavior.The survey presents MLPs as a prominent architecture for time series modeling.
  • Residual and Basis Models: N-BEATS uses deep fully connected stacks with separate backcast and forecast branches, while N-HiTS adds multirate sampling and hierarchical interpolation.N-BEATS does not include time-series-specific components for capturing temporal patterns.
  • Linear and Frequency-Domain Models: DLinear shows that simple linear regression in raw space can deliver strong modeling performance and efficiency compared with complicated deep architectures.Many forecasting MLP models rely on simple linear layers.
  • Linear and Frequency-Domain Models: FITS treats forecasting as interpolation in the complex frequency domain, while FreTS models inter-series and intra-series dependencies with frequency-domain MLPs.FreTS captures channelwise and time-wise dependencies in multivariate data.
  • Multiscale and Dynamical Models: TimeMixer introduces multiscale mixing, TiDE incorporates exogenous variables, and Koopa hierarchically disentangles dynamics using Koopman theory and DMD.These designs extend MLP-based modeling beyond simple linear layers over discrete time points.

4.2 Recurrent Neural Networks

RNN-based models exploit sequential state updates for time series while addressing gradient vanishing and multivariate correlations. The surveyed extensions combine recurrence with convolution, attention, state-space formulations, or continuous-time dynamics.

  • Overview: RNNs suit time series because they are designed for sequential data, but their variants primarily address gradient vanishing and mutual correlations among variables.These concerns motivate much of the RNN-based time series literature.
  • RNN Variants: LSTNet combines recurrence and convolution to capture short-term local dependencies and long-term patterns, adding recurrent skips for periodic dependencies.DA-RNN instead uses dual-stage attention to adaptively extract relevant series at each time step.
  • State Space Models: Classical state space models represent probabilistic dependencies through Markovian states, whereas traditional fitting handles each series separately and misses shared dataset-level patterns.Deep state-space approaches combine structural assumptions with learned nonlinear patterns.
  • State Space Models: DSSM uses an RNN to parameterize a linear state space model, while S4 introduces a new state-space parameterization for sequence modeling.These approaches fuse classical state-space structure with deep neural networks.
  • Continuous-Time Models: Neural ODEs extend recurrent models to continuous time by parameterizing hidden-state derivatives with neural networks.Their continuous-time formulation has been explored for irregularly sampled time series and for incorporating physical priors.

4.3 Convolutional Neural Networks

CNN-based time series models exploit temporal continuity to capture local patterns, while newer designs expand their receptive field or reshape series to model multi-scale periodic structure.

  • CNNs provide a competitive backbone because time series semantics are primarily encoded in temporal variation and convolutions capture local features.
  • 1D CNNs discern local temporal patterns, whereas SCINet uses hierarchical downsample-convolve-interact operations to capture dependencies across resolutions.
  • Dilated causal convolutions in Wavenet and TCN enlarge temporal context for long-range causality, but TCN can struggle with global relationships because of its limited receptive field.
  • MICN combines local and global convolution kernels, while ModernTCN separates depthwise convolutions and convolutional feed-forward networks to model different dependencies.
  • TimesNet transforms one-dimensional series into multiple two-dimensional tensors based on estimated periods, arranging inter-period and inner-period variations along different axes.
  • TimesNet applies Inception blocks and maps representations back to one dimension, capturing multi-scale intra-period and inter-period variation for four analysis tasks.

4.4 Graph Neural Networks

Graph neural networks model multivariate time series as spatiotemporal graphs, representing variables as nodes to capture relationships among variables and their temporal evolution.

  • GNNs address complicated nonlinear correlations in multivariate time series by modeling the data as a spatiotemporal graph.
  • Each graph node represents a variable, allowing GNNs to extract neighboring relationships and track temporal changes in node attributes.
  • GNN architectures primarily seek to model latent topological relations in multivariate data.
  • Existing GNN-based models use either pre-defined graph structures or graphs learned dynamically from the data.
  • StemGNN combines Graph Fourier Transform and Discrete Fourier Transform to model multivariate time series in the spectral domain.

4.5 Transformers

Transformer-based time series models use self-attention to capture long-term dependencies and multivariate correlations, with designs differing in representation granularity and attention efficiency.

  • Transformers provide a powerful time series backbone because self-attention captures long-term temporal dependencies and entangled multivariate correlations.
  • Transformer representations are commonly organized as point-wise, patch-wise, or series-wise tokens.
  • Point-wise Dependency: Point-wise models embed raw values and timestamp information into high-dimensional representations using projection, positional encoding, and temporal encoding.
  • Point-wise Dependency: Because canonical attention has quadratic computational complexity, efficient-attention methods modify the mechanism to reduce complexity.
  • Patch-wise Dependency: Patch-based models split series into temporal patches, embed them with projections and positional embeddings, and learn patch-wise dependencies.
  • Patch-wise Dependency: Pathformer dynamically selects patch sizes through adaptive pathways to learn multi-scale temporal representations.
  • Patch-wise Dependency: Crossformer adds cross-time and cross-dimension attention to capture dependencies between patch tokens from different variables.
  • Series-wise Dependency: Series-wise approaches tokenize entire series; iTransformer projects each variable into a global representation and applies vanilla Transformer attention to mutual correlations.

5 TIME SERIES FOUNDATION MODELS

Time series foundation models extend large-scale pre-training to temporal data through encoder-based, decoder-based, and language-model-empowered approaches. Their design choices trade specialized temporal fidelity against generalized reasoning.

  • Time series foundation models use large-scale pre-training to learn complex temporal dependencies and dynamics from extensive time series data.
  • Encoder-based TSFM: Encoder-based models include MOMENT, which uses masked reconstruction on univariate series, and Moirai, which pre-trains multivariate data as a flattened sequence.
  • Recent foundation-model development is tracked chronologically by model release date in the paper’s timeline.
  • Decoder-based TSFM: Decoder-based models generate forecasts through rollout architectures, including Lag-Llama with lag covariates and TimesFM with input patching.
  • Language-model-based approaches tokenize time series before applying pretrained language models, using methods such as digit strings or scaled quantized vocabularies.
  • Chronos trains a probabilistic forecaster with scaled and quantized values in a fixed vocabulary using cross-entropy loss.
  • Native temporal foundation models are presented as preferable for purely numerical scenarios because their specialized architectures target fine-grained temporal patterns.

6 TIME SERIES LIBRARY

TSLib addresses limited task and domain coverage in existing benchmarks with a unified framework for evaluating deep time series models. Its experiments compare diverse architectures across tasks, foundation models, efficiency, and robustness.

  • Design and Implementation Principles: TSLib provides a unified framework for fair evaluation across diverse time series tasks, datasets, models, and standardized protocols.It implements 41 widely used and advanced deep time series models and supports extensible data and model components.
  • Diverse Data Formats: More than 30 real-world datasets span energy, transportation, economics, weather, and medicine, with varied formats and sampling frequencies.The library supports formats including .csv, .npz, and .txt, alongside scalable data integration and processing strategies.
  • Comprehensive Performance Evaluation: TSLib evaluates classification, imputation, forecasting, and anomaly detection, distinguishing forecasting into long- and short-term settings.Each task uses specific evaluation metrics, while experiments share standardized training strategies and test sets.
  • Quantitative Results: Transformer-based models, especially iTransformer and PatchTST, achieve state-of-the-art performance across diverse tasks, particularly long-term forecasting.GNNs remain competitive in selected tasks, while MLPs are efficient but weaker at high-level representation learning and RNNs are more viable for short-term tasks.
  • Quantitative Results: Increasing input context generally improves zero-shot forecasting, but excessively long inputs can obscure meaningful temporal patterns.Sundial with an input length of 2880 surpasses other TSFMs at 8192, indicating that context length should balance context richness and data quality.
  • Qualitative Analysis: MLP-based models and Mamba show superior efficiency, whereas CNN-based models are more time-consuming and tokenization design affects computational cost.The efficiency comparison measures training time, inference time, and memory usage under unified hyperparameter settings.
  • Qualitative Analysis: Adding random white noise causes only marginal performance degradation across most evaluated models in long- and short-term forecasting and classification.The robustness analysis selects the top three deep models for each task.

7 FUTURE DIRECTIONS

Future directions concern stronger pretraining, probabilistic forecasting, scalable long-sequence modeling, exogenous variables, heterogeneous series, and robustness to complex disturbances. These directions target broader temporal coverage and more reliable real-world deployment.

  • Time Series Pre-training: Self-supervised pretraining for time series is primarily organized around contrastive learning and masked modeling.Representative methods capture temporal structure, hierarchical representations, seasonal-trend factors, or masked-token context.
  • Probabilistic Forecasting: Probabilistic forecasting provides distributions over future outcomes, supporting risk assessment, confidence intervals, and decision-making under uncertainty.The passage identifies reliance on predefined probability distributions as a characteristic of many existing models.
  • Long Time Series Modeling: Longer time series remain difficult for deep models because high-frequency sampling produces lengthy histories that increase scalability and computational challenges.Existing methods commonly use patching techniques to handle long sequences.
  • Exogenous Variables: Modeling exogenous variables requires a unified framework for relationships between endogenous and exogenous variables.Such modeling is described as relevant to understanding interrelations, causality, performance, and interpretability.
  • Heterogeneous Time Series: Heterogeneous time series challenge current models through varying sampling rates, irregularities, different length scales, and fixed-size input requirements.Future approaches must address dynamic, variable-length data while capturing underlying patterns.
  • Model Robustness: Robust deployment requires handling impulse spikes, non-stationary noise, and irregular missing values beyond typical stochastic fluctuations.These disturbances can arise from sensor instabilities, environmental changes, or intermittent transmission failures.

8 CONCLUSION

The paper surveys deep time series models across tasks and architectures while introducing TSLib as a fair benchmark. It separates basic modules from architectures and uses experiments to inform model design and practical usage.

  • Conclusion: The survey covers forecasting, classification, imputation, and anomaly detection from both basic-module and architecture perspectives.This organization provides a model-centric view of current deep time series design principles.
  • Conclusion: TSLib enables flexible and comprehensive evaluation, supporting experiments that provide insights for model design and practical usage.The paper presents the survey and library as resources intended to support future research.
Loading 2407.13278v3…