Source-linked AI summary
A Systematic Review for Transformer-based Long-term Series Forecasting
Liyilei Su, Xumin Zuo, Rui Li, Xin Wang, Heng Zhao, Bingding Huang
TL;DR
LTSF requires models that can handle long sequences and their dependencies, while existing forecasting practice also faces limited large datasets and evaluation constraints. This review examines transformer architectures and LTSF adaptations, surveys datasets and metrics, and summarizes training practices and research directions. It concludes that transformers offer strong modeling capacity for long-term dependencies but face time-complexity challenges and trade-offs when reducing complexity.
Problem
LTSF involves large sequences whose complexity, memory demands, and dependency modeling challenge forecasting, while sufficiently large public datasets remain scarce.
Method
The review analyzes transformer architecture and LTSF adaptations, then summarizes datasets, evaluation metrics, training strategies, and research directions.
Results
Transformers provide strong modeling capacity and capture long-term dependencies in LTSF, but reducing their time complexity may lose interdependencies and compromise prediction accuracy.
Takeaways & Limitations
Combining techniques in compound transformer-based models is presented as a promising direction for addressing intricate LTSF problems.
Takeaways & Limitations
RNN-based alternatives remain constrained by sequential processing and back-propagation through time, with LSTM and GRU training susceptible to gradient vanishing and explosion.
Abstract
from arXiv · showhide
The emergence of deep learning has yielded noteworthy advancements in time series forecasting (TSF). Transformer architectures, in particular, have witnessed broad utilization and adoption in TSF tasks. Transformers have proven to be the most successful solution to extract the semantic correlations among the elements within a long sequence. Various variants have enabled transformer architecture to effectively handle long-term time series forecasting (LTSF) tasks. In this article, we first present a comprehensive overview of transformer architectures and their subsequent enhancements developed to address various LTSF tasks. Then, we summarize the publicly available LTSF datasets and relevant evaluation metrics. Furthermore, we provide valuable insights into the best practices and techniques for effectively training transformers in the context of time-series analysis. Lastly, we propose potential research directions in this rapidly evolving field.
1 Introduction
Time-series forecasting has progressed from assumption-heavy statistical models through machine learning and deep learning toward transformers, whose self-attention supports learning long-term dependencies. The review surveys transformer-based LTSF architectures, their effectiveness, datasets, metrics, and training strategies.
- Statistical and machine-learning approaches: Traditional AR, MA, ARMA, and ARIMA methods rely on assumptions such as stationarity, normal distribution, linear correlation, and independence that may not hold in real-world time series.These assumptions limit their effectiveness when data are non-stationary.
- Statistical and machine-learning approaches: Machine-learning models such as SVMs and AdaBoost address nonlinear, multivariate, and heteroskedastic forecasting by deriving statistical features within sliding windows.The features include minimum, maximum, mean, and variance.
- Deep-learning approaches: Deep-learning models improve TSF nonlinear modeling by automatically learning complex patterns and feature representations while capturing long-term dependencies and handling large-scale data.The review also identifies applications in hierarchical, intermittent, sparse multivariate, asynchronous, multi-objective, multi-granular, and multi-modal forecasting.
- Deep-learning approaches: RNNs, LSTMs, and GRUs process sequences and update parameters through back-propagation through time, but sequential processing and gradient vanishing or explosion can compromise efficiency and effectiveness for long dependencies.Architectural modifications and training techniques can alleviate gradient-related problems to some extent.
- Transformer-based approaches: Transformers use self-attention with pairwise query-key interactions to learn short- and long-term dependencies, and their performance has made them a mainstream approach for LTSF.The introduction describes attention, embedding, and encoder-decoder mechanisms as components of the transformer architecture.
- Review scope: The review covers transformer architecture, transformer-based LTSF developments, effectiveness, public datasets and evaluation metrics, and existing training strategies.Its stated structure spans architecture analysis, recent LTSF applications, evaluation, resources, training practices, and conclusions.
2 Transformer
The original Transformer combines self-attention, multi-head attention, encoder-decoder components, feed-forward networks, embeddings, and positional information. These mechanisms enable parallel sequence processing while restoring information about token order.
- Architecture: The review first analyzes the original Transformer architecture and its constituent operations, while restricting this section to the original design.Transformer variants for time-series analysis are discussed later.
- Self-attention: Self-attention maps a query and key-value pairs to an output by weighting values according to query-key interactions.The attention score uses a scaled query-key dot product, followed by softmax normalization and multiplication by the value matrix.
- Self-attention: Scaling the query-key dot product by the square root of the key dimension prevents excessively large scores from producing diminutive softmax gradients that hinder training.The scaling factor is presented as essential when the key dimension is large.
- Multi-head attention: Multi-head attention applies self-attention to multiple transformed input groups, concatenates their representations, and linearly transforms the combined result.Each head uses independent linear transformations of Q, K, and V before the representations are combined.
- Encoder and decoder: The encoder contains multi-head attention and a two-layer feed-forward network with residual connections, layer normalization, and dropout, while the decoder adds encoder-decoder attention.The decoder’s middle component interacts with the encoder output, referred to as Memory.
- Positional encoding: Because self-attention and linear transformations alone do not preserve sequence order, the Transformer adds positional embeddings after token embeddings.The review describes token embedding as a dense vector representation and positional embedding as the mechanism that delineates temporal sequence.
3 Transformer-based architectures in LTSF
LTSF models must address both computational complexity and long-term temporal dependency, motivating transformer-based architectures that capture sequence correlations through self-attention.
- Transformer architectures: The section introduces recent transformer-based architectures after analyzing the key problems that LTSF networks must address.A summary of these architectures is provided in Table 1.
- Key challenges: LTSF involves complexity and dependency challenges because long sequences increase training, computational, and memory demands while requiring accurate temporal-relationship modeling.The review identifies sequence length, resource requirements, and dependency capture as central design constraints.
- Key challenges: Long-sequence forecasting must learn periodic, non-stationary mixtures of short- and long-term repeated patterns.These characteristics impose stringent requirements on models’ ability to learn temporal dependence.
- Transformer architectures: Transformers capture correlations among sequence elements through self-attention, but their high computational complexity and unclear use of location information remain concerns.The passage contrasts self-attention’s interpretability with computational costs and limitations in positional information use.
4 Transformer effectiveness for LTSF
The review examines transformer effectiveness in LTSF amid debate over simpler alternatives, then considers transformer-based and language-model-based approaches alongside the need for larger datasets.
- Effectiveness debate: Transformer effectiveness in time-series forecasting became a major research question after a simple model outperformed a considerably more complex transformer model.The review nevertheless answers the effectiveness question affirmatively.
- Time-series LLMs: Transformer-based language models may learn complex dependencies in numeric time series when augmented with temporal textual sequences.This expectation is linked to evidence that LLMs learn complex relationships among textual sequences.
- Time-series LLMs: Time-series LLM research follows either pre-training a dedicated time-series model or adapting time series for existing language models through fine-tuning.The second strategy requires mechanisms that enable language models to process time-series tasks.
- Effectiveness debate: Transformers remain relevant because they can handle extensive time-series information that straightforward linear models may not handle as effectively.The comparison is presented as circumstance-dependent rather than universally favoring one model class.
- Data requirements: Large-scale pre-training is constrained by the scarcity, limited size, and limited generalizability of publicly available time-series datasets.Traffic and Electricity are identified as common public datasets used for existing pre-trained time-series models.
5 Public datasets and evaluation metrics
The review surveys common LTSF applications and public datasets while discussing evaluation metrics for assessing prediction performance.
- Scope: This section summarizes common LTSF applications, relevant public datasets, and prediction-performance evaluation metrics.It frames datasets and metrics as central resources for LTSF tasks.
5.1 Common applications and public datasets
LTSF is applied across finance, energy, transportation, meteorology, and medicine, with public datasets covering diverse economic, operational, environmental, and mobility measurements.
- Finance: Finance applications include forecasting economic and fiscal cycles, stock trends, prices, income, expenses, and profitability.These forecasts support investment strategies and financial planning.
- Finance: Financial datasets include gas-station revenue, Japanese company records, stock prices, exchange rates, S&P 500 and SSE indices, and market attributes.The listed datasets span daily observations and multivariate financial information.
- Energy: Energy applications use LTSF for resource planning, demand forecasting, power-generation prediction, energy policy, and supply-chain management.The review connects these uses with planning, risk reduction, efficiency, and sustainable development.
- Energy: Energy datasets cover household electricity, solar production, customer electricity consumption, wind potential, transformer measurements, and photovoltaic generation.Examples include data from 137 photovoltaic plants, 321 customers, and power transformers.
- Transportation: Transportation applications include traffic-flow prediction, traffic planning, congestion forecasting, accident-risk prediction, and traffic-safety management.The review lists datasets involving drivers, vehicles, roads, taxis, bicycles, parking, freeway sensors, and traffic speed.
- Transportation: Transportation datasets include Paris metro passenger flow, PeMS traffic data, parking occupancy, METR-LA freeway information, PEMS-BAY sensor readings, taxi trips, bike trajectories, and population density.The datasets span multiple cities, transport modes, and temporal resolutions.
- Meteorology: Meteorological LTSF targets long-term climate trends, climate-disaster warnings, sea-surface temperature, and marine meteorology.The review presents these forecasts as inputs to climate-response decision-making and hazard mitigation.
- Medicine: Medical LTSF supports drug-development decisions by forecasting toxicity, pharmacokinetics, pharmacodynamics, and medical needs.These predictions are used for drug design, screening, and medical-resource planning.
5.2 Evaluation metrics
The review organizes forecasting metrics into scale-dependent, scale-independent, and scaled-error groups. It highlights their use cases and limitations, including scale comparability, outlier sensitivity, zero observations, and benchmark dependence.
- Forecasting metrics are grouped as scale-dependent, scale-independent, and scaled-error measures according to how they handle data scale.
- Scale-dependent measures: RMSE, MAE, and MdAE are scale-dependent measures useful for comparing methods on the same dataset, but not across datasets with different scales.RMSE and MSE are more sensitive to outliers than MAE and MdAE.
- Scale-independent measures: Percentage-error measures such as MAPE and MdAPE support comparisons across datasets but become undefined or infinite when observations equal zero.They can also be highly skewed near zero and penalize positive errors more heavily than negative errors.
- Scale-independent measures: Relative-error measures divide forecast errors by errors from a benchmark method, commonly a random walk using the last observation as its forecast.MRAE, MdRAE, and GMRAE are examples of relative-error metrics.
- Scaled errors: Scaled errors compare predictions with an underlying native method, and MASE greater than 1 indicates worse performance than that native prediction.MASE is more susceptible to outliers than MdASE, while both metrics only express performance relative to the basic method.
6 Training strategies
Training strategies for transformer-based LTSF include normalization, decomposition, temporal information enhancement, and choices between iterative and direct multi-step forecasting. Transformer variants also address quadratic attention costs and autoregressive error accumulation.
- Preprocessing: Zero-mean normalization and seasonal-trend decomposition are common preprocessing strategies for making time-series data more predictable.Decomposition separates the input into a trend-cyclical component and a seasonal component defined as the residual from the original sequence.
- Temporal information: Temporal embeddings, temporal convolutional layers, and learnable timestamps enhance positional and broader temporal information that self-attention cannot preserve alone.The review identifies local ordering and hierarchical timestamps such as weeks, months, and years as important temporal information.
- Forecasting strategy: Iterated multi-step forecasting repeatedly applies a single-step forecaster, whereas direct multi-step forecasting optimizes the multi-step objective simultaneously.Iterated forecasting has smaller prediction variance but remains subject to autoregressive estimation effects.
- Transformer variants: Transformer variants address vanilla-transformer limitations including quadratic self-attention time and memory complexity and error accumulation from autoregressive decoding.Examples include LogTrans with a dedicated decoder for iterative forecasting and Informer with a generative-style decoder.
7 Conclusion
The review examines transformer architectures for LTSF, their adaptations, training practices, and supporting datasets and metrics. It highlights strong modeling of long-term dependencies alongside time-complexity challenges and proposes compound models and LLM integration as future directions.
- 7 Conclusion: Self-attention and positional encoding can provide performance comparable to or better than RNNs and LSTM/GRU variants while improving computational efficiency.The review associates this efficiency with overcoming shortcomings of recurrent sequence processing.
- 7 Conclusion: The review examines transformer applications to LTSF, including architectural foundations, adaptations, training practices, datasets, and evaluation metrics.These components address both the use of transformers and the practical resources needed to train and assess them.
- 7 Conclusion: Transformer architectures offer strong modeling capacity for capturing long-term dependencies but face time-complexity challenges in LTSF.The review presents this capability and limitation as central characteristics of transformer-based LTSF.
- 7 Conclusion: Reducing complexity may lose interdependencies between data points and compromise prediction accuracy.This trade-off motivates careful architectural design rather than complexity reduction alone.
- 7 Conclusion: Compound models that combine complementary techniques are identified as a promising direction for transformer-based LTSF research.The review also points toward new model designs, data-processing techniques, and benchmarking approaches.
Conflict of interest
The authors declare no conflict of interest, and no datasets were generated or analyzed in this article.
- Conflict of interest: The authors declare that they have no conflict of interest.This statement addresses the paper's conflict-of-interest disclosure.
- Conflict of interest: Data sharing is not applicable because no datasets were generated or analyzed during the study.The statement concerns data availability for this review article.