Source-linked AI summary
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, Xifeng Yan
TL;DR
Time series forecasting needs models that handle related series, long-term dependencies, local context, and long sequences efficiently. The paper applies Transformer with convolutional self-attention and LogSparse attention, reporting favorable results on synthetic and real-world data, including state-of-the-art performance in varied settings.
Problem
Canonical Transformer attention is insensitive to local context and has quadratic space complexity, while forecasting also involves long-term dependencies and known covariates.
Method
The paper uses causal convolutions to produce locally informed queries and keys, and LogSparse attention to reduce memory usage to O(L(log L)^2).
Results
Experiments on synthetic and real datasets show Transformer captures long-term dependencies, convolutional self-attention improves performance, and sparse models can achieve better results under the same memory budget.
Takeaways & Limitations
Transformer-based forecasting can model long-term dependencies while the proposed locality and sparsity mechanisms improve applicability to fine-grained, memory-constrained time series.
Takeaways & Limitations
The forecasting setup assumes associated time-based covariates are known over the entire time period.
Abstract
from arXiv · showhide
Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length $L$, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$ memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.
1 Introduction
The paper applies Transformer to time series forecasting while addressing canonical Transformer’s locality insensitivity and quadratic memory growth. It introduces convolutional self-attention and LogSparse Transformer to improve local-context modeling and enable longer sequences under memory constraints.
- Traditional forecasting models fit each series independently and require manual selection of trend, seasonality, and other components.
- RNNs are difficult to train because of gradient vanishing and exploding problems, despite variants such as LSTM and GRU.
- Long- and short-term repeating patterns coexist in real-world forecasting, making long-term dependency modeling critical for performance.
- Canonical self-attention is insensitive to local context, which may increase anomaly sensitivity and create optimization issues, while its space complexity grows quadratically with sequence length.
- The paper proposes convolutional self-attention and LogSparse Transformer, with O(L(log L)^2) space complexity, and evaluates them on synthetic and real datasets.Causal convolutions produce queries and keys, incorporating local context into attention.
2 Related Work
Related work spans statistical, matrix-based, and neural forecasting methods. The paper positions Transformer as attractive for long-range dependencies but constrained by quadratic attention memory costs on extremely long sequences.
- ARIMA is widely used because of its statistical properties and Box-Jenkins model-selection methodology, but its linear assumption and limited scalability restrict large-scale forecasting.
- ARIMA fits each time series individually, preventing information sharing across related series.
- Deep neural networks address related-series forecasting by capturing shared information through architectures combining RNNs, AR models, MLPs, or state-space models.
- Transformer attention can model extremely long-range sequences, but its space complexity grows quadratically with sequence length, making fine-grained long-series forecasting computationally prohibitive.
3 Background
The forecasting setup models related univariate series using shared parameters, known time-based covariates, and a one-step-ahead conditional distribution. Transformer then uses masked multi-head self-attention followed by feedforward processing.
- Problem definition: The problem consists of N related univariate time series, with the next τ time steps predicted for every series.
- Problem definition: Known time-based covariates, such as day-of-the-week and hour-of-the-day, are assumed available across the entire forecasting period.
- Problem definition: The model learns a shared one-step-ahead distribution p(z_t|z_1:t−1, x_1:t; Φ) using observations and covariates.
- Problem definition: Observations and covariates are concatenated into augmented vectors y_t and an input matrix Y_t for prediction.
- Transformer: Transformer is instantiated with multi-head self-attention to capture long- and short-term dependencies and temporal patterns across attention heads.
- Transformer: Each attention head projects Y into query, key, and value representations, computes vector outputs, masks future positions, and passes the result to a feedforward sublayer.
4 Methodology
The methodology addresses Transformer’s locality and memory limitations for time-series forecasting with convolutional self-attention and LogSparse attention. Convolution incorporates local context into query-key matching, while logarithmic sparsity preserves information flow with substantially lower memory growth.
- 4.1 Enhancing the locality of Transformer: Convolutional self-attention uses causal convolutions to transform inputs into queries and keys that incorporate local context.Causal convolutions prevent access to future information, while local shapes can inform query-key similarity instead of point-wise values.
- 4.1 Enhancing the locality of Transformer: Canonical Transformer attention patterns evolve from shared daily patterns in layer 2 to weekend-focused and highly concentrated patterns in deeper layers.The traffic-f visualization uses a 10-layer canonical Transformer with full attention and examines layers 2, 6, and 10.
- 4.2 Breaking the memory bottleneck of Transformer: O(L(log L)^2) total memory usage enables information flow between any cells by using O(log L) attention per cell and stacking up to O(log L) layers.The construction requires up to ⌊log2 L⌋ + 1 layers for every cell to access preceding information.
- 4.2 Breaking the memory bottleneck of Transformer: LogSparse self-attention lets each cell attend to itself and previous cells selected at exponentially increasing step sizes.This selects logarithmically spaced historical positions rather than all previous cells.
- 4.2 Breaking the memory bottleneck of Transformer: Local attention adds dense access to a left window of size O(log2 L), while restart attention applies LogSparse attention separately to input subsequences.Both variants preserve the sparse-attention complexity, and local and restart attention can be combined.
5 Experiments
Experiments on synthetic and real-world datasets evaluate long-term dependency modeling, convolutional self-attention, sparse attention, and performance across forecasting granularities.
- Synthetic datasets: Transformer and DeepAR are compared using ρ-quantile loss on synthetic piece-wise sinusoidal datasets with t0 ranging from 24 to 192.The experiment uses a 3-layer canonical Transformer and a 3-layer LSTM-based DeepAR baseline.
- Synthetic datasets: As t0 increases, DeepAR performance drops significantly, while canonical Transformer maintains accuracy when t0 ≥96.The synthetic task predicts the final 24 steps after increasingly long histories containing irrelevant signals.
- Convolutional self-attention: Convolutional self-attention converges faster and reaches lower training errors than selected kernel-size alternatives on electricity-c and traffic-c.The comparison uses kernel sizes k ∈{1, 3, 9}.
- Sparse attention: On traffic-f with strong long-term dependencies, sparse convolutional Transformer slightly outperforms full attention in R0.5 under equal input length.Full attention generally performs better in most same-input-length comparisons, but convolutional self-attention benefits constrained full-attention models on both fine-grained datasets.
- Further exploration: Across datasets with various granularities, the proposed method achieves the best performance overall.The evaluation includes electricity-f, traffic-f, solar, wind, and M4-Hourly datasets, with R0.5/R0.9-loss reported where applicable.
6 Conclusion
The paper finds that Transformer can capture long-term dependencies in time series, while convolutional self-attention improves performance across real-world settings. Sparse attention models perform better on long-term-dependent data under the same memory budget.
- Transformer can capture long-term dependencies, while LSTM may suffer on time series forecasting.
- Convolutional self-attention further improves Transformer performance and achieves state-of-the-art results across different real-world settings.
- Sparse attention models achieve better results than competing models on data with long-term dependencies under the same memory budget.
A.1 Proof of Theorem 1
The proof shows that LogSparse attention connects each cell to all preceding cells through short directed paths. Binary decompositions establish a logarithmic path-length bound, while reordering selected steps yields multiple feasible paths.
- Each LogSparse layer lets cell l attend to cells indexed by l − 2^⌊log2 l⌋ through l − 1 and l itself.
- Every cell receives information from all previous cells and itself after stacking a sufficient number of layers.
- A path from cell j to cell l has length no greater than ⌊log2 l⌋ + 1.
- Reordering the binary-decomposition steps creates multiple paths, whose count grows at O(⌊log2(l − j)⌋!).
A.2 Training
Training uses sliding windows from long time series, learnable positional embeddings, covariates, and a decoder-only Transformer. Optimization and dataset-specific sampling choices vary across experiments.
- Training instances are fixed-history, fixed-horizon sliding windows sampled from original long time series without accessing test data.
- Learnable position embeddings and normalized temporal covariates encode dataset-specific calendar information, age, and time-series identity.
- The models use a Transformer decoder-only mode to predict time-series distributions from current inputs and hidden states.
- BERTAdam with warmup and learning-rate annealing stabilizes training on electricity-f and traffic-f, where vanilla Adam is unstable.
- Training-window counts range from 10K to 500K across datasets, while Transformer models generally use 8 heads and 3 layers.
A.3 Evaluation
Evaluation reserves dataset-specific temporal segments as test sets, including one-week, seven-day, or 210-day periods depending on the dataset.
- Test sets are formed from dataset-specific final or dated temporal segments, including one week, seven days, or the last 210 days.
A.4 Implementation of sparse attention and its memory cost
The sparse-attention implementation estimates memory from the attention matrix and tunes subsequence and local-attention lengths so sparse and full models use comparable memory. A traffic-c example shows the resulting attention patterns capturing both hourly and daily seasonality.
- The implementation calculates LogSparse attention’s relative memory usage ideally from the attention matrix, the Transformer memory bottleneck.
- For electricity-f, the sparse model uses Le1 = 768, subsequence length Le1sub = 7, and full attention uses Le2 = 293.The same number of layers is stacked so memory usage can be compared layer by layer.
- Each sparse-attention cell attends to at most 2∗Le1loc = 14 cells per subsequence and 2∗Le1sub = 112 cells in total.
- The sparse and full attention models have comparable memory usage under the electricity-f setting, and the same procedure applies to traffic-f.
- In traffic-c, learned attention patterns attend to matching weekday or weekend times, capturing both hourly and daily seasonality.The displayed example uses an 8-day time-series window and its corresponding masked attention matrix.