Source-linked AI summary
Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Version
Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, Shirui Pan
TL;DR
Long-sequence multivariate forecasting requires efficient modeling of long-term dependencies while accommodating variables with distinct temporal dynamics. Triformer combines linear-complexity patch attention and compact variable-specific parameters, and experiments report superior accuracy across four datasets. Its patch-query design, however, constrains input lengths and forecasting horizons after training.
Problem
Canonical self-attention has quadratic complexity in sequence length, while shared parameters may fail to capture distinct temporal patterns across variables.
Method
Triformer uses patch attention in a triangular structure with exponentially shrinking layers and lightweight variable-specific projection matrices.
Results
Triformer outperforms all baselines on all datasets in overall accuracy and is reported to outperform state-of-the-art methods across four datasets.
Takeaways & Limitations
The approach combines linear-complexity long-sequence modeling with parameters tailored to distinct variable dynamics without compromising efficiency and memory usage.
Takeaways & Limitations
Because Patch Attention learns one pseudo-timestamp per patch, trained models cannot support dynamic input lengths or different forecasting horizons.
Abstract
from arXiv · showhide
A variety of real-world applications rely on far future information to make decisions, thus calling for efficient and accurate long sequence multivariate time series forecasting. While recent attention-based forecasting models show strong abilities in capturing long-term dependencies, they still suffer from two key limitations. First, canonical self attention has a quadratic complexity w.r.t. the input time series length, thus falling short in efficiency. Second, different variables' time series often have distinct temporal dynamics, which existing studies fail to capture, as they use the same model parameter space, e.g., projection matrices, for all variables' time series, thus falling short in accuracy. To ensure high efficiency and accuracy, we propose Triformer, a triangular, variable-specific attention. (i) Linear complexity: we introduce a novel patch attention with linear complexity. When stacking multiple layers of the patch attentions, a triangular structure is proposed such that the layer sizes shrink exponentially, thus maintaining linear complexity. (ii) Variable-specific parameters: we propose a light-weight method to enable distinct sets of model parameters for different variables' time series to enhance accuracy without compromising efficiency and memory usage. Strong empirical evidence on four datasets from multiple domains justifies our design choices, and it demonstrates that Triformer outperforms state-of-the-art methods w.r.t. both accuracy and efficiency. This is an extended version of "Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting", to appear in IJCAI 2022 [Cirstea et al., 2022a], including additional experimental results.
1 Introduction
Long-sequence multivariate forecasting needs efficient modeling of long-term dependencies while accounting for distinct temporal patterns across variables. Triformer addresses these needs with linear-complexity patch attention, a triangular multi-layer structure, and compact variable-specific parameters.
- Long-term forecasting remains less studied than short-term forecasting, despite attention models capturing long-term dependencies better than RNNs and TCNs.
- Canonical self-attention has O(H^2) complexity for H timestamps, whereas Triformer targets linear complexity O(H).
- Triformer splits time series into patches and uses Patch Attention within a triangular architecture whose layer sizes shrink exponentially.
- Variable-agnostic projection matrices force one parameter space to capture an average pattern across variables with distinct temporal dynamics.
- Compact variable-specific matrices are factorized with shared matrices to capture distinct patterns without increasing parameter space or computation overhead.
- Experiments on four public multivariate datasets report that Triformer outperforms state-of-the-art methods in accuracy and efficiency.
2 Related Work
Related work distinguishes forecasting methods by horizon and variable parameterization, while efficient attention and hierarchical designs address long sequences with different trade-offs. Triformer differs through linear patch attention and a distinct layer-shrinking mechanism for time series.
- Forecasting methods are categorized by short versus long horizons and variable-agnostic versus variable-specific modeling.
- Short-term models commonly use RNNs or TCNs, whereas long-term self-attention models offer superior accuracy but incur quadratic memory and runtime overhead.
- Sparse attention methods reduce complexity to O(H(log H)^2) or O(H log H), while Triformer is designed for linear complexity.
- Prior variable-specific methods either require additional metadata or learn separate weight matrices, while Triformer uses a lightweight approach without relying on such metadata.
- Pooling-based transformers reduce layer sizes with pooling after self-attention, whereas patch attention uses a learnable query per patch and does not require pooling.
- Unlike hierarchical transformers that model dependencies between patches quadratically, Triformer is designed for time-series forecasting with linear patch attention.
3 Preliminaries
The paper formulates multivariate forecasting as predicting future observations from historical observations and reviews self-attention as a dependency-modeling operation. It identifies quadratic attention cost and shared projections across variables as key limitations.
- A multivariate time series records N variables over time, and forecasting maps H historical observations to F future predictions.
- For each variable, self-attention transforms the series into query, key, and value matrices using learnable projection matrices.
- Self-attention computes outputs as weighted sums of values, with attention scores derived from queries and keys.
- Computing canonical attention scores requires O(H^2) time and memory, motivating the search for linear-complexity alternatives.
- Related attention models apply the same projection matrices WQ, WK, and WV to every variable, whereas Triformer seeks variable-specific projections.
4 Triformer
Triformer combines linear Patch Attention with triangular stacking for efficient multiscale forecasting and compact variable-specific parameters for distinct temporal dynamics. Its recurrent patch connections preserve information flow despite local attention, while factorized parameters limit overhead.
- Linear Patch Attention: Patch Attention replaces within-patch self-attention with a pseudo timestamp that aggregates information from each real timestamp in O(S) complexity.Naive self-attention within a patch remains quadratic, O(S^2).
- Linear Patch Attention: Recurrent connections link updated pseudo timestamps across patches, compensating for Patch Attention’s temporal receptive field being limited to patch size S.The connection maintains temporal information flow between patches.
- Triangular Stacking: Triangular stacking feeds only pseudo timestamps into subsequent layers, shrinking layer sizes exponentially while preserving overall linear complexity O(H).For patch sizes S_l ≥ 2, the sum of layer input sizes is bounded by 2·H.
- Triangular Stacking: Aggregating pseudo timestamps from every layer supplies multiscale features and multiple short gradient paths to the predictor.The aggregate outputs represent different temporal scales rather than only the final layer’s representation.
- Variable-Specific Modeling: Variable-specific modeling addresses distinct temporal patterns by factorizing each projection matrix into shared outer matrices and a compact variable-specific middle matrix.The middle matrix B(i) is generated from variable memory M(i), while L and R are shared across variables.
- Variable-Specific Modeling: The generator-based parameterization uses N·m memory parameters plus m·a^2 generator overhead instead of directly learning N·a^2 middle-matrix parameters.Sharing the outer matrices also acts as an implicit regularizer and facilitates knowledge sharing among variables.
5 Experiments
Experiments on four real-world multivariate forecasting datasets evaluate Triformer’s accuracy, efficiency, ablations, and hyperparameter sensitivity. Triformer achieves the best accuracy across all settings while remaining efficient on long inputs.
- Experimental Setup: Experiments use four real-world datasets with historical timestamps H to forecast future timestamps F across multiple long-horizon settings.The datasets include ETTh1, ETTm1, ECL, and Weather; ECL contains 321 variables.
- Experimental Results: Triformer outperforms all baselines on all datasets and achieves the best accuracy in every reported setting.Autoformer is the second-best method in overall accuracy, while AGCRN generally outperforms the variable-agnostic attention baselines.
- Experimental Results: At H=1024 and F=2048 on ECL, Informer falls behind Triformer while Autoformer runs out of memory.This additional experiment compares Triformer with the two most competitive baselines, Autoformer and Informer.
- Ablation Study: Ablations show significant accuracy losses when removing variable-specific modeling, triangular multi-layer stacking, or multi-scale outputs.The naive variable-specific method also uses significantly more parameters, while the generator adds very limited time.
- Ablation Study: VSM significantly improves Informer accuracy, indicating that the variable-specific component generalizes beyond Triformer.The study adds VSM to Informer and reports a significant accuracy increase.
- Hyper-Parameter-Sensitivity Analysis: Patch-size and layer choices trade accuracy, runtime, and parameter count, with (8, 6) offering a faster but less accurate two-layer setting.The authors conclude that Triformer provides flexibility across accuracy, efficiency, and parameter-size requirements.
- Hyper-Parameter-Sensitivity Analysis: Accuracy is insensitive to memory size m across {3, 5, 16, 32}, whereas a=3 causes a significant drop and a=5 is sufficient.Increasing a beyond 5 to 32 does not improve accuracy.
- Efficiency: Triformer is faster than Informer and Autoformer as H varies, remains nearly steady relative to StemGNN, and is fastest across all F settings.All methods require less than 13.3 ms per inference in the reported experiments.
6 Limitations and Possible Solutions
Triformer has two stated limitations: its learnable parameters depend on forecasting settings, and it cannot forecast different horizons once trained. A generator or encoder-decoder architecture are proposed as possible remedies.
- Patch Attention’s learnable parameter count depends on the forecasting settings, including the input size and window size.With H = 160 and S = 10, the model learns 16 pseudo-timestamps; changing to H = 200 would require 40.
- Possible solutions: A generator could produce pseudo-timestamps from patch indices, decoupling the number of patches from the learned pseudo-timestamps.This would require learning only the generator’s parameters and could make the model more flexible.
- Triformer cannot forecast different horizons after training.An encoder-decoder architecture could replace the fully connected predictor and enable autoregressive forecasting.
7 Conclusion and Outlook
Triformer combines triangular patch attention for linear complexity with lightweight variable-specific projection matrices for distinct temporal patterns. Experiments on four datasets report that it outperforms other state-of-the-art methods for long-sequence multivariate forecasting, while future work targets dynamic input lengths and curriculum learning.
- Triformer uses triangular patch attentions to ensure linear complexity and lightweight variable-specific projection matrices to capture distinct temporal patterns.The projection matrices are tailored to each variable’s time series.
- Experiments on four datasets show that Triformer outperforms other state-of-the-art methods for long-sequence multivariate time series forecasting.
- Outlook: Future work will explore support for dynamic input lengths and improved model training through curriculum learning.