Source-linked AI summary
Parallel Spatio-Temporal Attention-Based TCN for Multivariate Time Series Prediction
Fan Jin, Ke Zhang, Yipan Huang, Yifei Zhu, Baiping Chen
TL;DR
Long-horizon multivariate forecasting requires modeling diverse data streams and extended histories, but RNN-based approaches face gradient, memory, and serial-computation limitations. PSTA-TCN combines parallel spatio-temporal attention with stacked TCN backbones, achieving faster training, improved stability, and stronger forecasting accuracy than competing methods.
Problem
Multivariate forecasting must learn from numerous diverse data streams and long historical contexts, while RNN-based approaches have vanishing-gradient, short-memory, and serial-architecture limitations.
Method
PSTA-TCN combines parallel spatial and temporal attention branches with two stacked TCN backbones to capture exogenous-feature correlations, temporal dependencies, and patterns across window sizes.
Results
PSTA-TCN trains 14 times faster than DSTP and 12 times faster than DARNN, improves long-term prediction stability, and consistently achieves the lowest RMSE and MAE across tests.
Takeaways & Limitations
Parallel attention lets the TCN framework retain parallel computation while improving stability and forecasting performance for single-step and multi-step predictions.
Abstract
from arXiv · showhide
As industrial systems become more complex and monitoring sensors for everything from surveillance to our health become more ubiquitous, multivariate time series prediction is taking an important place in the smooth-running of our society. A recurrent neural network with attention to help extend the prediction windows is the current-state-of-the-art for this task. However, we argue that their vanishing gradients, short memories, and serial architecture make RNNs fundamentally unsuited to long-horizon forecasting with complex data. Temporal convolutional networks (TCNs) do not suffer from gradient problems and they support parallel calculations, making them a more appropriate choice. Additionally, they have longer memories than RNNs, albeit with some instability and efficiency problems. Hence, we propose a framework, called PSTA-TCN, that combines a parallel spatio-temporal attention mechanism to extract dynamic internal correlations with stacked TCN backbones to extract features from different window sizes. The framework makes full use parallel calculations to dramatically reduce training times, while substantially increasing accuracy with stable prediction windows up to 13 times longer than the status quo.
I. INTRODUCTION
The paper motivates PSTA-TCN as a parallel alternative to attention-based RNNs for multivariate, long-horizon forecasting. It combines spatio-temporal attention with stacked TCNs to target RNN limitations while improving speed, stability, and accuracy.
- Multivariate forecasting must model numerous diverse data streams and use historical patterns to predict future system behavior.
- Attention-based RNNs remain limited by vanishing gradients, insufficient temporal context, and serial computation across time steps.
- PSTA-TCN combines parallel spatial and temporal attention with two stacked TCN backbones to extract exogenous-series correlations and temporal dependencies.
- 14 times faster than DSTP and 12 times faster than DARNN, PSTA-TCN substantially reduces training time.
- PSTA-TCN improves stability for long-term and long-history prediction and outperforms advanced forecasting methods in single-step and multi-step accuracy.
- TCNs offer parallel computation and hierarchical long-range modeling, but very long sequences can remain inefficient or unstable without additional attention mechanisms.
III. SPATIO-TEMPORAL ATTENTION BASED TCN
The paper formulates forecasting from a window of exogenous and target series to predict one or more future target values. A learned nonlinear mapping F operates on the combined historical inputs.
- A. Notation and Problem Statement: The exogenous input X is represented across n dimensions and a window of length T.
- A. Notation and Problem Statement: The target series Y contains the historical values y1 through yT and has the same window length T.
- A. Notation and Problem Statement: The model predicts future values ŷT+1 through ŷT+τ, where τ denotes the number of future time steps.
- A. Notation and Problem Statement: The forecasting objective maps the exogenous series X and target series Y to future predictions through a nonlinear function F.
B. Model
PSTA-TCN processes multivariate inputs through parallel spatial- and temporal-attention branches, followed by stacked TCN backbones and dense prediction layers. Its TCN design uses causal, dilated convolutions and residual connections to model long histories efficiently.
- Parallel attention: PSTA-TCN sends multivariate inputs through parallel spatial- and temporal-attention branches before two identical stacked TCN backbones.The spatial branch extracts correlations among exogenous and target series, while the temporal branch captures dependencies across the window.
- Prediction output: The processed branch outputs pass through dense layers and are summed to produce the final multi-step prediction.The output is a sequence of τ future values, where T is the input window size and n is the exogenous-series dimension.
- Spatial attention: Spatial attention weights features within each time step, normalizing their learned importance with a softmax operation.The resulting weights represent the importance of each feature at that time step.
- Temporal attention: Temporal attention produces a weight vector across window time steps for each exogenous series, then normalizes it with softmax.The vector reflects each series’ importance across the historical window.
- Stacked TCN backbones: Causal TCN convolutions use only current and earlier states, preventing future information from leaking into predictions.This preserves the temporal ordering required for forecasting.
- Stacked TCN backbones: Dilated convolutions expand receptive fields exponentially, while residual connections support deeper networks for ultra-long sequences without vanishing gradients.The effective history per layer is (k−1)d, with k denoting kernel size and d the dilation factor.
A. Datasets
The study evaluates long-term human-motion prediction using wearable-sensor data from participants performing repeated squat sessions. Acceleration and angular-velocity streams are windowed and chronologically divided into training and test sets.
- Data collection: The experiment predicts human motion from acceleration and angular-velocity measurements collected by four wearable microsensors.Sensors recorded three-axis data from the arms and knees and connected to a mobile visualization app via Bluetooth.
- Data collection: Ten participants performed five sessions of ten squats while the sensors sampled motion at 50 times per second.The collected streams include acceleration and angular velocity along three axes.
- Prediction task: The prediction target is a sequence of future resultant acceleration values, with τ denoting the number of prediction steps.Historical acceleration, angular velocity, and resultant values form the model input.
- Dataset preparation: The dataset contains 1.96 million data points and is chronologically split into training and test sets at a 4:1 ratio.Each split is segmented into sliding windows, which are then randomly shuffled to reduce overfitting.
B. Baseline methods
The evaluation compares PSTA-TCN with recurrent, attention-based recurrent, and vanilla TCN baselines for single-step and multi-step prediction. Experiments vary historical window size and prediction horizon to assess long-range behavior.
- Compared methods: The baselines include LSTM, GRU, DARNN, DSTP, and vanilla TCN, spanning recurrent, attention-based recurrent, and convolutional approaches.DARNN and DSTP are the attention-based recurrent state-of-the-art methods, while vanilla TCN provides a convolutional baseline.
- Reported comparison: For single-step prediction, LSTM and GRU perform below the attention-based recurrent methods, while DSTP is marginally better than DARNN.The reported comparison attributes the recurrent models’ weaker performance to their lack of attention.
- Reported comparison: As prediction steps increase, RNN-based methods lose accuracy more sharply than TCN-based methods.PSTA-TCN remains accurate for very long sequences and is described as more stable than the RNN alternatives.
C. Hyperparameter setting and evaluation metrics
The experiments evaluate single-step and multi-step prediction under specified window sizes, using RMSE and MAE as accuracy metrics.
- Experiments use batch size 64, initial learning rate 0.001, and random seed 1111.
- Single-step experiments compare models across window sizes T ∈ {32, 64, 128, 256}.These settings represent different amounts of historical information.
- Evaluation uses root mean squared error (RMSE) and mean absolute error (MAE).Both metrics measure forecasting accuracy.
- Lower RMSE and MAE values indicate better accuracy.
D. Results
PSTA-TCN performs strongly across single-step and multi-step forecasting, with lower RMSE and MAE across tests and greater stability at longer horizons and windows.
- PSTA-TCN consistently achieved the lowest RMSE and MAE across all reported tests by a substantial margin.The results cover both single-step and multi-step predictions.
- TCN and PSTA-TCN were significantly more accurate, with accuracy increasing again after the window size passed 128.Accuracy fluctuated as historical windows expanded because longer inputs increased dependency-capture and training difficulty.
- RNN-based methods declined significantly more than TCN-based methods as the number of prediction steps increased.
- PSTA-TCN remained remarkably accurate on very long sequences and was more stable than RNN-based methods.The paper attributes this to better extraction of spatio-temporal dependencies from historical information.
- PSTA-TCN maintained high accuracy beyond the 32-step point where vanilla TCN began to decline.
A. Time complexity
The paper compares training-time behavior and module contributions, emphasizing parallel computation and the combined effect of spatial attention, temporal attention, and parallel TCN backbones.
- A. Time complexity: At T = 256, DSTP took 46 times longer than vanilla TCN and 14 times longer than PSTA-TCN to train.DARNN took 42 times longer than TCN and 13 times longer than PSTA-TCN.
- A. Time complexity: RNN training time increases with window size because serial computation and attention complexity make longer histories more expensive.TCNs use parallel computing to reduce training time.
- A. Time complexity: PSTA-TCN sacrifices part of TCN’s training-time reduction for spatio-temporal attention, producing greater long-sequence stability and higher accuracy than RNNs.
- B. Ablation studies: PSTA-TCN outperformed variants using only spatial attention or only temporal attention by a considerable margin.
- B. Ablation studies: Accuracy improved considerably only when spatial attention, temporal attention, and parallel backbones were combined.
- B. Ablation studies: Parallel TCN backbones provided additional information and improved multi-step forecasting, especially as the prediction horizon grew longer.The paper associates this with stronger expression ability from additional parameters.
C. Influence of Hyperparameters
PSTA-TCN performance depends on selecting suitable backbone hyperparameters and prediction-window sizes. The reported results show non-monotonic effects, with concrete optima for both settings.
- Hyperparameter influence: RMSE first falls and then rises as hidden dimension, layer count, and kernel size vary, indicating an optimal hyperparameter choice.The reported examples are H=12, L=8, and K=7.
- Hyperparameter influence: H=12, L=8, and K=7 are reported as optimal choices for hidden dimension, number of layers, and kernel size, respectively.
- Window-size influence: Prediction is poor at small window sizes, while accuracy decreases when the window exceeds a threshold because attention-based importance evaluation becomes distorted.The text attributes small-window weakness to insufficient historical information and reports an optimal value of 32.