PaperlayerView exact revision on arXiv

Source-linked AI summary

DSTP-RNN: a dual-stage two-phase attention-based recurrent neural networks for long-term and multivariate time series prediction

Yeqi Liu, Chuanyang Gong, Ling Yang, Yingyi Chen

arXiv:1904.07464v1cs.LGstat.ML

TL;DR

Long-term multivariate time-series prediction must capture spatial correlations, cross-time spatio-temporal relationships, and long-term temporal dependencies, while existing attention-based RNNs mainly perform well for one-step and short-term prediction. DSTP-RNN and DSTP-RNN-Ⅱ enhance spatial and temporal attention, outperform state-of-the-art methods across four datasets and different prediction horizons, but performance decreases as the time step increases.

  • Problem

    Long-term multivariate time-series prediction remains challenging because models must capture spatial correlations, cross-time relationships, and temporal dependencies between series.

  • Method

    The paper proposes DSTP-RNN models that enhance spatial and temporal attention to capture spatio-temporal relationships in long-term multivariate prediction.

  • Results

    DSTP-RNN and DSTP-RNN-Ⅱ achieve the best results against state-of-the-art methods across four datasets and different prediction time steps.

  • Takeaways & Limitations

    The models provide an attention-based approach for representing spatio-temporal relationships in long-term multivariate time-series prediction.

  • Takeaways & Limitations

    Prediction performance decreases as the time step increases, and deeper spatial attention is not necessary because deeper-layer weights become evenly distributed.

Abstract

from arXiv · show

Long-term prediction of multivariate time series is still an important but challenging problem. The key to solve this problem is to capture the spatial correlations at the same time, the spatio-temporal relationships at different times and the long-term dependence of the temporal relationships between different series. Attention-based recurrent neural networks (RNN) can effectively represent the dynamic spatio-temporal relationships between exogenous series and target series, but it only performs well in one-step time prediction and short-term time prediction. In this paper, inspired by human attention mechanism including the dual-stage two-phase (DSTP) model and the influence mechanism of target information and non-target information, we propose DSTP-based RNN (DSTP-RNN) and DSTP-RNN-2 respectively for long-term time series prediction. Specifically, we first propose the DSTP-based structure to enhance the spatial correlations between exogenous series. The first phase produces violent but decentralized response weight, while the second phase leads to stationary and concentrated response weight. Secondly, we employ multiple attentions on target series to boost the long-term dependence. Finally, we study the performance of deep spatial attention mechanism and provide experiment and interpretation. Our methods outperform nine baseline methods on four datasets in the fields of energy, finance, environment and medicine, respectively.

1 Introduction

Long-term multivariate time-series prediction is challenging because models must capture spatial correlations, cross-time spatio-temporal relationships, and temporal dependencies across series. The paper proposes DSTP-RNN and DSTP-RNN-Ⅱ, extending attention-based RNNs with dual-stage attention, target-information modeling, and deep spatial attention.

  • Problem: Long-term multivariate time-series prediction must represent spatial correlations, cross-time spatio-temporal relationships, and temporal relationships between different series.
  • Limitations: Attention-based encoder-decoder methods achieve strong one-step and short-term forecasting but do not perform well for long-term prediction.
  • DSTP-RNN: DSTP-RNN uses a dual-stage two-phase structure to capture spatio-temporal relationships and enhance spatial correlations between multivariate series.The two phases use consecutive attention mechanisms with and without target series, while the dual stages combine spatial attention for original series with temporal attention for the final-phase hidden state.
  • Target and no-target information mechanism: DSTP-RNN-Ⅱ uses parallel spatial attention to extract relationships between target and exogenous series while emphasizing past target values for long-term dependence.
  • Deep spatial attention: Deep spatial attention is studied through interpretation experiments to assess its effect on spatio-temporal relationships.
  • Application in many fields: The methods are evaluated on four open datasets from different fields, with extensive experiments reporting state-of-the-art results.

2 Related Work

Related work spans classical, recurrent, and attention-based approaches to multivariate time-series prediction. The paper identifies limitations in spatial, spatio-temporal, and long-term dependency modeling, then studies DSTP-RNN, DSTP-RNN-Ⅱ, and DeepAttn.

  • Classical and recurrent methods: Classical methods such as ARIMA capture target-series seasonality and long-term dependence but ignore exogenous spatial correlations and assume stable changes.These assumptions limit ARIMA and its variants for non-stationary and multivariable time prediction.
  • Classical and recurrent methods: SVR models exogenous series in high-dimensional space to capture simultaneous spatial correlations, but they ignore temporal dependence.The resulting discrete spatial distribution cannot represent temporal characteristics.
  • Classical and recurrent methods: RNNs, including LSTM and GRU, maintain continuous time dependence but lack attribute selection for identifying spatial features between different series.Their gating mechanisms incorporate external information at the current moment, while vectorized simultaneous attributes remain unselected.
  • Proposed attention mechanisms: The paper studies hierarchical DSTP-RNN, hierarchical-parallel hybrid DSTP-RNN-Ⅱ, and deep-attention DeepAttn for time-series prediction.Its two-phase attention extracts target–exogenous spatial correlations, shifting from decentralized first-phase weights to concentrated second-phase weights, while additional target attention captures long-term dependence.
  • Attention-based methods: Two-stage attention-based RNNs combine spatial attention, temporal attention, and context vectors to model spatial correlations and spatio-temporal relationships across times.However, existing methods overlook target–exogenous spatial correlations, while single-layer spatial weights are scattered, interference-prone, and suited mainly to one-step or short-term prediction.

3 Preliminary

This section defines the notation for exogenous and target series within a window of size T, along with future target values over τ prediction steps. It formulates forecasting as predicting those future values from the observed histories of both series.

  • The model considers n (n ≥1) exogenous series and one target series over a window of size T.
  • Y denotes the target series within the length-T window, while Z contains the first-phase attention output and target series.
  • Ŷ represents future target values from y_T+1 through y_T+τ, where τ is the number of time steps to predict.
  • Given past exogenous and target values, the task is to predict the target over the next τ time steps using a function F of the observed histories.

4 Model

DSTP-RNN uses dual-stage attention to extract spatial correlations among exogenous and target series, then applies temporal attention to capture longer-term dependence. DSTP-RNN-Ⅱ adds parallel attention for target and non-target information, while DeepAttn stacks a third spatial-attention layer.

  • Overall framework: DSTP-RNN and DSTP-RNN-Ⅱ use dual-stage attention, selecting spatial features first and temporal features second.The spatial module contains a two-phase structure, while temporal attention operates on encoder hidden states.
  • DSTP-RNN-Ⅱ: DSTP-RNN-Ⅱ uses parallel attention modules so target and non-target information jointly influence spatial-response selection.Its first phase independently models target–exogenous relationships, and the resulting outputs are combined before second-phase attention.
  • Deep attention: DeepAttn stacks one additional spatial-attention unit on DSTP-RNN, using the second-phase weighted data as input.The model reports the results of the third layer of the spatial-attention mechanism.
  • Spatial attention: The first spatial-attention phase extracts exogenous-series correlations, while the second reweights features and captures exogenous–target correlations.The first phase produces decentralized response weights from original data; the second produces stationary and concentrated weights from the weighted data.
  • Temporal attention: Temporal attention selects encoder hidden states most related to the target value, extending dependence beyond relationships within a fixed time window.The context vector fuses encoder hidden states to represent temporal relationships across the window.
  • LSTM encoding and decoding: One-layer LSTM units encode series within both spatial-attention phases and temporal attention, while the decoder aligns context vectors with target values.This alignment helps maintain temporal relationships when updating the decoder hidden state.

5 Experiments

Experiments evaluate DSTP-RNN and DSTP-RNN-Ⅱ on four open datasets spanning energy, finance, environment, and medicine, using nine baselines and long-term prediction settings. Both proposed models outperform state-of-the-art methods across all datasets, with DSTP-RNN-Ⅱ generally performing better than DSTP-RNN.

  • Datasets and baselines: Nine baseline models, including classical nonlinear methods, kernel methods, and state-of-the-art neural networks, are compared at the same time-step size.The baselines include ARIMA, SVR, LSTM, GRU, Encoder-Decoder, Input-Attn-RNN, Temp-Attn-RNN, DA-RNN, and GeoMAN.
  • Module analysis: Attention-module interpretation identifies negative performance from the deep spatial mechanism.The experiments compare the effects of individual modules and provide interpretations of the attention-based RNN components.
  • Prediction results: DSTP-RNN and DSTP-RNN-Ⅱ exceed state-of-the-art models on all datasets, and DSTP-RNN-Ⅱ achieves better results than DSTP-RNN.The models also outperform methods using single-stage attention mechanisms across all datasets.
  • Prediction results: On NASDAQ100, the proposed models are very close to the ground truth with sufficient training data and outperform other baselines when training data is limited.This provides dataset-specific evidence for the models’ prediction performance under different training-data conditions.

NASDAQ100 Test Set

On the NASDAQ100 test set, the two-phase spatial attention structure outperforms state-of-the-art methods across prediction horizons, while DSTP-RNN-Ⅱ performs particularly well for short-term prediction. Adding target-series information to the final attention phase helps preserve temporal dependence between target and exogenous series.

  • Adding attention mechanisms improves performance over an encoder-decoder without attention, while DARNN and GeoMAN generally outperform single-attention models.
  • The two-phase attention structure outperforms state-of-the-art models across different prediction time steps on the NASDAQ100 dataset.
  • DSTP-RNN-Ⅱ generally achieves better short-term results than DSTP-RNN, indicating the importance of target and non-target information for detailed spatial correlations.
  • Concatenating target-series information in the second or last attention phase better maintains temporal dependence between target and exogenous series.
  • DSTP-RNN and DSTP-RNN-Ⅱ outperform state-of-the-art methods in both short-term and long-term prediction.

6 Conclusion & Future work

The paper concludes that DSTP-RNN and DSTP-RNN-Ⅱ improve spatial and temporal attention for long-term multivariate prediction, outperforming state-of-the-art methods across four datasets and prediction horizons. Their two-phase spatial attention and target-information integration produce more stable spatial correlations and stronger spatio-temporal modeling.

  • Conclusion: The proposed models enhance attention to spatial correlations and temporal relationships for long-term, multivariate time-series prediction.The paper presents DSTP-RNN and DSTP-RNN-Ⅱ as two novel attention-based RNN models.
  • Conclusion: DSTP-RNN and DSTP-RNN-Ⅱ achieved the best long-term prediction results across four datasets and different time-step predictions.The authors attribute this performance to enhanced attention mechanisms for spatial correlations and temporal relationships.
  • Conclusion: The first attention phase generates violent but decentralized weights, while the second produces stationary and concentrated weights.This two-phase response pattern yields more stable and effective extracted spatial correlations.
  • Conclusion: DSTP-RNN-Ⅱ and corresponding target-information concatenation improve capture of spatio-temporal relationships between target and exogenous series.The conclusion links this improvement to the effectiveness of both target and non-target information in human neuron signals.
Loading 1904.07464v1…