Source-linked AI summary
A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, Garrison Cottrell
TL;DR
The paper addresses NARX models’ difficulty capturing long-term temporal dependencies and selecting relevant driving series. It proposes DA-RNN, which combines input attention over driving series with temporal attention over encoder states. Experiments on SML 2010 and NASDAQ 100 Stock show that DA-RNN outperforms state-of-the-art time-series prediction methods.
Problem
Existing NARX approaches have difficulty capturing long-term temporal dependencies and selecting relevant driving series for prediction.
Method
DA-RNN combines an encoder input-attention mechanism that selects driving series with a decoder temporal-attention mechanism that selects encoder hidden states across time.
Results
DA-RNN outperformed state-of-the-art time-series prediction methods on the SML 2010 and NASDAQ 100 Stock datasets.
Takeaways & Limitations
DA-RNN adaptively selects relevant input features and captures long-term temporal dependencies while providing an interpretable attention scheme.
Takeaways & Limitations
The paper identifies DA-RNN’s potential use in computer vision and leaves ranking and binary coding for future work.
Abstract
from arXiv · showhide
The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them can capture the long-term temporal dependencies appropriately and select the relevant driving series to make predictions. In this paper, we propose a dual-stage attention-based recurrent neural network (DA-RNN) to address these two issues. In the first stage, we introduce an input attention mechanism to adaptively extract relevant driving series (a.k.a., input features) at each time step by referring to the previous encoder hidden state. In the second stage, we use a temporal attention mechanism to select relevant encoder hidden states across all time steps. With this dual-stage attention scheme, our model can not only make predictions effectively, but can also be easily interpreted. Thorough empirical studies based upon the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can outperform state-of-the-art methods for time series prediction.
1 Introduction
The paper frames NARX prediction as learning nonlinear mappings from target history and exogenous series, while addressing limitations in long-range dependency modeling and feature selection with dual-stage attention.
- 1 Introduction: Traditional ARMA variants cannot model nonlinear relationships or differentiate among exogenous input terms.
- 1 Introduction: NARX predicts the current target from previous target values and current and past values of multiple driving series.It learns a nonlinear mapping ˆy_t = F(y_1, y_2, ···, y_{t−1}, x_1, x_2, ···, x_t).
- 1 Introduction: Many existing nonlinear approaches use predefined forms, while recurrent neural networks offer greater flexibility for capturing nonlinear relationships.
- 1 Introduction: Encoder-decoder networks deteriorate as input sequences lengthen, creating a problem when predictions rely on long target and driving-series segments.
- 1 Introduction: DA-RNN uses input attention to select relevant driving series and temporal attention to select relevant encoder states across time.The two mechanisms are integrated within a recurrent architecture for time-series prediction.
2 Dual-Stage Attention-Based RNN
This section formulates NARX prediction and presents DA-RNN, which uses input attention to select driving series and temporal attention to select encoder states. LSTM-based encoder and decoder components model long-term dependencies and produce the final prediction.
- Notation and Problem Statement: NARX learns a nonlinear mapping from previous target values and current or past driving-series values to the current target value.The target estimate is written as ˆyT = F(y1, · · · , yT −1, x1, · · · , xT ).
- Encoder with input attention: The encoder uses an LSTM to map each input vector to a hidden state and capture long-term dependencies through its memory cell.The LSTM cell state sums activities over time, helping address vanishing gradients.
- Encoder with input attention: DA-RNN uses input attention to adaptively select relevant driving series by referring to the encoder's previous hidden and cell states.The attention weights measure feature importance, are normalized by softmax, and produce a selectively extracted input for the encoder.
- Decoder with temporal attention: The decoder applies temporal attention to weight encoder hidden states across all time steps and forms a time-specific context vector.The context vector is combined with the target-series inputs before updating the decoder LSTM.
- Training procedure: The final prediction is produced from the decoder's last hidden state and context vector, with the model trained by minibatch Adam optimization.The training setup uses minibatches of 128 and a learning rate beginning at 0.001, reduced by 10% after every 10000 iterations.
3 Experiments
Experiments evaluate DA-RNN on indoor-temperature and NASDAQ stock datasets against multiple baselines, then examine its robustness, interpretability, and parameter sensitivity. DA-RNN achieves the strongest reported prediction metrics, suppresses noisy inputs, captures long-term dependencies, and is relatively robust to window length.
- Datasets and Setup: Experiments use SML 2010 for indoor-temperature forecasting and NASDAQ 100 Stock for predicting the NASDAQ 100 index from driving stock series.The studies compare methods across two datasets with distinct target and exogenous-series settings.
- Parameter Settings and Evaluation Metrics: DA-RNN is compared with ARIMA, NARX RNN, Encoder-Decoder, Attention RNN, and Input-Attn-RNN using RMSE, MAE, and MAPE.The RNN-based baselines receive driving series and previous target values, while ARIMA considers only the target history.
- Time Series Prediction: DA-RNN achieves the best MAE, MAPE, and RMSE across both datasets, while Input-Attn-RNN consistently outperforms Encoder-Decoder and Attention RNN.The results attribute the stronger performance to combining input attention with temporal attention.
- Time Series Prediction: DA-RNN generally fits NASDAQ 100 ground truth better than Encoder-Decoder and Attention RNN in the visual comparison.The comparison uses hidden-state sizes of 128 for the baselines and 64 for DA-RNN.
- Interpretation: DA-RNN assigns larger weights to original driving series and smaller weights to randomly permuted noisy series, indicating online input selection and noise suppression.The experiment adds 81 noisy series to the 81 original NASDAQ driving series.
- Parameter Sensitivity: When the window length T is relatively large, DA-RNN significantly outperforms Input-Attn-RNN, while remaining relatively robust when T is too short or too long.The comparison evaluates T values of 3, 5, 10, 15, and 25 across both datasets.
4 Conclusion
DA-RNN combines input and temporal attention to select relevant driving series and capture long-term temporal dependencies in time series. Experiments on SML 2010 and NASDAQ 100 show it outperforms state-of-the-art methods, while its broader use remains prospective.
- DA-RNN uses an encoder with input attention and a decoder with temporal attention to select relevant inputs and capture long-term dependencies.
- Extensive experiments on the SML 2010 and NASDAQ 100 Stock datasets demonstrated that DA-RNN outperformed state-of-the-art time series prediction methods.
- DA-RNN may also serve as a general feature learning tool for computer vision tasks, with ranking and binary coding identified as future applications.