Source-linked AI summary
Evaluation of deep learning models for multi-step ahead time series prediction
Rohitash Chandra, Shaurya Goyal, Rishabh Gupta
TL;DR
The paper addresses limited evidence on deep learning models for multi-step time series prediction by evaluating several architectures and training algorithms on benchmark univariate datasets. Across the reported problems, bidirectional and encoder-decoder LSTMs provide the best performance, while RNNs show the lowest confidence intervals.
Problem
Prior work had limited evaluation of deep learning models, particularly LSTM variants, for multi-step-ahead time series prediction as the horizon increases.
Method
The study compares recurrent, LSTM, bidirectional LSTM, encoder-decoder LSTM, CNN, and simple neural networks trained with SGD or Adam on benchmark univariate series.
Results
Bidirectional and encoder-decoder LSTM networks provide the best performance across the simulated and real-world time series problems.
Takeaways & Limitations
Bidirectional and encoder-decoder LSTMs are applicable to multi-step time series mapping, with past and future state information identified as a key feature of their performance.
Abstract
from arXiv · showhide
Time series prediction with neural networks has been the focus of much research in the past few decades. Given the recent deep learning revolution, there has been much attention in using deep learning models for time series prediction, and hence it is important to evaluate their strengths and weaknesses. In this paper, we present an evaluation study that compares the performance of deep learning models for multi-step ahead time series prediction. The deep learning methods comprise simple recurrent neural networks, long short-term memory (LSTM) networks, bidirectional LSTM networks, encoder-decoder LSTM networks, and convolutional neural networks. We provide a further comparison with simple neural networks that use stochastic gradient descent and adaptive moment estimation (Adam) for training. We focus on univariate time series for multi-step-ahead prediction from benchmark time-series datasets and provide a further comparison of the results with related methods from the literature. The results show that the bidirectional and encoder-decoder LSTM network provides the best performance in accuracy for the given time series problems.
I. INTRODUCTION
Multi-step prediction must remain accurate as the horizon grows despite chaotic and noisy data. The paper evaluates selected deep learning architectures against simpler neural networks and related methods on benchmark univariate series.
- Multi-step prediction produces a series of future estimates, with the prediction horizon defining how far ahead the model forecasts.
- Increasing horizons make low-error forecasting difficult because time series can be chaotic and noisy.
- Recurrent neural networks model temporal sequences, while LSTM networks address canonical RNN limitations in learning long-term dependencies.
- Bidirectional RNNs use separate forward and backward hidden layers so outputs can receive information from past and future states.
- The paper addresses limited prior evaluation of deep learning methods for multi-step time series prediction, especially as the prediction horizon increases.
- The study compares standard, bidirectional, and encoder-decoder LSTMs and CNNs with simple neural networks trained using SGD or Adam.
A. MULTI-STEP TIME SERIES PREDICTION
Multi-step forecasting literature includes recursive, direct, and hybrid strategies alongside neural, probabilistic, and application-specific methods. The paper also reconstructs time series into embedded phase spaces before prediction.
- Recursive strategies feed one-step predictions into later horizons, while direct strategies model multiple future targets separately.
- Hybrid approaches combine recursive and direct forecasts, with ensemble methods reported to outperform standalone strategies in related work.
- Prior applications span physiological signals, flood forecasting, emissions, photovoltaic power, earthquakes, and CPU load.
- RNNs are used for temporal information, while Bayesian RNNs can provide uncertainty quantification in predictions.
- Time series are embedded as delayed-coordinate vectors using a time delay T and embedding dimension D before multi-step prediction.
B. SHALLOW LEARNING VIA SIMPLE NEURAL NETWORKS
The paper treats backpropagation networks and multilayer perceptrons as simple neural networks and compares SGD with Adam training. Simple RNNs instead retain temporal context through recurrent hidden states.
- Simple neural networks are defined as backpropagation networks and multilayer perceptrons typically trained with stochastic gradient descent.
- SGD uses one learning rate for all weight updates and keeps that rate unchanged during training.
- Adam adapts the learning rate separately for each parameter using first and second gradient moments.
- An Elman RNN uses a context layer that stores previous hidden-layer outputs to propagate temporal information into future states.
- Simple RNN training commonly uses backpropagation through time, which propagates error through states defined across time.
D. LSTM NETWORKS
LSTM networks address simple RNN difficulty with long-term dependencies through memory cells and gates. Bidirectional variants additionally preserve information from both past and future directions.
- D. LSTM NETWORKS: LSTM networks use memory cells and gates to improve retention of long-term dependencies in temporal sequences.
- D. LSTM NETWORKS: The LSTM input, forget, and output gates regulate the unit’s internal memory and hidden state.
- D. LSTM NETWORKS: Bidirectional RNNs process sequences with independent forward and backward recurrent layers connected to a shared output layer.
- D. LSTM NETWORKS: Bidirectional LSTMs run information from past to future and future to past, preserving state information from both directions.
F. ENCODER-DECODER LSTM NETWORKS
The section presents encoder-decoder LSTM networks as sequence-to-sequence models for handling variable-length inputs and outputs in multi-step time series prediction, alongside the paper’s benchmark evaluation setup.
- Encoder-decoder LSTM networks: Encoder-decoder LSTM networks encode input sequences into a latent representation and decode that representation into output sequences.The encoder computes hidden states sequentially, while the decoder defines a distribution over the output sequence.
- Encoder-decoder LSTM networks: In multi-step time series prediction, encoder-decoder LSTMs accommodate variable-length input and output sequences.
- Experimental design: The evaluation uses simulated and real-world benchmark time series, including Mackey-Glass, Lorenz, Henon, Rossler, Sunspot, Lazer, and ACI-financial data.The simulated series are listed in the experimental description, while the real-world series include solar-activity, Santa Fe, and financial data.
- Experimental design: RMSE is the main performance measure used across different prediction horizons.The metric compares observed and predicted data, with N denoting the length of the observed data.
B. RESULTS
Across benchmark real-world and simulated series, deep learning models generally outperform simple neural networks, although errors usually increase with prediction horizon. LSTM variants perform strongly overall, with ED-LSTM and BD-LSTM often leading, while CNN can outperform LSTM on selected simulated series.
- Prediction error generally increases as the prediction horizon increases across real-world and simulated problems.
- On ACI-Finance, deep learning methods perform better than FNN-SGD and FNN-Adam, with LSTM providing the best overall performance.
- On Sunspot, LSTM, ED-LSTM, and BD-LSTM outperform the other methods, while FNN-SGD performs worst and RNN outperforms CNN.
- Deep learning methods generally outperform simple neural networks on the Lorenz, Mackey-Glass, and Rossler simulated series.
- CNN outperforms LSTM on the Mackey-Glass and Rossler simulated time series, the first reported instances of this pattern in the results.
- ED-LSTM and BD-LSTM provide the best performance across most simulated series, including Henon, Mackey-Glass, Lorenz, and Rossler.
- Lazer is an outlier: LSTM performance improves as the prediction horizon increases, whereas other real-world and simulated problems generally deteriorate.
C. COMPARISON WITH THE LITERATURE
The study compares its methods with related literature while cautioning that differences in models, preprocessing, error measures, and reporting make the comparison unfair. Its deep learning methods perform best across most prediction horizons, and FNN-Adam surpasses CMTL in nearly all problems.
- Literature comparisons are not fully fair because prior studies use different models, data processing, error measures, and reporting practices.Some prior work reports only the best experimental run rather than mean and standard deviation.
- The proposed deep learning methods achieve the best performance for most prediction horizons in the compared problems.
- FNN-Adam overtakes CMTL in all time-series problems except one.
V. DISCUSSION
Across the benchmark problems, BD-LSTM and ED-LSTM rank among the strongest models, while RNN and CNN show the most robust performance across initializations. Prediction error generally increases with horizon, although Lazer is an outlier and CNN surpasses LSTM on two simulated series.
- BD-LSTM and ED-LSTM provide one of the best accuracy performances across the time-series problems.
- RNN has the lowest confidence intervals across problems, followed by CNN, indicating more robust accuracy across different weight initializations.
- Prediction error generally increases with horizon because direct multi-step prediction relies on current values while the information gap grows.
- BD-LSTMs process forward and backward sequence information, preserving past and future states in a way associated with better multi-step prediction performance.
VI. CONCLUSION AND FUTURE WORK
The paper concludes that encoder-decoder and bidirectional LSTM networks perform best across simulated and real-world multi-step time-series problems. Future evaluations should extend beyond univariate series to multivariate, spatiotemporal, and additional real-world forecasting tasks.
- Encoder-decoder and bidirectional LSTM networks provide the best performance for both simulated and real-world time-series problems.
- Future work should evaluate the deep learning models on multivariate time-series prediction problems.
- The authors also identify spatiotemporal forecasting and applications such as air pollution and energy forecasting as feasible extensions.
SOFTWARE AND DATA
The paper provides an open-source Python implementation and accompanying data for the evaluated methods.
- Open-source Python implementation and data are provided for the respective methods.