Source-linked AI summary
EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction
Youru Li, Zhenfeng Zhu, Deqiang Kong, Hua Han, Yao Zhao
TL;DR
Traditional LSTMs capture long-term dependencies but have limited ability to weight sub-window features across multiple time steps. EA-LSTM adds evolutionary attention learning and competitive random search for attention-layer parameters, and experiments report competitive or higher prediction accuracy than baseline and state-of-the-art methods. The formulation assumes historical sliding-window inputs, with discrete historical targets for classification.
Problem
Traditional LSTMs insufficiently assign different attention levels to sub-window features across multiple time steps during temporal modeling.
Method
EA-LSTM adds an attention layer for importance-based sampling and uses competitive random search to optimize its attention parameters.
Results
EA-LSTM achieved higher prediction accuracy than baseline methods and higher predicted precision than DA-RNN on the SML 2010 dataset.
Takeaways & Limitations
Attention-based models can be enhanced by improving attention-layer training, not only by introducing a more complex attention mechanism.
Takeaways & Limitations
The formulation assumes historical sliding-window feature sequences and uses discrete historical targets for classification problems.
Abstract
from arXiv · showhide
Time series prediction with deep learning methods, especially long short-term memory neural networks (LSTMs), have scored significant achievements in recent years. Despite the fact that the LSTMs can help to capture long-term dependencies, its ability to pay different degree of attention on sub-window feature within multiple time-steps is insufficient. To address this issue, an evolutionary attention-based LSTM training with competitive random search is proposed for multivariate time series prediction. By transferring shared parameters, an evolutionary attention learning approach is introduced to the LSTMs model. Thus, like that for biological evolution, the pattern for importance-based attention sampling can be confirmed during temporal relationship mining. To refrain from being trapped into partial optimization like traditional gradient-based methods, an evolutionary computation inspired competitive random search method is proposed, which can well configure the parameters in the attention layer. Experimental results have illustrated that the proposed model can achieve competetive prediction performance compared with other baseline methods.
1 Introduction
Time series prediction must model complex temporal dependencies, motivating attention-enhanced LSTMs and evolutionary optimization of attention weights. EA-LSTM uses competitive random search to improve attention-layer training and reports higher accuracy than baseline methods.
- Time series prediction supports analysis and decision-making across clinical, financial, traffic, and human-action applications.
- Complex sequence dependence among input variables makes suitable predictive modeling crucial for real data.
- The model adds an attention layer to multivariate LSTMs, weighting periods by their importance within a sliding time window.
- Traditional LSTMs capture long-term dependencies but insufficiently assign different attention levels to sub-window features across multiple time steps.
- Competitive random search trains attention parameters instead of using gradient-based optimization, aiming to avoid local optima through broader search.
- Experiments on real time-series datasets covering regression and classification report higher prediction accuracy than baseline methods.
2 Preliminaries
The paper formulates time-series prediction over historical sliding-window features and targets. It learns a nonlinear mapping from the feature sequence to the predicted value, with discrete historical targets for classification.
- Time-series prediction uses a historical sequence of values as input and can be framed as regression or classification.
- The training input is a sliding-window feature matrix X=(X1, X2, ..., XT), with Xt belonging to X and time-step length L.
- Historical target values are represented as y=(y1, y2, ..., yT−1); for classification problems, these targets are discrete.
- A nonlinear mapping f is learned from the history-driven sequence feature X and corresponding target y to produce predicted value ỹT.
3 Methodology
The methodology augments LSTMs with attention-based sampling and trains the attention layer using competitive random search. Collaborative training transfers attention parameters and loss feedback while standard optimization learns the remaining LSTM parameters.
- Evolutionary Attention-Based LSTM: EA-LSTM adds an attention layer to LSTMs to assign different weights to periods in a sliding time window.The weighted input is then fed into LSTM networks for temporal modeling.
- Collaborative Training: Collaborative training shares parameters and loss feedback between EA-LSTM and competitive random search, using mini-batch SGD with Adam for non-attention LSTM parameters.Parameters outside the attention layer are learned with backpropagation through time.
- Evolutionary Attention-Based LSTM: The LSTM cell uses input, forget, cell-state, and output computations to model nonlinear mappings from weighted temporal inputs.The forget gate determines which information from the previous time step is dropped, while the hidden output is used for prediction.
- Competitive Random Search: Competitive random search generates attention-parameter combinations through binary encoding, selection, recombination, mutation, and repeated optimization-space rebuilding.Its rebuilding process controls the direction of random searching and produces new-generation attention-weight spaces.
- Competitive Random Search: Candidate attention weights are evaluated by prediction loss, champion subsets are selected, and their combinations are traversed repeatedly to form new candidates.The search receives error feedback from the prediction network when evaluating attention-weight subsets.
- Competitive Random Search: The search operators randomly select subsegments, recombine binary-coded genes, and mutate a random index before generating new attention-weight candidates.The number of selected subsections is not fixed, and recombination may occur at even or odd indices according to random judgment.
4 Experiments
The experiments evaluate EA-LSTM across two regression datasets and the MSR Action3D recognition dataset, using tuned LSTM settings and comparisons with traditional, recurrent, and attention-based baselines.
- Datasets: Experiments cover Beijing PM2.5 and SML2010 regression datasets plus MSR Action3D human action recognition.The regression datasets support time-series prediction evaluation, while MSR tests temporal modeling in classification.
- Parameter Settings: The basic LSTM is tuned over time steps, hidden units, and batch size using dataset-specific grid-search ranges.The search includes L, m, and b, with different ranges for regression and MSR experiments.
- Parameter Settings and Sensitivity: EA-LSTM hyperparameters are selected after grid searching, with results summarized through RMSE box-whiskers over five testing results.The box-whisker displays middle value, quartiles, minimum, maximum, and outliers.
- Training Attention Layer: Competitive random search uses an optimization space of 36, encoding length 6, six champion weights, and 20 epochs.The attention-search configuration is chosen to balance solving efficiency.
- Performance Comparison: EA-LSTM improves performance over SVR, GBRT, RNN, LSTM, and GRU baselines on both public regression benchmarks.Baseline results are averaged across five training and testing runs to reduce random errors.
- Performance Comparison: On SML2010, EA-LSTM achieves higher predicted precision than DA-RNN and shows that attention models can benefit from improved attention-layer training.The comparison uses identical training, validation, and testing splits.
5 Conclusion
EA-LSTM combines evolutionary attention learning with competitive random search to capture both long-term dependencies and local information within sampling windows. Experiments report competitive prediction performance against state-of-the-art methods, while the paper also presents evolutionary computation for training difficult neural-network substructures.
- EA-LSTM uses competitive random search to train an evolutionary attention-based LSTM for time series prediction.The attention parameters are confirmed during temporal relationship mining for importance-based sampling.
- The model captures long-term dependencies while utilizing local information within one sampling window through varied attention distributions.
- Experiments show competitive prediction performance compared with state-of-the-art methods.
- The paper introduces evolutionary computation to substructure training in deep neural networks and reports good performance.The stated motivation is improving neural networks that are hard to train.
6 Ackowledgments
The work was jointly sponsored by Chinese national and university funding programs.
- The research was jointly sponsored by three Chinese funding programs and lists their grant numbers.The sponsors include national research, natural science, and central-university funding bodies.