Source-linked AI summary
LSTM Fully Convolutional Networks for Time Series Classification
Fazle Karim, Somshubra Majumdar, Houshang Darabi, Shun Chen
TL;DR
Time-series classifiers often require heavy feature engineering, motivating end-to-end models with minimal preprocessing. The paper augments FCNs with LSTM or attention-LSTM branches and evaluates them across UCR benchmarks, reporting improved state-of-the-art performance and benefits from fine-tuning.
Problem
Existing time-series classification approaches often rely on heavy feature extraction and engineering, while the paper seeks effective end-to-end classification with minimal preprocessing.
Method
The paper augments a fully convolutional network with either an LSTM RNN or an attention LSTM sub-module, and applies iterative fine-tuning to trained models.
Results
Both proposed models outperform state-of-the-art models on at least 43 UCR datasets, with Wilcoxon comparisons yielding p-values below 0.05.
Takeaways & Limitations
LSTM-FCN improves FCN performance with a nominal parameter increase, while ALSTM-FCN additionally enables visual inspection of the LSTM decision process.
Abstract
from arXiv · showhide
Fully convolutional neural networks (FCN) have been shown to achieve state-of-the-art performance on the task of classifying time series sequences. We propose the augmentation of fully convolutional networks with long short term memory recurrent neural network (LSTM RNN) sub-modules for time series classification. Our proposed models significantly enhance the performance of fully convolutional networks with a nominal increase in model size and require minimal preprocessing of the dataset. The proposed Long Short Term Memory Fully Convolutional Network (LSTM-FCN) achieves state-of-the-art performance compared to others. We also explore the usage of attention mechanism to improve time series classification with the Attention Long Short Term Memory Fully Convolutional Network (ALSTM-FCN). Utilization of the attention mechanism allows one to visualize the decision process of the LSTM cell. Furthermore, we propose fine-tuning as a method to enhance the performance of trained models. An overall analysis of the performance of our model is provided and compared to other techniques.
I. INTRODUCTION
Time series classification has used feature engineering and ensembles, while deep models such as FCN and ResNet reduce preprocessing demands. This paper proposes LSTM- and attention-augmented FCN models for end-to-end classification across 85 UCR benchmarks.
- Prior approaches: Feature-based classifiers require substantial feature extraction and engineering to represent time-series patterns.Examples include BoW, TSBF, BOSS, BOSSVS, and WEASEL.
- Prior approaches: Ensemble methods achieve state-of-the-art performance by combining multiple classifiers or transformed representations.PROP integrates 11 classifiers, while COTE fuses 35 classifiers.
- Deep learning approaches: FCN and ResNet perform end-to-end classification without heavy preprocessing or feature engineering, unlike MCNN, whose performance depends strongly on preprocessing and hyperparameter tuning.The paper builds on FCN to reduce these preparation requirements.
- Proposed models: The paper augments FCN with either an LSTM RNN or an attention-equipped LSTM RNN, producing LSTM-FCN and ALSTM-FCN models.Both models also support visualization of class-relevant regions through convolutional Class Activation Maps, while Attention LSTM visualizes input regions.
- Evaluation: Both proposed models are tested on all 85 UCR time-series benchmarks and outperform most state-of-the-art models.The models are designed for end-to-end classification with minimal preprocessing.
II. BACKGROUND WORKS
Temporal convolutions extract evolving patterns from time-series signals through layered one-dimensional filters. The FCN branch uses convolution, normalization, activation, and global average pooling to support classification with fewer parameters.
- Temporal Convolutions: A temporal convolutional network receives a time-series signal whose length may vary across sequences.The input at time t is represented as a feature vector, and each layer can have its own number of time steps.
- Temporal Convolutions: Each convolutional layer applies one-dimensional filters that capture how input signals evolve over time.The filters are parameterized by layer-specific weights and biases, with d denoting filter duration.
- FCN architecture: The FCN feature-extraction block consists of convolution, batch normalization, and either ReLU or parametric ReLU activation.This block uses temporal convolutions to transform the input before classification.
- FCN architecture: Global average pooling reduces the parameter count before classification in fully convolutional networks.The proposed architecture uses the fully convolutional block as a feature-extraction module.
B. Recurrent Neural Networks
Recurrent neural networks model temporal behavior by updating hidden states through directed connections. LSTMs add gating and memory to address vanishing gradients, while attention is introduced to help with long-term dependencies.
- Recurrent Neural Networks: RNNs exhibit temporal behavior because directed connections update a hidden vector across time steps.The hidden state is used to make predictions and can serve as input to stacked RNN layers.
- Recurrent Neural Networks: The hidden state supports prediction, and stacking RNNs creates deeper recurrent architectures.The output distribution is produced with softmax, while sigmoid functions appear in recurrent updates.
- Long Short-Term Memory RNNs: LSTM RNNs address the vanishing-gradient problem of ordinary RNNs by incorporating gating functions into their state dynamics.Each LSTM maintains hidden and memory vectors that control state updates and outputs.
- Long Short-Term Memory RNNs: LSTM computations use sigmoid gates and a tanh candidate state combined through elementwise operations.The recurrent and projection matrices parameterize these gate and candidate-state computations.
- Attention motivation: Attention is introduced because LSTMs can struggle with long-term dependencies in long sequences.The paper connects attention mechanisms to this limitation of recurrent processing.
D. Attention Mechanism
Attention represents input positions through context vectors formed from weighted annotations. In this paper, attention-equipped LSTM cells replace ordinary LSTM cells to form the ALSTM-FCN architecture.
- Attention mechanism: An attention mechanism forms a context vector from annotations that summarize the input sequence while emphasizing positions near each relevant output position.Each annotation contains information about the whole input sequence with stronger focus on nearby portions.
- Attention mechanism: The context vector is computed as a weighted sum of encoder annotations.The weights determine how strongly each annotation contributes to the context representation.
- Attention mechanism: Each annotation weight is computed from an alignment score between an output position and an input position.The alignment model evaluates how well the input around position j matches the output around position i.
- Attention mechanism: A jointly trained feedforward alignment model produces soft alignments that permit gradient backpropagation through the attention computation.This alignment mechanism is described as part of the attention approach.
- ALSTM-FCN: Replacing LSTM cells with Attention LSTM cells converts the LSTM-FCN architecture into ALSTM-FCN.The architecture-level substitution is shown in Figure 1.
A. Network Architecture
The architecture augments a fully convolutional feature extractor with an LSTM or Attention LSTM branch, combining both representations for classification. A dimension shuffle lets the recurrent branch process the same input in a different view, which the paper identifies as key to improved performance.
- A. Network Architecture: The proposed models augment a fully convolutional block with an LSTM or Attention LSTM block followed by dropout.The fully convolutional and recurrent branches process the same time series input before their outputs are concatenated for classification.
- A. Network Architecture: The fully convolutional block uses temporal convolutions and global average pooling, while the recurrent branch contributes a separate learned representation.The convolutional block contains three stacked temporal convolutional blocks, and pooling reduces the representation before classification.
- A. Network Architecture: The fully convolutional and LSTM branches perceive the same time series in two different views.The convolutional branch treats a length-N series as N time steps, whereas the recurrent branch receives a transposed representation.
- A. Network Architecture: A dimension shuffle transposes the temporal dimension, presenting a univariate series of length N as N variables at a single time step for the LSTM branch.This transformation changes the recurrent branch's input interpretation without changing the underlying series.
- A. Network Architecture: When the LSTM instead received the univariate series over N time steps, performance fell because of rapid overfitting on short sequences and failure to learn long-term dependencies on long sequences.The paper identifies the dimension-shuffle approach as key to the architecture's enhanced performance.
C. Fine-Tuning of Models
Fine-tuning is applied after initial model training through repeated retraining on the original dataset with progressively smaller learning rates and batch sizes. The procedure is evaluated across the UCR datasets alongside the models’ training settings.
- C. Fine-Tuning of Models: Fine-tuning reuses knowledge from an initial model trained on the same dataset.The paper describes fine-tuning as transfer learning applied within the original dataset.
- C. Fine-Tuning of Models: Each fine-tuning repetition starts from the previous iteration’s weights, halves the learning rate, and halves batch size every alternate iteration.The process continues until the learning rate reaches 1e−4 and batch size reaches 32.
- C. Fine-Tuning of Models: The fine-tuning procedure is generally repeated K times, with K set as 5.Each repetition retrains the model on the original dataset using the updated settings.
- C. Fine-Tuning of Models: Experiments used all 85 UCR datasets, generally trained for 2000 epochs with an initial batch size of 128 and an 80% dropout rate after the recurrent layer.The number of LSTM cells was selected by searching from 8 to 128 cells.
- C. Fine-Tuning of Models: Models were trained with Adam using an initial learning rate of 1e−3 and a final learning rate of 1e−4, without additional preprocessing on the UCR datasets.All models were fine-tuned, and Table I reports scores before and after fine-tuning.
A. Evaluation Metrics
The paper evaluates the proposed models with accuracy, rank-based statistics, and mean per class error. These measures assess both predictive performance and comparative ranking across datasets.
- A. Evaluation Metrics: Evaluation uses accuracy, rank-based statistics, and mean per class error.The metric set follows the evaluation framework stated by Wang et al.
- A. Evaluation Metrics: Rank-based evaluation includes arithmetic rank, geometric rank, and the Wilcoxon signed rank test.The arithmetic and geometric ranks summarize dataset ranks, while the Wilcoxon test compares median ranks.
- A. Evaluation Metrics: The Wilcoxon signed rank test compares the proposed model’s median rank with the median rank of existing state-of-the-art models.This provides a statistical comparison of model rankings across datasets.
- A. Evaluation Metrics: Mean Per Class Error is the arithmetic mean of the per-class error.The paper uses MPCE as an additional performance measure alongside accuracy and rank-based statistics.
B. Results
Across the UCR benchmarks, both proposed models generally outperform state-of-the-art methods, while fine-tuning maintains or improves accuracy and is more effective for LSTM-FCN.
- Both ALSTM-FCN and LSTM-FCN, with and without fine-tuning, outperform state-of-the-art models on at least 43 datasets.
- Fine-tuning maintains or improves overall accuracy for both proposed models.
- Fine-tuning reduces MPCE by 0.0035 for LSTM-FCN and 0.0007 for ALSTM-FCN.
- Fine-tuning is more effective for LSTM-FCN than ALSTM-FCN on the UCR datasets.The paper attributes this discrepancy to LSTM-FCN having fewer total parameters and therefore a lower rate of overfitting.
- ALSTM-FCN provides a visual representation of the attention vector, while its fine-tuning effect is less significant than for the general LSTM model.
V. CONCLUSION & FUTURE WORK
The proposed LSTM-FCN and ALSTM-FCN models improve deep-neural-network time-series classification with nominal preprocessing and parameter increases. Future work includes explaining ALSTM-FCN’s weaker performance on some datasets and extending the models to multivariate series.
- LSTM-FCN augments FCN performance with a nominal increase in the number of parameters.
- The proposed models are trainable end-to-end with nominal preprocessing and achieve improved performance relative to FCN models.
- ALSTM-FCN enables visual inspection of the LSTM RNN decision process and provides a strong baseline independently.
- Further research is needed to understand why ALSTM-FCN fails to match general LSTM performance on some datasets.
- Extension of the proposed models to multivariate time series has not been explored.