PaperlayerView exact revision on arXiv

Source-linked AI summary

Insights into LSTM Fully Convolutional Networks for Time Series Classification

Fazle Karim, Somshubra Majumdar, Houshang Darabi

arXiv:1902.10756v3cs.LGstat.ML

TL;DR

The paper addresses the lack of evidence explaining why LSTM-FCN and ALSTM-FCN perform well on time series classification. It conducts 3627 ablation experiments, finding that conjoined recurrent and convolutional blocks perform better and that normalization and dimension-shuffle choices affect results.

  • Problem

    The paper investigates why LSTM-FCN and ALSTM-FCN perform well on UCR time series classification benchmarks.

  • Method

    The study performs 3627 ablation experiments comparing normalization schemes, model blocks, dimension shuffle, and recurrent-block replacements.

  • Results

    Conjoined LSTM/ALSTM and FCN features improve performance, whole-dataset z-normalization differs statistically from per-sample normalization, and dimension shuffle improves LSTM-FCN overall.

  • Takeaways & Limitations

    The ablations support using recurrent and convolutional blocks together, applying dimension shuffle before LSTM, and choosing whole-dataset normalization when training data represents the global population.

Abstract

from arXiv · show

Long Short Term Memory Fully Convolutional Neural Networks (LSTM-FCN) and Attention LSTM-FCN (ALSTM-FCN) have shown to achieve state-of-the-art performance on the task of classifying time series signals on the old University of California-Riverside (UCR) time series repository. However, there has been no study on why LSTM-FCN and ALSTM-FCN perform well. In this paper, we perform a series of ablation tests (3627 experiments) on LSTM-FCN and ALSTM-FCN to provide a better understanding of the model and each of its sub-module. Results from the ablation tests on ALSTM-FCN and LSTM-FCN show that the LSTM and the FCN blocks perform better when applied in a conjoined manner. Two z-normalizing techniques, z-normalizing each sample independently and z-normalizing the whole dataset, are compared using a Wilcoxson signed-rank test to show a statistical difference in performance. In addition, we provide an understanding of the impact dimension shuffle has on LSTM-FCN by comparing its performance with LSTM-FCN when no dimension shuffle is applied. Finally, we demonstrate the performance of the LSTM-FCN when the LSTM block is replaced by a GRU, basic RNN, and Dense Block.

I. INTRODUCTION

Time series classification assigns discrete labels to sequential sensor data, with efficiency and accuracy as central challenges. The paper situates LSTM-FCN and ALSTM-FCN among state-of-the-art deep models and reviews temporal convolutions used for time-series signals.

  • I. INTRODUCTION: Time series classification maps sequentially ordered data points to discrete class labels.Inputs are represented as time series signals with feature vectors at successive time steps.
  • I. INTRODUCTION: Time-series signals arise from sensors monitoring domains including weather, stock markets, EEG, and ECG.
  • I. INTRODUCTION: Efficient and accurate classification remains a central challenge in time series classification.
  • I. INTRODUCTION: LSTM-FCN and ALSTM-FCN are state-of-the-art deep learning models for classifying datasets from the UCR repository.
  • I. INTRODUCTION: Temporal convolution networks apply one-dimensional filters across layers to discover signal evolution over time.

B. Recurrent Neural Networks

Recurrent neural networks maintain hidden states across time steps to model sequential data, while LSTM and GRU gating mechanisms address vanishing gradients. The reviewed formulations define recurrent predictions, LSTM memory updates, and GRU gates.

  • B. Recurrent Neural Networks: RNNs preserve a hidden vector as a state that updates at each time step.
  • B. Recurrent Neural Networks: An RNN prediction is computed from the previous hidden state and normalized with a softmax operation.
  • B. Recurrent Neural Networks: RNNs can be stacked by feeding one layer’s hidden-state output into the next layer.
  • B. Recurrent Neural Networks: LSTM and GRU architectures address the vanishing-gradient issue affecting RNNs.
  • B. Recurrent Neural Networks: LSTM cells use gating functions and memory vectors to regulate state updates and outputs.
  • B. Recurrent Neural Networks: GRUs use update and reset gates with fewer gates and parameters than LSTMs.

E. Fully Connected (Dense) Layer

A fully connected layer transforms an input vector through multiplication by a trainable weight matrix, optionally adds a trainable bias, and applies a nonlinear activation.

  • E. Fully Connected (Dense) Layer: A fully connected layer computes a dense matrix multiplication of the input vector with trainable parameters.
  • E. Fully Connected (Dense) Layer: The layer may add a trainable bias and apply a nonlinear activation such as ReLU, sigmoid, or tanh.
  • E. Fully Connected (Dense) Layer: The layer output is determined by a weight matrix, bias vector, and activation function.

III. EXPERIMENTS

The experiments evaluate LSTM-FCN and ALSTM-FCN on UCR benchmark datasets using controlled training procedures, z-normalized inputs, and classification accuracy and MPCE. The study compares model variants and reports performance against baseline models.

  • III. EXPERIMENTS: LSTM-FCN and ALSTM-FCN are trained on released UCR benchmark datasets with grid-searched LSTM cell sizes of 8, 64, or 128.Models are trained for 2000 epochs with batch size 128 and dropout after recurrent layers.
  • III. EXPERIMENTS: Training uses Adam with learning-rate scheduling, He initialization for convolutional layers, and zero-padding for variable-length series.
  • III. EXPERIMENTS: Inputs are z-normalized, and models are evaluated using classification accuracy and mean-per-class-error.MPCE is defined as the average error of each class across datasets.
  • III. EXPERIMENTS: The study makes model codes and weights available for the evaluated experiments.
  • III. EXPERIMENTS: Table I compares LSTM-FCN and ALSTM-FCN with baseline models, marking matches or improvements over state-of-the-art results.

IV. DATASET ABLATION TEST

The study compares dataset-level and per-sample z-normalization on 43 previously unprocessed UCR datasets. Dataset-level normalization performs better on more datasets, but the recommended scheme depends on whether training data represents the broader population.

  • Dataset-level z-normalization performs better on 34 datasets with LSTM-FCN and 30 with ALSTM-FCN than per-sample normalization.The comparison uses the 43 UCR datasets that were not previously padded, normalized, or pre-processed.
  • 4.57e-07 was the Wilcoxon signed-rank p-value, below the corrected significance level of 0.025.The authors therefore conclude that the two normalization schemes perform differently.
  • Whole-dataset normalization is recommended only when the training set sufficiently represents the dataset’s global population.Without such prior information or domain knowledge, the authors consider per-sample normalization safer.

V. MODEL ABLATION TESTS

The model ablation tests examine how the LSTM-FCN and ALSTM-FCN components contribute to performance. They assess individual components, a linear baseline, and empirical and statistical comparisons across selected UCR datasets.

  • The ablation study evaluates LSTM-FCN and ALSTM-FCN on 61 sample-z-normalized UCR datasets.The selected datasets outperform the SOTA non-ensemble classifiers BOSS and WEASEL.
  • All Wilcoxon signed-rank tests use a significance level of 0.05.The paper states this testing convention before presenting its null and alternative hypotheses.
  • The study examines individual model components, their comparison with a linear baseline, and empirical and statistical performance differences.This analysis focuses on the role of the LSTM recurrent module alongside the FCN block.

A. Fully Convolutional Block

The fully convolutional block uses stacked temporal convolutions to transform time-series inputs into features intended to separate classes. Linear SVM tests show that these extracted features outperform raw signals on many datasets.

  • The FCN block contains three stacked temporal convolutional blocks with 128, 256, and 128 filters.A figure visualizes a sample after transformation by randomly selected filters from the convolutional blocks.
  • CNN filters across the layers act as feature extractors that transform data into more separable classes.The paper describes early filters as capable of learned noise reduction and later filters as further transforming the data.
  • A linear SVM applied to FCN features performs better than one applied to raw signals on 41 LSTM-FCN datasets and 45 ALSTM-FCN datasets.

B. LSTM/ALSTM Recurrent Block

The recurrent block’s standalone features contribute little after dimensional shuffle, unlike the stronger separation observed from the combined architecture. Linear SVM comparisons indicate that LSTM-derived features alone generally do not make the classes linearly separable.

  • LSTM-block features outperform raw-input features on only 19 datasets, while ALSTM-block features do so on only 4 datasets.These comparisons use tuned linear SVM classifiers applied to the extracted features versus raw input.
  • Dimensional shuffle before the recurrent block limits the standalone contribution of its extracted features to overall performance.
  • The standalone LSTM block does not separate the data into linearly separable classes.

C. LSTM/ALSTM Concatenated With FCN Block

The ablation results show that LSTM/ALSTM and FCN features are more effective when combined, while dimension shuffle changes how the recurrent block contributes to classification.

  • C. LSTM/ALSTM Concatenated With FCN Block: Concatenating LSTM/ALSTM features with FCN features produces a more robust representation that better separates dataset classes.The combined representation improves model performance over either component alone.
  • C. LSTM/ALSTM Concatenated With FCN Block: 49 datasets favored combined ALSTM-FCN features over either ALSTM or FCN features, with p-value 1.34e-08.
  • C. LSTM/ALSTM Concatenated With FCN Block: 54 datasets favored combined LSTM-FCN features over either LSTM or FCN features, with p-value 1.22e-08.The corrected significant alpha value was 0.02.
  • C. LSTM/ALSTM Concatenated With FCN Block: With dimension shuffle, the LSTM processes one time step containing N variables rather than N time steps containing one variable.This transposes an input of N time steps and 1 variable into N variables and 1 time step.
  • C. LSTM/ALSTM Concatenated With FCN Block: Dimension-shuffled LSTM-FCN outperformed the non-shuffled model in 258 of 384 experiments, tied in 27, and lost in 99.Average accuracy improved by 6.00% in wins and decreased by 5.26% in losses; the Wilcoxon p-value was 3.69E-17.

E. Replacing LSTM with GRU, RNN, and a Dense Layer

The study replaces the dimension-shuffled LSTM block with GRU, basic RNN, or sigmoid Dense blocks across matched experiments. LSTM-FCN statistically outperforms the alternatives, although GRU-FCN does not statistically outperform Dense-FCN.

  • E. Replacing LSTM with GRU, RNN, and a Dense Layer: The replacement study evaluates GRU-FCN, RNN-FCN, and sigmoid Dense-FCN using 384 experiments per model across 128 datasets.Each recurrent alternative uses 8, 64, or 128 cells; Dense-FCN uses the corresponding unit counts.
  • E. Replacing LSTM with GRU, RNN, and a Dense Layer: Wilcoxon signed-rank tests show dimension-shuffled LSTM-FCN statistically outperforms GRU-FCN, RNN-FCN, and Dense-FCN.
  • E. Replacing LSTM with GRU, RNN, and a Dense Layer: LSTM-FCN outperformed Dense-FCN in 231 experiments, tied in 35, and performed worse in 118.
  • E. Replacing LSTM with GRU, RNN, and a Dense Layer: GRU-FCN did not statistically outperform Dense-FCN, with 160 wins, 49 ties, and 175 losses across 384 experiments.Each experiment was run once, so random initialization may introduce deviation across repeated runs.

VI. CONCLUSION & FUTURE WORK

Ablation experiments clarify how LSTM-FCN and ALSTM-FCN components contribute to time-series classification. The study also identifies unresolved robustness issues across UCR datasets and directions for future deployment and streaming use.

  • 3627 ablation experiments provide a better understanding of LSTM-FCN, ALSTM-FCN, and their sub-modules.
  • The LSTM/ALSTM and FCN blocks perform better when applied together, rather than as separate components.
  • LSTM-FCN outperforms GRU-FCN, RNN-FCN, and Dense-FCN when the recurrent block is replaced.
  • Future work should examine underperformance on a few UCR datasets and improve robustness across varied time-series data.
Loading 1902.10756v3…