PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep learning for time series classification: a review

Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, Pierre-Alain Muller

arXiv:1809.04356v4cs.LGcs.AIstat.ML

TL;DR

Time series classification has many proposed methods but limited evidence on which deep neural architectures provide state-of-the-art performance. This paper systematically evaluates deep learning approaches across univariate and multivariate datasets, finding that end-to-end models can achieve current state-of-the-art performance.

  • Problem

    The study addresses open questions about state-of-the-art deep neural networks, effective architectures, initialization sensitivity, and interpretability for time series classification.

  • Method

    The paper unifies deep learning approaches under a taxonomy, reimplements nine end-to-end classifiers, and evaluates them on univariate and multivariate benchmarks.

  • Results

    End-to-end deep learning achieved current state-of-the-art time series classification performance, particularly with Fully Convolutional and deep Residual Networks.

  • Takeaways & Limitations

    The study provides a unified account and publicly available framework for comparing deep learning approaches to time series classification.

  • Takeaways & Limitations

    The study does not thoroughly evaluate data augmentation, transfer learning, training and testing time, or normalization effects.

Abstract

from arXiv · show

Time Series Classification (TSC) is an important and challenging problem in data mining. With the increase of time series data availability, hundreds of TSC algorithms have been proposed. Among these methods, only a few have considered Deep Neural Networks (DNNs) to perform this task. This is surprising as deep learning has seen very successful applications in the last years. DNNs have indeed revolutionized the field of computer vision especially with the advent of novel deeper architectures such as Residual and Convolutional Neural Networks. Apart from images, sequential data such as text and audio can also be processed with DNNs to reach state-of-the-art performance for document classification and speech recognition. In this article, we study the current state-of-the-art performance of deep learning algorithms for TSC by presenting an empirical study of the most recent DNN architectures for TSC. We give an overview of the most successful deep learning applications in various time series domains under a unified taxonomy of DNNs for TSC. We also provide an open source deep learning framework to the TSC community where we implemented each of the compared approaches and evaluated them on a univariate TSC benchmark (the UCR/UEA archive) and 12 multivariate time series datasets. By training 8,730 deep learning models on 97 time series datasets, we propose the most exhaustive study of DNNs for TSC to date.

1 Introduction

This introduction frames time series classification as a challenging, rapidly expanding data-mining problem and motivates deep learning as a scalable alternative to complex traditional methods. The paper compares recent DNN approaches, evaluates them broadly, and investigates performance, complexity, initialization, and interpretability.

  • Hundreds of TSC algorithms have been proposed since 2015 as temporal data availability has increased, underscoring the problem’s importance and difficulty.
  • NN-DTW is a strong traditional baseline, while HIVE-COTE’s 37 classifiers and hyperparameter cross-validation make it computationally intensive and impractical for some applications.
  • The paper investigates which DNN architectures achieve state-of-the-art TSC performance, reduce complexity relative to HIVE-COTE, remain robust to random initialization, and improve interpretability.
  • The study compares recent deep learning approaches through a common Python, Keras, and TensorFlow framework, training architectures end-to-end on raw time series using GPUs.
  • The evaluation covers 97 univariate and multivariate datasets, including 12 MTS datasets, where deep learning can address dimensionality through compositional smoothness and GPU parallelism.
  • The experiments trained about 1 billion parameters and found DNNs significantly outperformed NN-DTW while a deep residual network was not significantly different from COTE and HIVE-COTE.

2 Background

Section 2 introduces definitions, reviews the theoretical foundations of training deep neural networks for time series classification, and presents a taxonomy of DNN approaches with real-world applications.

  • 2 Background: The section covers foundational definitions, theoretical background on training DNNs for TSC, and a taxonomy of DNNs illustrated with real-world data-mining applications.These components are presented in sequence to support understanding of the paper’s methods and comparisons.

2.1 Time series classification

Time series classification operates on univariate or multivariate time series paired with one-hot class labels. Its goal is to train a classifier that maps possible inputs to a probability distribution over class values.

  • Time series classification: A univariate time series is an ordered set of T real values, where T denotes its length.The series is represented as X = [x1, x2, . . . , xT ].
  • Time series classification: An M-dimensional multivariate time series consists of M different univariate time series, each in R^T.It is represented as X = [X1, X2, . . . , XM].
  • Time series classification: A dataset contains time-series and one-hot label pairs, with each label vector encoding one of K classes.For class j, the corresponding vector element is 1 and all other elements are 0.
  • Time series classification: TSC trains a classifier to map possible inputs to a probability distribution over class-variable values.The labels are the class variable values predicted by the classifier.

2.2 Deep learning for time series classification

The review frames time series classification around deep neural networks that learn hierarchical representations through layered nonlinear transformations. It focuses on MLPs, CNNs, and ESNs, while describing training, inference, and architectural limitations relevant to TSC.

  • Deep learning framework: DNNs for TSC learn hierarchical representations through compositions of parameterized layers, with each layer applying a nonlinear transformation to the preceding output.This computation proceeds by feed-forward propagation through L layers.
  • Training and inference: Training initializes network weights and optimizes a differentiable loss, commonly categorical cross entropy, using gradient descent and backpropagation.Transfer learning is discussed as an alternative using a pretrained source model, but is not studied empirically.
  • Training and inference: During inference, the probabilistic classifier performs a forward pass on unseen data, predicts the maximum-probability class, and is evaluated using accuracy.The review contrasts this probabilistic output with non-probabilistic classifiers such as NN-DTW.
  • Architectures: The review focuses on three widely adopted end-to-end DNN architectures for TSC: MLPs, CNNs, and ESNs.These architectures are presented as the principal model types examined in the review.
  • MLP: MLPs are fully connected, but they assign separate weights to timestamps, treating time-series elements independently and losing temporal information.For an M-dimensional input of length T, a neuron's weights contain T × M values.
  • Echo State Network: ESNs were proposed for time series to mitigate RNN limitations by eliminating hidden-layer gradient computation, thereby reducing training time and avoiding vanishing gradients.RNNs are also described as difficult to train and parallelize and as prone to vanishing gradients on long time series.

2.3 Generative or discriminative approaches

Deep learning approaches for time series classification are organized into generative and discriminative models. Generative methods learn representations before classification, whereas discriminative methods either use engineered features or learn representations end-to-end.

  • Generative or discriminative approaches: TSC deep learning approaches divide into generative and discriminative models, with further subgroups detailed in the taxonomy.This organization is illustrated in Figure 5.
  • Generative approaches: Generative models typically use unsupervised training before classification to learn a useful time-series representation.Examples include stacked denoising auto-encoders and generative CNN models with deconvolution and up-sampling.
  • Discriminative approaches: Discriminative models directly map raw time series or hand-engineered features to class-probability outputs.They comprise hand-engineered-feature models and end-to-end models.
  • Discriminative approaches: Hand-engineered discriminative approaches transform time series into images or extract domain-specific features before classification.Image transformations include Gramian fields, recurrence plots, and Markov transition fields; sensor-derived velocity is an example of a domain-specific feature.
  • Discriminative approaches: End-to-end deep learning jointly learns features while fine-tuning a domain-agnostic classifier, with CNNs widely applied because they learn spatially invariant filters from raw inputs.MLPs can lose temporal information and produce features that are no longer time-invariant.

3 Approaches

Section 3 explains the rationale for selecting discriminative end-to-end approaches and details nine deep learning architectures, including their advantages and drawbacks.

  • 3 Approaches: The section justifies choosing discriminative end-to-end approaches for the empirical evaluation.
  • 3 Approaches: It then describes nine deep learning architectures and discusses each architecture’s advantages and drawbacks.

3.1 Why discriminative end-to-end approaches ?

The study excludes generative models because they are often less accurate, more complicated, classifier-dependent, and frequently proposed outside classification. It therefore evaluates discriminative end-to-end deep learning approaches, selecting nine architectures validated on standard archives and not targeted at specialized subtasks.

  • Generative models were excluded because they are mainly proposed for non-classification tasks or as components of larger classification schemes.
  • Generative models are usually less accurate than direct discriminative models and require the additional step of fitting a time series generator.The authors note that this added implementation complexity has been a barrier, particularly when code is unavailable.
  • Their accuracy also depends heavily on the chosen off-the-shelf classifier, which may not be a neural network.
  • The evaluation is limited to discriminative end-to-end models, excluding classifiers that incorporate feature engineering.
  • Nine approaches were selected to cover architectures including CNNs, Fully CNNs, MLPs, ResNets, and ESNs, using validation on the UCR/UEA and/or MTS archives.Approaches addressing specialized TSC subtasks, such as imbalanced-data classification, were excluded.

3.2 Compared approaches … 3.2.4 Encoder

The paper compares nine deep-learning architectures for time series classification, spanning traditional fully connected, convolutional, residual, and attention-based designs. The compared models differ in parameter invariance, transferability, depth, and architectural mechanisms.

  • 3.2 Compared approaches: Nine architectures were selected to compare recent deep-learning approaches for time series classification.
  • 3.2.1 Multi Layer Perceptron: The MLP baseline uses four fully connected layers, including three 500-neuron ReLU hidden layers and a final softmax classifier.
  • 3.2.1 Multi Layer Perceptron: MLP parameter counts depend directly on input-series length, so transferability across different-length series is not trivial.
  • 3.2.2 Fully Convolutional Neural Network: FCN preserves series length through convolutions, replaces the final fully connected layer with Global Average Pooling, and was validated on 44 UCR/UEA datasets.
  • 3.2.2 Fully Convolutional Neural Network: FCN uses three convolutional blocks followed by Global Average Pooling and softmax classification, with filter configurations of 128×8, 256×5, and 128×3.
  • 3.2.3 Residual Network: ResNet is an 11-layer architecture with nine convolutional layers, residual shortcuts, Global Average Pooling, and a final softmax classifier.
  • 3.2.3 Residual Network: ResNet layers except the final one have parameter counts invariant across datasets, enabling pre-training and fine-tuning without modifying hidden layers.
  • 3.2.4 Encoder: Encoder is a hybrid CNN inspired by FCN that replaces Global Average Pooling with attention; the study implemented only end-to-end training from scratch.Encoder also differs through PReLU activations, dropout regularization, and max pooling, while its attention mechanism enables parameter invariance across layers.

3.2.5 Multi-scale Convolutional Neural Network · 3.2.6 Time Le-Net

MCNN validates an end-to-end deep architecture for UCR time-series classification by transforming subsequences into multivariate inputs before CNN processing. Time Le-Net uses a LeNet-inspired CNN with local pooling and augmentation, but its fully connected layer limits parameter efficiency and transferability.

  • 3.2.5 Multi-scale Convolutional Neural Network: MCNN was the earliest approach to validate an end-to-end deep learning architecture on the UCR Archive, using two convolutions, max pooling, an FC layer, and softmax.Its main drawback is a complex data-preprocessing step based on extracted subsequences rather than raw time series.
  • 3.2.5 Multi-scale Convolutional Neural Network: MCNN applies window slicing, then identity mapping, down-sampling, and smoothing to transform each univariate subsequence into a multivariate input.The resulting representations are processed by independent convolutions in parallel.
  • 3.2.5 Multi-scale Convolutional Neural Network: MCNN concatenates first-stage convolution outputs before a second convolutional layer, a 256-neuron sigmoid FC layer, and a class-sized softmax classifier.Each convolution uses 256 sigmoid filters followed by max pooling.
  • 3.2.5 Multi-scale Convolutional Neural Network: MCNN cross-validates filter length and pooling factor by grid search, and only its first two of four layers are invariant and transferable.The grid search uses an unseen split from the training set.
  • 3.2.6 Time Le-Net: Time Le-Net is a LeNet-inspired CNN with two convolutions, local max-pooling operations, an FC layer, and a final softmax classifier.Unlike GAP, local pooling introduces invariance to small perturbations.
  • 3.2.6 Time Le-Net: Time Le-Net uses ReLU convolutions with filter length 5, followed by pooling lengths 2 and 4, reducing an input of length l to l/8.The first convolution has 5 filters and the second has 20 filters.
  • 3.2.6 Time Le-Net: Because its FC layer replaces GAP, only 2 of 4 layers are invariant, while parameter count depends on input length, limiting transferability to the first two convolutions.The transferable convolution parameters depend solely on the number and length of the chosen filters.
  • 3.2.6 Time Le-Net: Time Le-Net uses WS and Window Warping for augmentation against overfitting, including WS-based subsequence extraction and majority voting at test time.Window Warping produces time series of lengths l, 2l, and 1/2l; in the original paper, WS length is 0.9l.

3.2.7 Multi Channel Deep Convolutional Neural Network · 3.2.8 Time Convolutional Neural Network · 3.2.9 Time Warping Invariant Echo State Network

The reviewed architectures adapt convolutional or recurrent designs to time series classification through channel-wise processing, altered loss functions, pooling, and reservoir-based representations. MCDCNN processes multivariate channels independently, Time-CNN uses MSE with a fully connected output, and TWIESN projects raw time-series elements into reservoir space for classification.

  • 3.2.7 Multi Channel Deep Convolutional Neural Network: MCDCNN applies convolutions independently and in parallel to each dimension of a multivariate time series.This modification extends a traditional deep CNN to multivariate inputs.
  • 3.2.7 Multi Channel Deep Convolutional Neural Network: Each MCDCNN channel uses two convolutional stages with 8 length-5 filters, ReLU activations, and length-2 max pooling.Outputs from the second convolutional stage are concatenated across channels.
  • 3.2.7 Multi Channel Deep Convolutional Neural Network: MCDCNN concatenates channel outputs, feeds them to a 732-neuron ReLU fully connected layer, and uses softmax with one neuron per class.The classifier’s neuron count equals the number of classes in the dataset.
  • 3.2.8 Time Convolutional Neural Network: Time-CNN was proposed for both univariate and multivariate time series classification and replaces categorical cross-entropy with mean squared error.Its final layer is a traditional fully connected layer rather than a softmax classifier.
  • 3.2.8 Time Convolutional Neural Network: Time-CNN contains two convolutional layers with 6 and 12 filters, length-3 local average pooling, sigmoid activations, and a class-sized fully connected output.The network output has as many neurons as there are dataset classes.
  • 3.2.9 Time Warping Invariant Echo State Network: TWIESN is the only non-convolutional recurrent architecture tested and re-implemented, using raw time series to predict class-probability distributions.It is a time-warping-invariant variant of echo state networks.
  • 3.2.9 Time Warping Invariant Echo State Network: TWIESN projects each input timestamp into a higher-dimensional reservoir space and trains a Ridge classifier to predict each time-series element’s class.For univariate series, the projected space’s dimensions are inferred from the reservoir size.

3.3 Hyperparameters

This section reports the architecture and optimization hyperparameters used by the evaluated approaches, with TWIESN handled separately because its hyperparameters are incompatible with the other eight algorithms. For the remaining classifiers, model checkpoints were selected using either the training set or a validation split.

  • Hyperparameters: Tables 1 and 2 report the architecture and optimization hyperparameters for all approaches except TWIESN.TWIESN is excluded because its hyperparameters are not compatible with those of the other eight algorithms.
  • Hyperparameters: The other deep learning classifiers used model checkpointing based on either the training set or a validation set split from the training set.Thus, training for a fixed number of epochs did not necessarily determine the final selected model.

4 Experimental setup

The experiments evaluate deep learning models across comprehensive univariate and multivariate time-series archives, using preprocessing, repeated runs, and statistical testing for fair comparison. The study trains 8,730 models and accelerates computation through a 60-GPU cluster.

  • Datasets: 85 univariate datasets from the UCR/UEA archive were used, spanning series lengths from 24 to 2,709 and seven application categories.The categories are Image Outline, Sensor Readings, Motion Capture, Spectrographs, ECG, Electric Devices, and Simulated Data.
  • Datasets: 13 multivariate datasets from Baydogan’s archive were evaluated, excluding PeMS because of single-GPU memory limitations.The multivariate archive contains time series whose lengths can vary within the same dataset.
  • Preprocessing: Unequal-length multivariate series were linearly interpolated along each dimension to match the longest series length.This preprocessing addresses length-dependent architecture requirements in deep learning models.
  • Evaluation protocol: 97 datasets were each evaluated with nine deep learning models across 10 runs, producing 8,730 experiments.Each run retained the archive’s original train/test split and changed only the random weight initialization to estimate mean accuracy across runs.
  • Implementation: 60 GPUs reduced the approximate sequential runtime from 100 days on one GPU to less than one month.The cluster combined Nvidia GTX 1080 Ti, Tesla K20, K40, and K80 cards, and the framework used Keras with TensorFlow.
  • Evaluation protocol: Mean accuracy averaged over 10 test-set runs was used, while state-of-the-art comparisons averaged accuracy using median test error and applied Friedman and pairwise post-hoc tests.The statistical testing procedure followed recommendations from Demšar (2006).

5 Results

ResNet was the strongest deep-learning approach across the univariate and multivariate evaluations, although performance varied by domain and dataset size. Comparisons with classical classifiers favored ResNet empirically, while differences in training budgets and HIVE-COTE’s complexity limited direct practical conclusions.

  • Univariate results: ResNet significantly outperformed the other deep-learning approaches, winning 50 of 85 univariate problems with an average rank of almost 2.It also significantly outperformed FCN on the UCR/UEA archive.
  • Multivariate results: The three deep CNNs—ResNet, FCN, and Encoder—outperformed the MTS-specific Time-CNN and MCDCNN architectures on the 12 multivariate datasets.Adding the multivariate datasets did not significantly change the critical difference diagram relative to the univariate evaluation.
  • Dataset characteristics: ResNet dominated across dataset domains, except on ECG datasets, where FCN drastically beat it on 71.4% of the seven datasets.The ECG result is based on only seven datasets.
  • Dataset characteristics: Time-series length did not provide information about deep-learning performance, although TWIESN reached 96.8% accuracy on Meat with length 448.The results grouped DNN average ranks by dataset length, while TWIESN remained competitive on several long series.
  • Dataset characteristics: On DiatomSizeReduction, ResNet and FCN achieved their worst accuracy at 30%, whereas Time-CNN achieved the best accuracy at 95%.The passage identifies DiatomSizeReduction as the smallest dataset investigated in this analysis.

6 Visualization

The section presents Class Activation Maps (CAMs) to explain classifier decisions by highlighting influential time-series subsequences, and Multi-Dimensional Scaling (MDS) to visualize learned latent representations. Case studies show that these visualizations reveal discriminative regions and class separation, while MDS projections require careful interpretation because they simplify each series to one point.

  • Class Activation Maps: CAM produces a univariate time series whose values are learned weighted sums of the input variables at each time stamp.ResNet and FCN use Global Average Pooling to identify input regions contributing to a classification.
  • Class Activation Maps: On GunPoint, FCN and ResNet ignored non-discriminative plateaus and selected the same high-derivative regions as discriminative.The models achieved almost 100% accuracy on this binary dataset.
  • Class Activation Maps: On Meat, ResNet achieved 97% accuracy versus FCN’s 83%, while its CAM more confidently filtered discriminative and non-discriminative regions.FCN also highlighted a visually non-discriminative subsequence for classes 2 and 3.
  • Multi-Dimensional Scaling: MDS output must be interpreted cautiously because it highly simplifies the data space by representing each time series as a single point.The distance between points is based on the Euclidean distance between their GAP vectors.
  • Multi-Dimensional Scaling: MDS projections of FCN and ResNet latent representations separated the two classes even though raw-input projections were not linearly separable.The latent features were taken from the networks’ Global Average Pooling layers.
  • Multi-Dimensional Scaling: For Wine, ResNet’s hidden features projected data farther from the decision boundary than FCN’s learned representation, potentially explaining its better performance.The comparison used raw inputs and latent features from the GAP layers of both models.

7 Conclusion

The paper presents a large empirical study of deep neural networks for time series classification, organized through a unified generative–discriminative taxonomy and a public framework implementing nine classifiers. It concludes that end-to-end deep learning is promising, while data augmentation, transfer learning, and runtime comparisons remain insufficiently studied.

  • Contributions: The study re-implemented nine recently published end-to-end deep learning classifiers in a publicly available unified framework for time series classification.The approaches span domains including human activity recognition and sleep stage identification.
  • Contributions: Deep neural networks for time series classification are organized into two main categories: generative and discriminative models.The taxonomy covers successful deep learning approaches across multiple time series domains.
  • Limitations and future work: Deep learning remains insufficiently studied for data augmentation, transfer learning, and training and testing time beyond accuracy.The authors identify these topics as directions for extending the empirical study.
  • Conclusion: End-to-end deep learning is presented as an enticing approach because deeper architectures can automatically learn from annotated data as data mining repositories become more frequent.The conclusion specifically emphasizes automatic learning from annotated data.
Loading 1809.04356v4…