PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey

Navid Mohammadi Foumani, Lynn Miller, Chang Wei Tan, Geoffrey I. Webb, Germain Forestier, Mahsa Salehi

arXiv:2302.02515v2cs.LGcs.AIcs.CV

TL;DR

Time series classification and extrinsic regression are challenging tasks in which relevant features may not be known beforehand. This survey synthesizes deep-learning architectures and training methods, with coverage of applications including human activity recognition and satellite Earth observation. It reports substantial progress while noting that commonly used archives were not designed for deep learning.

  • Problem

    Time series classification and extrinsic regression are challenging tasks requiring analysis of ordered real-valued data and relationships with extrinsic variables.

  • Method

    The paper surveys deep-learning architectures, methodological refinements, training strategies, and applications for time series classification and extrinsic regression.

  • Results

    Deep network architectures and refinements have enabled significant progress in time series classification and regression, while InceptionTime remained competitive without surpassing the benchmark state of the art.

  • Takeaways & Limitations

    The taxonomy and synthesis organize current deep-learning methods and their use in human activity recognition and satellite Earth observation.

  • Takeaways & Limitations

    The commonly used UCR and UEA benchmarking archives were not designed for deep learning, and are relatively small for settings where deep learning often excels.

Abstract

from arXiv · show

Time Series Classification and Extrinsic Regression are important and challenging machine learning tasks. Deep learning has revolutionized natural language processing and computer vision and holds great promise in other fields such as time series analysis where the relevant features must often be abstracted from the raw data but are not known a priori. This paper surveys the current state of the art in the fast-moving field of deep learning for time series classification and extrinsic regression. We review different network architectures and training methods used for these tasks and discuss the challenges and opportunities when applying deep learning to time series data. We also summarize two critical applications of time series classification and extrinsic regression, human activity recognition and satellite earth observation.

1 INTRODUCTION

Time series classification and extrinsic regression are challenging analysis tasks, and deep learning is being explored because it can learn relevant features directly from raw data. This survey organizes rapidly evolving methods, reviews training strategies, and highlights applications in human activity recognition and Earth observation.

  • Motivation: Time series analysis is a major data-mining challenge, with classification predicting categorical labels and extrinsic regression modeling relationships with numeric external variables.TSER is distinguished from forecasting because it concerns the relationship between a time series and an extrinsic variable.
  • Motivation: Deep learning can learn informative latent representations from raw time series, reducing reliance on manual feature engineering.Its scalability and hardware compatibility also support efficient processing as data volumes and computational complexity grow.
  • Reported comparisons: InceptionTime is competitive but did not outperform the state of the art on existing benchmark archives.ConvTran is reported to outperform ROCKET in speed and accuracy when training sets exceed 10k samples.
  • Research gap: Existing benchmarks and prior surveys leave gaps because UCR and UEA archives were not designed for deep learning and earlier reviews omit newer methods.The omitted developments include InceptionTime, attention models, and self-supervised learning.
  • Survey scope: The survey proposes a methodological taxonomy for deep learning in TSC and TSER and reviews architectures, self-supervised pretexts, data augmentation, and transfer learning.Architectures discussed include MLPs, CNNs, RNNs, GNNs, and attention-based models.
  • Applications: The survey covers human activity recognition and Earth observation as key applications of time series classification and extrinsic regression.These applications are presented alongside the broader review of architectures and training methods.

2 BACKGROUND AND DEFINITIONS

This section defines time series data, supervised TSC and TSER, and the deep-learning taxonomy surveyed for these tasks. It distinguishes generative from end-to-end discriminative approaches and outlines the architectures, training strategies, and applications reviewed.

  • Time series data: Time series data are ordered collections of timestamped measurement vectors, which are univariate when D=1 and multivariate when D>1.The paper represents a series as X={(x_1,t_1),...,(x_T,t_T)}, where each x_i is D-dimensional.
  • Supervised learning tasks: TSC predicts categorical labels from finite classes, whereas TSER predicts a continuous numeric value external to the input series.TSC commonly uses one-hot encoded targets, while TSER uses numeric targets.
  • Supervised learning tasks: TSC networks commonly output class probabilities through softmax and cross-entropy training, while TSER networks output numeric values using regression losses such as MSE or MAE.Regression outputs typically use a linear final activation, though other activations and losses may be selected based on the target distribution.
  • Deep-learning approaches: The survey focuses mainly on end-to-end discriminative deep-learning approaches because generative methods require additional training and are typically less efficient.Most surveyed architectures were designed for TSC, but the paper states that most can be trivially adapted for TSER.
  • Survey scope: The proposed taxonomy organizes deep-learning models by network configuration and application domain while covering MLPs, CNNs, RNNs, GNNs, attention, self-supervision, augmentation, and transfer learning.The survey also summarizes human activity recognition and satellite earth observation applications and identifies future research directions.

3 SUPERVISED MODELS

This section reviews supervised deep-learning models for time series classification and extrinsic regression, emphasizing their architectures, adaptations, strengths, and limitations. It covers multilayer perceptrons, convolutional networks, recurrent models, transformers, and graph-based approaches.

  • 3.1 Multi-Layer Perceptron (MLP): MLPs are straightforward fully connected architectures, but their fixed feedforward processing limits temporal-dependency modeling.Studies combine MLPs with feature extractors such as Dynamic Time Warping to address this limitation.
  • 3.1 Multi-Layer Perceptron (MLP): DTW-NN replaces a neuron's standard dot product with dynamic time warping to align inputs with weights and handle temporal distortions and variable pattern lengths.This preserves a feedforward architecture while introducing elastic matching.
  • 3.2 CNN based models: CNNs were adapted for TSC because their convolutional structures learn temporal patterns, with reviewed models including MC-DCNN, MC-CNN, FCN, and ResNet.MC-DCNN applies convolutions independently to each input channel, whereas MC-CNN applies 1D convolutions to all channels simultaneously.
  • 3.2 CNN based models: FCN uses global average pooling instead of fully connected layers, enabling arbitrary-size inputs and supporting class activation maps for highlighting important input regions.Its adapted univariate TSC architecture contains three convolution blocks followed by global average pooling and a softmax classifier.
  • 3.2 CNN based models: ResNet was among the most accurate deep-learning TSC architectures on 85 univariate datasets, while combining ResNet and FCN improved accuracy with almost no extra computational cost.Residual connections are used to reduce vanishing-gradient effects in deep models.
  • 3.2 CNN based models: Imaging and multi-scale approaches transform series or use varied convolutional filters, but encoding methods such as GASF, GADF, and MTF did not significantly improve prediction outcomes.InceptionTime explores larger filters, while ConvTran combines novel positional encodings, a transformer block, and a convolution layer for multivariate classification.

4 SELF-SUPERVISED MODELS

Self-supervised learning is surveyed as a response to costly labeling and weak performance with limited supervision in time series. The section organizes methods by pretext task and covers contrastive, self-prediction, and related approaches for time series classification.

  • Motivation: Limited labeled data motivates increasing use of self-supervised representation learning for time series.Large labeled datasets can outperform sparsely labeled, small, or unsupervised settings, but obtaining annotations is costly.
  • Organization: The survey categorizes self-supervised approaches for time series classification into three groups based on their pretext tasks.The reviewed models are summarized in Table 4.
  • Contrastive learning: Contrastive methods learn representations by making related time-series views or segments similar and unrelated examples dissimilar.Examples use instance-based sampling, temporal neighborhoods, overlapping subseries, or time-frequency consistency.
  • Contrastive learning: Subsequence-based approaches define nearby segments as positive pairs and distant segments as negative pairs, exploiting local temporal smoothness.TNC uses stationary neighborhoods to improve contrastive-loss sampling quality.
  • Contrastive learning: TS2Vec learns hierarchical contextual representations by enforcing consistency on common portions of overlapping subseries with temporal and instance-wise losses.The method randomly samples two overlapping subseries from each input.
  • Self-prediction: Self-prediction methods reconstruct inputs or representations, including masked transformer pre-training and dataset-origin prediction for flexible convolution filters.BENDER models EEG sequences across differing hardware, while PHIT is reported to mitigate overfitting on small datasets.
  • Other pretext tasks: Series2Vec predicts similarity between two series in temporal and spectral domains rather than relying on conventional contrastive positive variants.The approach is introduced as a self-supervised representation-learning method for time series.

5 DATA AUGMENTATION

Data augmentation expands time-series training data with synthetic samples to address limited data and overfitting. However, image-based transformations often fail to preserve time-series temporal structure.

  • Motivation: Data augmentation generates synthetic samples to increase training-data size and variety without collecting additional real-world data.It is presented as particularly useful when training data availability is limited.
  • Time-series-specific design: Image-based transformations such as cropping, rotation, and flipping cannot be directly transferred reliably to time series.Operations such as rotation may disrupt intrinsic temporal structure.
  • Motivation: Deep learning models for time series classification are especially vulnerable to overfitting because of their many trainable parameters.The section motivates augmentation as a strategy for improving generalization beyond the training data.
  • Taxonomy: The survey reviews magnitude-, time-, frequency-, window-, and averaging-based methods for producing synthetic time series.These methods include jittering and other transformations, with further details developed in the following subsections.

Random Transformations.

Random transformations augment time series by perturbing values, temporal structure, or frequency-domain representations. The reviewed operations include magnitude, time, and Fourier-based distortions.

  • Magnitude transformations: Magnitude transformations include jittering, flipping, scaling, and smoothly varying magnitude warping.Scaling multiplies the series by a factor drawn from a Gaussian distribution.
  • Time-domain transformations: Time-domain transformations include slicing, random warping, time warping, time stretching, and time perturbation.These operations remove subsequences or introduce temporal distortions.
  • Frequency-domain transformations: Frequency-domain augmentation perturbs magnitude and phase spectra after applying a Fourier transform.The surveyed example introduces perturbations to both spectral components.

Window methods.

Window methods create augmented series by recombining or partitioning segments, preserving or varying temporal structure. Window slicing supports segment-level classification and voting, while window warping stretches or compresses segments.

  • Window methods: Window methods combine segments from same-class series to enrich the pool of training examples.This produces a variety of samples from existing time series.
  • Window slicing: Window slicing divides a series into segments that retain the original class label and are classified individually.A voting system combines slice-level predictions into the final label.
  • Window warping: Window warping uses DTW to stretch or compress segments along the temporal axis, introducing time-domain variability.The method is presented alongside window slicing as an augmentation approach.
  • Reported effects: Examples of window slicing and window warping are reported to enhance the diversity and representativeness of time-series datasets.The cited work illustrates applications of both methods.
  • Averaging methods: Averaging methods combine multiple time series to form a new unified series.The passage introduces averaging as another window-method augmentation strategy.

Averaging methods.

Time-series averaging methods address temporal dynamics through alignment or learned reconstruction, and have been used for data augmentation with classifiers.

  • Averaging and augmentation must account for noise and distortions in both time and magnitude.
  • Weighted Dynamic Time Warping Barycenter Averaging aligns time series while accounting for their temporal dynamics.
  • A ResNet classifier has been used with wDBA, demonstrating its effectiveness in practice.
  • Auto-encoder averaging learns and reconstructs time series to generate averaged representations for augmentation.

Selection of data augmentation methods.

Data augmentation effectiveness depends on the dataset and neural architecture, so selecting and tuning methods requires task-specific judgment.

  • Augmentation techniques can perform differently across datasets and neural network architectures.
  • A method effective in one scenario may not produce similar results in another.
  • Practitioners must carefully select and tune augmentation methods for the specific task.
  • Successful augmentation depends on understanding both the methods and the task’s specific demands.

6 TRANSFER LEARNING

Transfer learning in time-series classification can help, but its effectiveness depends on source–target similarity, data availability, and unresolved model-selection choices.

  • Transfer learning in TSC can produce positive or negative outcomes depending on the datasets chosen for transfer.
  • DTW-based dataset similarity can guide source-dataset selection and improve accuracy in a majority of cases.
  • Similarity-based multi-source transfer learning models relationships among sources in a mixed domain.
  • Open questions include selecting suitable source models, designing generic filters, and deciding between freezing layers and full-network fine-tuning.
  • Transfer learning is constrained by the scarcity of large, curated, annotated time-series datasets.

7 APPLICATIONS - RECENT DEVELOPMENTS AND CHALLENGES

The survey reviews deep-learning applications of TSC and TSER, focusing on human activity recognition and satellite Earth observation as distinct domains.

  • Applications: The survey focuses on human activity recognition and satellite Earth observation because the broader application range cannot be covered in detail.
  • Human Activity Recognition: Wearable HAR data are split into time windows, and models map multivariate sensor data from each window to activities.
  • Human Activity Recognition: HAR studies use CNNs, RNNs, and hybrid CNN-RNN models, but the review found no full attention or transformer model.
  • Human Activity Recognition: Bidirectional LSTMs performed best when long-term effects mattered, whereas CNNs were suggested for short-term movement patterns.
  • Satellite Earth Observation: In satellite Earth observation, attention layers improved straight GRU or LSTM models, while self-attention was more robust to noise than RNNs or CNNs.
  • Satellite Earth Observation: Temporal information can be extracted with 1D-CNNs, while 3D-CNNs can extract spatio-temporal information.

8 CONCLUSION

The survey reviews deep network architectures and refinements for time series classification and extrinsic regression, highlighting applications and resources that support further research.

  • The survey covers multilayer perceptrons, convolutional, recurrent, and attention-based networks for time series classification and extrinsic regression.
  • The paper highlights human activity recognition and satellite Earth observation as critical applications of time series classification and regression.
  • Deep network architectures and refinements have enabled significant progress in time series classification.
  • The survey aims to stimulate further research using deep learning techniques for time series classification and extrinsic regression.
  • A carefully curated collection of sources is provided to support continued study of these tasks.

A NON-DEEP LEARNING TIME SERIES CLASSIFICATION

Non-deep-learning time series classification includes distance-based, transformation-based, ensemble, and random-convolution approaches, balancing accuracy against computational cost and scalability.

  • Distance-based approaches commonly pair a 1-nearest-neighbour classifier with a time series distance measure.
  • Dynamic Time Warping compensates for distortions by aligning two time series while minimizing alignment cost.
  • Amerced DTW is described as the first distance significantly more accurate than DTW, while Ensemble of Elastic distances significantly outperforms individual distances.
  • Most distance measures have complexity O(L^2), making nearest-neighbour search costly and distance-based methods among the slowest TSC approaches.
  • HIVE-COTEv2.0 is among the most accurate classifiers on 26 multivariate and 142 univariate benchmark datasets but scales poorly to large, long series.
  • Rocket processed 109 univariate datasets in under 4 hours, compared with days for the previous fastest method, using random convolutional filters and a linear classifier.
  • Hydra combined with Multirocket is described as one of the fastest and most accurate methods for time series classification.

B DNN ARCHITECTURES FOR TIME SERIES

The survey describes fully connected, convolutional, recurrent, gated, and attention-based architectures for time series classification. It emphasizes how these architectures represent temporal structure and adapt general deep-learning designs to sequential data.

  • B.1 Fully Connected Networks: MLPs connect every neuron in one layer to every neuron in the next using weights, biases, and nonlinear activation functions.The first activation is the input series X.
  • B.1 Fully Connected Networks: MLPs treat time-series elements independently, limiting their ability to capture temporal dependencies.Their feedforward processing weights each time step individually in a fixed order.
  • B.2 Convolution Neural Networks (CNNs): The time-series-specific t-LeNet architecture adapts LeNet’s convolutional, pooling, and fully connected layer pattern to time series.Figure 3 presents the t-LeNet architecture.
  • B.2 Convolution Neural Networks (CNNs): CNNs learn local feature representations with convolution kernels, shared weights, nonlinear activations, pooling, and later fully connected layers.Pooling reduces feature-map resolution and supports shift-invariance, while final layers perform higher-level reasoning and classification.
  • B.3 Recurrent Neural Networks (RNN): RNNs process variable-length sequences by sharing parameters over time and maintaining hidden states that support sequence-to-sequence or sequence-to-one outputs.Time-windowed inputs can be labeled using the majority response within each window.
  • B.3.1 Long Short-Term Memory (LSTM) and B.3.2 Gated Recurrent Unit (GRU): LSTM and GRU architectures regulate recurrent state updates with gates, with GRUs using reset and update gates in a simpler, more computationally efficient design.LSTMs use hidden and memory vectors; the reviewed text also describes their role in addressing vanishing or exploding gradients.
  • B.4 Attention Mechanisms: Attention computes normalized weights by comparing the current target hidden state with source hidden states, then forms a context vector for prediction.The resulting refined hidden state incorporates the original hidden state and context.

C DATASETS

The survey catalogs datasets and sensing sources used in deep-learning studies of human activity recognition and Earth observation. Human-activity datasets vary in participants, sensors, and recorded activities, while Earth-observation studies draw on multiple satellites and instruments.

  • C.1 Human Activity Recognition Datasets: Human activity recognition studies commonly use public datasets documented by participants, sensors, recorded activities, and study references.Table 7 lists commonly used datasets, while larger dataset lists are cited separately.
  • C.1 Human Activity Recognition Datasets: Common activity sets include activities of daily living such as walking, running, sitting, standing, and ascending or descending stairs.Specialized datasets also cover gait freezing in Parkinson’s disease, falls, and manufacturing activities.
  • C.2 Earth Observation Satellites and Instruments: Earth-observation studies use data collected by a range of satellites and instruments, with source-specific references and study lists summarized in Table 8.Further instrument and data-collection details are provided by the cited references.
Loading 2302.02515v2…