PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals

Yuxin Zhang, Yiqiang Chen, Jindong Wang, Zhiwen Pan

arXiv:2107.12626v2cs.AIcs.LGeess.SP

TL;DR

Unsupervised anomaly detection for multi-sensor time series must learn spatial-temporal normal patterns despite noisy training data, a gap in prior work. CAE-M combines convolutional reconstruction, MMD regularization, linear and nonlinear temporal prediction, and joint optimization; experiments report superior performance over state-of-the-art methods across three datasets. The approach is intended to provide robust anomaly detection, while fine-grained point-based detection and sparse operations remain future work.

  • Problem

    Prior methods have limited ability to jointly capture spatial-temporal dependencies and distinguish noisy, normal, and abnormal training data in unsupervised multi-sensor anomaly detection.

  • Method

    CAE-M combines a deep convolutional autoencoder with MMD regularization, Attention-based Bidirectional LSTM and autoregressive prediction, jointly optimized through reconstruction and prediction errors.

  • Results

    CAE-M outperforms state-of-the-art techniques on three multi-sensor time-series datasets, with at least 6% F1-score improvement reported for the Fatigue dataset.

  • Takeaways & Limitations

    The composite framework models generalized normal patterns by combining spatial-temporal correlation modeling with noise-aware representation learning and temporal prediction.

  • Takeaways & Limitations

    Future work targets point-based fine-grained anomaly detection and proper sparse operations for multi-sensor data.

Abstract

from arXiv · show

Nowadays, multi-sensor technologies are applied in many fields, e.g., Health Care (HC), Human Activity Recognition (HAR), and Industrial Control System (ICS). These sensors can generate a substantial amount of multivariate time-series data. Unsupervised anomaly detection on multi-sensor time-series data has been proven critical in machine learning researches. The key challenge is to discover generalized normal patterns by capturing spatial-temporal correlation in multi-sensor data. Beyond this challenge, the noisy data is often intertwined with the training data, which is likely to mislead the model by making it hard to distinguish between the normal, abnormal, and noisy data. Few of previous researches can jointly address these two challenges. In this paper, we propose a novel deep learning-based anomaly detection algorithm called Deep Convolutional Autoencoding Memory network (CAE-M). We first build a Deep Convolutional Autoencoder to characterize spatial dependence of multi-sensor data with a Maximum Mean Discrepancy (MMD) to better distinguish between the noisy, normal, and abnormal data. Then, we construct a Memory Network consisting of linear (Autoregressive Model) and non-linear predictions (Bidirectional LSTM with Attention) to capture temporal dependence from time-series data. Finally, CAE-M jointly optimizes these two subnetworks. We empirically compare the proposed approach with several state-of-the-art anomaly detection methods on HAR and HC datasets. Experimental results demonstrate that our proposed model outperforms these existing methods.

1 INTRODUCTION

Unsupervised anomaly detection for multi-sensor time series must capture spatial-temporal dependencies while handling noisy, unlabeled training data. CAE-M addresses these challenges with jointly optimized reconstruction and prediction components, and outperforms state-of-the-art techniques across three datasets.

  • Multi-sensor time-series anomaly detection is difficult because existing methods cannot simultaneously capture spatial and temporal dependencies.
  • CAE-M combines a deep convolutional autoencoder for spatial features with Attention-based Bidirectional LSTM and autoregressive prediction for temporal dependencies.
  • MMD regularization reduces noisy-data influence by encouraging low-dimensional representations toward a target distribution, reducing overfitting risk.
  • Jointly optimizing reconstruction and prediction in one end-to-end objective streamlines learning and is intended to avoid local minima associated with separately trained models.
  • Experiments on three multi-sensor time-series datasets show superior performance over state-of-the-art techniques, with component and convergence analyses supporting robust performance.

2 RELATED WORK

Prior unsupervised anomaly-detection methods include traditional, reconstruction, forecasting, and composite models, but each has limitations for spatial-temporal multi-sensor data. The paper positions CAE-M as an end-to-end composite approach combining complementary representations and jointly trained components.

  • Unsupervised anomaly detection methods are broadly categorized into traditional and deep-learning approaches based on whether labels are used during training.
  • Traditional methods are designed mainly for static data, while autoregressive methods model time series using previous output values.
  • Composite models: Composite models combine reconstruction and prediction branches, including encoder-decoder architectures using LSTM or ConvLSTM components.
  • Deep anomaly detection: Reconstruction models learn normal-data representations but may memorize inputs, whereas forecasting models may retain only recent values needed for prediction.
  • Composite models: CAE-M contributes spatial-temporal modeling, MMD regularization, parallel Attention-based BiLSTM and autoregressive prediction, and end-to-end joint training.

3.1 Notation

The paper models multi-sensor time-series samples as matrices assembled from signals generated by multiple sensors, then defines anomalies relative to predefined classes whose labels remain unseen during training.

  • 3.1 Notation: An input sample contains N = nm signals from n sensors, with each signal represented over a shared intersection length T.Each sensor generates m signals, and X ∈ R^N×T collects the aligned signals.
  • 3.1 Notation: The dataset pairs each input sample Xi with a classification label yi drawn from K predefined classes.The label set is Y = {1, 2, · · ·, K}.
  • 3.1 Notation: An input is called anomalous when it does not belong to any of the K predefined classes with high confidence.The class labels are unseen during unsupervised training.

3.2 Overview

CAE-M addresses limitations of prior unsupervised methods by combining convolutional feature extraction with predictive modeling in a jointly structured network for multi-sensor time series.

  • 3.2 Overview: Existing methods struggle to capture temporal dependencies, handle noise and anomalies in normal samples, and jointly model multi-sensor spatial-temporal patterns.These challenges motivate a composite architecture rather than relying on conventional or isolated models.
  • 3.2 Overview: Separating feature extraction from predictive model building can lead multi-stage approaches to local optima.The overview identifies this separation as an additional challenge.
  • 3.2 Overview: CAE-M feeds convolutional-autoencoder representations into a predictive network to encode spatial information and reduce the effect of noisy data.The approach is presented as an integrated convolutional autoencoding memory network.

3.3 Characterization Network

The characterization network extracts low-dimensional spatial features and reconstruction errors from multivariate sensor signals, while MMD regularization aligns latent representations with a target distribution to reduce noise-related overfitting.

  • 3.3 Characterization Network: The characterization network fuses multivariate signals across sensors to learn representative feature representations.This network is the spatial feature-learning component of CAE-M.
  • 3.3 Characterization Network: The low-dimensional representation combines abstracted multivariate features with reconstruction error measured using distance metrics.The optimization combines reconstruction loss with a distribution-based regularization term.
  • 3.3.1 Deep feature extraction: Given x ∈ R^N×T, the deep convolutional autoencoder maps the multi-sensor matrix to a low-dimensional representation and reconstructs an input of the same shape.The encoder and decoder perform feature extraction and reconstruction, respectively.
  • 3.3.1 Deep feature extraction: Convolutional and pooling layers reduce representation dimensions, while transposed convolutions expand the hidden representation back into the original input space.The encoder uses max pooling; the decoder uses transposed convolution layers.
  • 3.3.1 Deep feature extraction: The reconstruction error is the difference between the original input x and reconstruction x′, typically quantified by Mean Squared Error.MSE measures how close x′ is to x, with the l2-norm appearing in the loss expression.
  • 3.3.2 Handling noisy data: MMD regularization encourages the low-dimensional representation distribution to resemble a target distribution, bringing noisy representations closer to normal training-data representations.The stated purpose is to reduce overfitting to noisy and abnormal data.

3.4 Memory Network

The memory network predicts current temporal features from their history using complementary nonlinear and linear predictors, then forms prediction errors from their outputs and the true values.

  • 3.4 Memory Network: CAE-M compares future predictions with the next feature value while concurrently using reconstruction and prediction analyses to capture spatial-temporal patterns.The memory network receives characterization features from multiple time steps.
  • 3.4 Memory Network: The characterization network supplies each time step with reduced low-dimensional features and reconstruction error as the memory network input.These components are concatenated into the temporal feature representation.
  • 3.4 Memory Network: The memory network predicts the current feature zh from past values [z1, z2, ..., zh−1] using nonlinear and linear predictors.This design directly targets temporal dependency modeling.
  • 3.4.1 Non-linear prediction: The nonlinear predictor uses an attention-based Bidirectional LSTM to incorporate context from both temporal directions and emphasize relevant time steps.Attention produces a weighted context representation used as the predicted temporal feature.
  • 3.4.2 Linear prediction: The parallel Autoregressive model uses lagged observations and learned weights to forecast short-term temporal values.The paper incorporates AR alongside the nonlinear memory network because short-term modeling can be effective in specific real datasets.
  • 3.4 Memory Network: The final prediction error compares predictor output with the true value zh and integrates nonlinear and linear prediction outputs.The error is computed over a subsample of training data using the Frobenius norm notation described in the passage.

3.5 Joint optimization

CAE-M jointly trains reconstruction, distribution regularization, and linear and nonlinear forecasting objectives in an end-to-end hybrid model. These components target compact representations, anomaly separation, and temporal prediction.

  • Joint objective: End-to-end optimization is proposed because separately trained multi-step models can become stuck in local optima.
  • Joint objective: The compound objective combines reconstruction error, MMD regularization, nonlinear prediction error, and linear prediction error.The four terms are weighted by λ1, λ2, and λ3.
  • Optimization goals: The reconstruction objective improves the low-dimensional representation by minimizing average reconstruction error across samples and time steps.
  • Optimization goals: MMD regularization encourages the latent representation distribution to resemble a target distribution so anomalies deviate from normal data in reduced dimensions.
  • Optimization goals: Linear and nonlinear predictors use current and past latent values to minimize forecasting errors and express the next temporal slice.The predictors are integrated to update the low-dimensional feature and reconstruction error.
  • Hyperparameters: In practice, λ1 = e −04, λ2 = 0.5, and λ3 = 0.5 usually achieve desirable results.MMD is treated as a regularization term, with parameter selection performed in Section 4.8.1.

3.6 Inference

CAE-M derives a decision threshold from training losses and classifies each test sequence by comparing its loss with that threshold. Losses above the threshold are labeled abnormal.

  • Threshold calculation: The decision threshold is computed from the training dataset's sample losses.Err(xi) is the sum of the loss function for sample xi, and μ is the average training loss.
  • Threshold calculation: The threshold setting follows a normal training distribution centered at μ with one standard deviation σ.
  • Decision rule: A testing sequence is predicted abnormal when Err(xi) > THR and normal otherwise.
  • Procedure: Algorithm 1 presents the complete CAE-M training and inference procedure.

4 EXPERIMENTS

Experiments evaluate CAE-M for unsupervised anomaly detection using three multi-sensor datasets, traditional and deep baselines, and component variants. The reported comparisons show that CAE-M outperforms most existing methods.

  • 4.1 Datasets: Experiments use PAMAP2, CAP, and Mental Fatigue datasets covering activity recognition, sleep-state detection, and mental-fatigue detection.These datasets contain multi-sensor time-series signals and serve as anomaly-detection testbeds.
  • 4.1 Datasets: PAMAP2 contains 18 activities from 9 subjects wearing three inertial measurement units, with smaller activity classes treated as anomalies.The remaining activity categories form the normal class.
  • 4.1 Datasets: CAP includes physiological waveforms from 16 healthy subjects and 92 patients, while the experiment labels healthy subjects normal and sleep-disordered-breathing patients anomalous.
  • 4.1 Datasets: The Mental Fatigue dataset uses wearable physiological signals from 6 healthy subjects, labeling non-fatigue samples normal and fatigue samples anomalous.Fatigue samples account for a fifth of the total.
  • 4.2 Baselines and metrics: CAE-M is compared with KPCA, ABOD, OCSVM, HMM, and deep anomaly-detection methods using mean precision, mean recall, and F1 score.Accuracy is not used because anomaly-detection classes are often highly imbalanced.
  • 4.4 Results: CAE-M outperforms most existing methods in the reported mean precision, recall, and F1-score comparisons.A Wilcoxon signed-rank analysis reports an average p-value of 0.0077 for comparisons with other baselines.
  • 4.8 Ablation studies: Ablation baselines remove prediction, reconstruction and MMD, attention, AR, or MMD components to evaluate their effectiveness.The variants include CAE-Mw/oPre, CAE-Mw/oRec+MMD, CAE-Mw/oATTENTION, CAE-Mw/oAR, and CAE-Mw/oMMD.

4.4 Results and Analysis

CAE-M is evaluated across datasets, sleep stages, unseen subjects, noise levels, component ablations, parameter settings, and convergence behavior. Results generally favor CAE-M, including higher F1 scores, subject generalization, robustness analyses, and convergence within fewer than 40 iterations.

  • Overall results: CAE-M outperforms most traditional and deep anomaly detection methods on mean precision, recall, and F1 score.The comparison uses Table 2 across the evaluated datasets.
  • Dataset results: About 7% improvement at F1 score is achieved on the CAP dataset compared with existing methods.
  • Dataset results: At least 6% improvement at F1 score is achieved on the Fatigue dataset despite noise or misclassification patterns in the data.Manual labeling of fatigue and non-fatigue data is difficult, which may introduce noise or misclassification.
  • Fine-grained analysis: CAE-M achieves the best testing performance across both local and global domains in fine-grained sleep-stage experiments.The analysis concerns intra-class diversity and compares performance under similar-distribution and broader settings.
  • Generalization analysis: CAE-M reaches an average F1 score of 0.86 on previously unseen testing subjects under LOSO evaluation.The analysis also reports more stable performance on repeated measurements within subject 1.
  • Sensitivity and convergence: CAE-M reaches steady performance within fewer than 40 iterations across three datasets.Parameter sensitivity analysis identifies λ1 = e−04, λ2 = 0.5, and λ3 = 0.5 as the optimal parameter setting.

5 CONCLUSION AND FUTURE WORK

CAE-M models generalized normal patterns in multi-sensor time-series data by capturing spatial-temporal correlations while reducing overfitting from noisy and anomalous training data. Experiments on HAR and HC datasets show better performance than baseline methods, while future work targets finer-grained detection and sparse operations.

  • CAE-M captures spatial-temporal correlations in multi-sensor time-series data to model generalized normal patterns for anomaly detection.Its composite framework combines spatial characterization with temporal prediction.
  • A Deep Convolutional Autoencoder with an MMD penalty characterizes signals and reduces overfitting caused by noise and anomalies in training data.
  • Bidirectional LSTM with Attention and an Auto-regressive model represent non-linear and linear temporal dependencies for prediction.
  • CAE-M performs better than other baseline methods in extensive empirical studies on HAR and HC datasets.
  • Future work will investigate point-based fine-grained anomaly detection and sparse operations for multi-sensor data.
Loading 2107.12626v2…