PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-World Anomaly Detection by using Digital Twin Systems and Weakly-Supervised Learning

Andrea Castellani, Sebastian Schmitt, Stefano Squartini

arXiv:2011.06296v1cs.LG

TL;DR

The paper addresses industrial anomaly detection when normal measurements are plentiful but labeled anomalies are scarce and costly to obtain. It combines Digital Twin-generated normal-operation data with a small set of real anomalous samples through Cluster Centers and Siamese Autoencoder methods. The proposed weakly supervised methods outperform the evaluated state-of-the-art approaches, with SAE false-positive and false-negative rates around 9% and 12%.

  • Problem

    Industrial anomaly detection requires reliable methods despite abundant unlabeled measurements, scarce labeled anomalies, and difficult dataset labeling.

  • Method

    The paper generates a year of normal-operation data with a Digital Twin and combines it with a small labeled real-anomaly set for Cluster Centers and Siamese Autoencoder detection.

  • Results

    The proposed weakly supervised algorithms outperform state-of-the-art approaches across all performance measures; SAE false-positive and false-negative rates are around 9% and 12%.

  • Takeaways & Limitations

    The approach remains effective with only five labeled anomalous samples, while the best-performing SAE supports anomaly detection using synthetic normal data and few real anomalies.

  • Takeaways & Limitations

    The evaluation assumes labeled anomalies contain no noise, although real-world labeling errors and changing anomaly definitions are expected; expert-feedback evaluation remains future work.

Abstract

from arXiv · show

The continuously growing amount of monitored data in the Industry 4.0 context requires strong and reliable anomaly detection techniques. The advancement of Digital Twin technologies allows for realistic simulations of complex machinery, therefore, it is ideally suited to generate synthetic datasets for the use in anomaly detection approaches when compared to actual measurement data. In this paper, we present novel weakly-supervised approaches to anomaly detection for industrial settings. The approaches make use of a Digital Twin to generate a training dataset which simulates the normal operation of the machinery, along with a small set of labeled anomalous measurement from the real machinery. In particular, we introduce a clustering-based approach, called Cluster Centers (CC), and a neural architecture based on the Siamese Autoencoders (SAE), which are tailored for weakly-supervised settings with very few labeled data samples. The performance of the proposed methods is compared against various state-of-the-art anomaly detection algorithms on an application to a real-world dataset from a facility monitoring system, by using a multitude of performance measures. Also, the influence of hyper-parameters related to feature extraction and network architecture is investigated. We find that the proposed SAE based solutions outperform state-of-the-art anomaly detection approaches very robustly for many different hyper-parameter settings on all performance measures.

I. INTRODUCTION

Industrial anomaly detection needs reliable methods despite scarce labeled anomalies and abundant unlabeled sensor data. The paper combines Digital Twin-generated normal-operation data with a small set of real anomalous samples for weakly supervised detection.

  • Motivation: Industry 4.0 monitoring produces sensor time series in which failure precursors may support predictive maintenance.Reliable anomaly detection is important because component or system failures increase maintenance, overhaul, and repair costs.
  • Problem setting: Weakly supervised learning addresses settings with abundant unlabeled data and very few labeled samples, especially anomalies.Fully labeled datasets are difficult to create because anomalous scenarios require substantial expert effort.
  • Digital Twin: Digital Twins can simulate machinery accurately enough to generate large datasets representing normal operating conditions.Depending on fidelity, simulations range from qualitatively correct behavior to near copies of physical systems.
  • Proposed direction: The proposed approach uses a complete year of Digital Twin normal-operation data and a small real-world set of labeled anomalies.This combination is intended to model rare but normal states while increasing sensitivity to anomalies.
  • Contributions: The paper contributes a clustering-based algorithm, a Siamese Autoencoder architecture, and comparisons with state-of-the-art anomaly detectors.It also investigates performance across feature-extraction and network-architecture hyperparameters.

II. RELATED WORKS

Prior work covers supervised, unsupervised, and deep anomaly detection, while Digital Twin applications have rarely focused on anomaly-detection algorithms. This paper positions its contribution within the smaller body of weakly supervised approaches.

  • Existing approaches: Anomaly detection research spans applications including energy, manufacturing, network sensors, healthcare, and video surveillance.Deep methods commonly use dimensionality reduction and reconstruction to identify anomalies.
  • Digital Twins: Digital Twin research includes design, production, and prognostics, but few studies apply Digital Twins directly to anomaly detection.Existing Digital Twin anomaly-detection references focus more on presenting the Digital Twin system than the detection algorithm.
  • Weak supervision: Only a small number of weakly supervised anomaly-detection approaches have been proposed compared with supervised and unsupervised methods.Prior work includes domain-adversarial transfer and Siamese-network approaches, including applications of Siamese Autoencoders.
  • Paper positioning: The proposed pipeline combines Digital Twin normal-operation data with a small real-world anomalous set during training, then scores physical-machine measurements during operation.An anomaly score is calculated for each real-world sample after the detector is trained.

A. Cluster Centers

The Cluster Centers method models normal operation with clusters learned from Digital Twin data and refines anomaly scores using labeled real-world anomalies. Its score combines distance from normal cluster centers with an anomaly-proximity penalty.

  • Normal-operation modeling: Cluster Centers first applies unsupervised clustering to Digital Twin normal samples to identify statistical modes of machinery operation.The method uses k-means, although other clustering algorithms are suitable when cluster centers can be computed.
  • Cluster design: The clusters are intended to cover statistical variation, transitions, and switching behavior rather than represent semantically correct operational states.Consequently, the number of clusters should exceed the guessed or known number of true operating states.
  • Anomaly score: Real-world samples are scored by their distance to the nearest cluster center learned from Digital Twin normal data.This establishes an unsupervised anomaly score based on the distribution of normal operation.
  • Weak supervision: A penalty increases the anomaly score when a real-world sample lies near a labeled anomalous sample.The penalty uses η to control the labeled-anomaly influence and ζ as a regularization factor.
  • Comparison and design choice: Unlike regular KNN scoring, Cluster Centers uses distances to cluster centers and includes the labeled-anomaly penalty term.The authors retain the simple penalty formulation rather than introducing additional exponents or constrained clustering.

B. Siamese Autoencoder

The Siamese Autoencoder learns to reconstruct normal data while separating normal and anomalous latent representations. Its loss combines reconstruction, contrastive, and partial contrastive objectives, enabling training with many more normal than anomalous samples.

  • Architecture: Siamese networks use two identical shared-weight networks to assess whether paired samples come from the same distribution.The proposed architecture applies this principle to anomaly detection with autoencoder encoders and decoders.
  • Architecture: The autoencoder maps inputs to lower-dimensional latent representations and reconstructs them in the original data space.The encoder produces h(x), while the decoder produces D(E(x)).
  • Training objectives: Training seeks low reconstruction error for normal samples, compact normal embeddings, poor anomalous reconstruction, and separated normal-anomalous embeddings.These behaviors make reconstruction and latent-space differences useful for anomaly scoring.
  • Loss functions: The loss uses reconstruction, contrastive, and partial contrastive contributions calculated over pairs of Digital Twin and selected samples.The reconstruction term uses normal-operation mean squared error, while contrastive terms shape latent distances and anomalous reconstruction.
  • Weak supervision: The pair construction supports highly unbalanced training datasets with |N| ≫ |A|.A very large number of pairs can be formed from the large normal dataset and very small anomalous dataset.
  • Variants and scoring: The paper evaluates feature-based SAE and raw-time-series CNN-SAE variants with symmetric feed-forward or one-dimensional convolutional autoencoders.The anomaly score uses reconstruction error and embedding distance for a new real-world sample.

IV. DIGITAL TWIN

The Digital Twin models the facility’s electrical power, HVAC, and CHP systems, but cannot reproduce exact minute-scale transient behavior. It nevertheless achieves high-fidelity simulation of power and energy consumption over windows of at least several hours.

  • Digital Twin model: The Digital Twin simulates the facility’s electrical power, HVAC, and CHP systems using Green City and SimulationX based on Modelica.The model was calibrated against measurement data in a separate publication.
  • Validation and scope: 2.18% discrepancy was measured between simulated and real-world electrical and thermal energy production during validated correct-operation periods in 2018.The validation covered several periods of correct machine operation.
  • Validation and scope: The Digital Twin cannot exactly simulate transient machine behavior, so raw time-series comparison at one-minute sampling is unsuitable.Its high-fidelity simulation applies to power and energy consumption over reasonably large windows of at least a few hours.

V. COMPARATIVE METHODS

The study compares direct simulation error with established statistical, kernel, neighborhood, dimensional-reduction, and neural anomaly detectors. Unsupervised methods train on Digital Twin normal data, while weakly supervised and supervised methods additionally use a few labeled anomalies.

  • Baseline methods: Mean Absolute Error directly compares measured and simulated raw time series as an anomaly score.The comparison also includes Isolation Forest and k-Nearest Neighbors methods.
  • Baseline methods: The benchmark includes Isolation Forest, k-Nearest Neighbors, One-Class SVM, Local Outlier Factor, PCA, and Feed-Forward Autoencoder approaches.The methods cover tree-based, neighborhood, kernel, density, reconstruction, and neural approaches.
  • Training regimes: Unsupervised algorithms train on the Digital Twin dataset and are evaluated on real-world data, while weakly supervised and supervised algorithms also use randomly sampled anomalies.One One-Class SVM is additionally trained and tested using real-world data only.

VI. EXPERIMENTAL SETUP

Experiments use a facility-monitoring dataset centered on CHP energy systems, with minute-level measurements and manually labeled anomalies. A synthetic Digital Twin dataset reproduces the same time series and processing for 2018.

  • Dataset description: The real-world dataset records heat, cold, electricity, and weather sensors from a medium-sized company’s infrastructure monitoring system.The experiments focus on the combined heat and power module, where natural gas produces heat and electrical power.
  • Dataset description: 658081 sample points were collected at one-minute intervals from November 2017 to February 2019.The recorded period spans more than one year.
  • Dataset description: The synthetic Digital Twin dataset contains the same time series and processing as the real-world dataset, with only 2018 simulated.This provides the synthetic counterpart used in the experiments.
  • Anomaly labels: 100 manually identified failure instances account for 24.2% of the samples labeled anomalous.Anomalies include flat sensor-failure time series and valid-looking individual readings whose combination indicates abnormal operation.

B. Data Pre-Processing and Evaluation Metrics

The experimental pipeline extracts statistical, contextual, and temporal features, standardizes or encodes them, and evaluates models with multiple metrics and cross-validation. Feature and network hyperparameters are selected through partial grid search, with final configurations and repeated training reported.

  • Feature extraction: Statistical features are extracted from raw data with sliding windows, while contextual features capture shutdown counts and working time.Time features encode working days and season of the year.
  • Feature preprocessing: Feature vectors are standardized with z-score normalization, while time and contextual values use one-hot encoding.The z-score is Z = (X − x̄)/σ, where x̄ is the mean and σ is the standard deviation of X.
  • Evaluation: Performance is measured with F2 Score, AUC ROC, and Average Precision, with PRC-based measures emphasized for imbalanced classes.Average Precision is used for algorithm comparison and hyperparameter search because it is threshold-free.
  • Validation and search: Ten-fold cross-validation combines the Digital Twin dataset with labeled anomalies for training, while real-world data use 20% validation and 80% final testing.A partial grid search selects feature-extraction and neural-architecture hyperparameters through iterated one-parameter searches.
  • Dataset construction: The final feature-vector dataset contains 8737 Digital Twin samples and 10945 real-world samples, each with six features.Among the real-world samples, 2162 are labeled anomalies.
  • Training details: Weakly supervised training uses only 10 randomly selected labeled anomalies, and neural models use Adam, early stopping, and gradient clipping.Each experiment is repeated 10 times, with mean and standard deviation reported on the test set.

VII. EXPERIMENTAL RESULTS

The experiments compare unsupervised and weakly-supervised anomaly detection methods on real-world facility-monitoring data, using Digital Twin data for training.

  • The study evaluates proposed unsupervised and weakly-supervised methods against state-of-the-art anomaly detection algorithms on real-world data.

A. Results obtained with Digital Twin data simulation

Digital Twin-trained weakly-supervised methods outperform unsupervised approaches, with SAE achieving the strongest reported performance while requiring only a few labeled anomalies.

  • 10 randomly selected anomalous samples were added to training for the weakly-supervised methods.
  • AP = 0.84 was achieved by the best unsupervised method, FF-AE, although its training time increased substantially.
  • SAE achieved AP = 0.872, AUC ROC = 0.935, and F2 = 0.823, outperforming all evaluated unsupervised methods across the reported metrics.Its performance showed fairly high variance across experiments.
  • False positive rate decreased from 12% with unsupervised training to less than 10% with weakly-supervised training, reaching about 9% for SAE.The authors describe the SAE rate as large but just about acceptable for the real-world application.
  • CNN-SAE accurately reconstructs normal time series, whereas anomalous samples produce an uncorrelated output.
  • The Siamese approaches achieved AP above 0.85 with only 5 labeled anomaly samples.Performance generally increased with more labeled anomalies, but noisy labels may reduce the benefit of additional samples.

B. Results obtained with only Real-World data

Training only on real-world data preserves the ranking pattern but lowers performance because the training data include anomalous samples rather than reflecting only normal operation.

  • Using only real-world unlabeled data retained FF-AE as the best unsupervised algorithm and SAE as the best overall algorithm.Weakly-supervised training again used 10 randomly selected labeled anomalous samples.
  • Performance was lower for every investigated algorithm on both AP and ROC when trained only with real-world data.
  • Table III compares evaluated algorithms trained with Digital Twin data, highlighting globally best and best unsupervised scores with different formatting.Weakly-supervised methods used only 10 additional labeled anomalies.
  • Table IV reports AP, ROC, and percentage differences relative to training with Digital Twin data for algorithms trained only on real-world data.
  • The real-world training data were noisy and contained anomalous samples instead of representing only normal machinery operation modes.

VIII. FINAL REMARKS AND CONCLUSION

The paper proposes Digital Twin-trained weakly supervised anomaly detectors combining simulated normal-operation data with a small set of labeled real anomalies. Across evaluations, the Siamese Autoencoder performs best, while performance declines substantially without simulated data and remains imperfect in real-world use.

  • The proposed methods target multivariate time-series anomaly detection using Digital Twin simulation data and a small labeled anomaly set from actual monitoring.
  • Simple one-day mean and standard-deviation features with one-hour stride achieve the best observed feature-extraction results.
  • Performance degrades markedly for all investigated algorithms when only real-world data replaces Digital Twin data during training.
  • The Siamese Autoencoder operating on time-series features is the overall best-performing algorithm, while Feed Forward Autoencoder performs best among unsupervised methods.
  • All proposed weakly supervised algorithms outperform the compared state-of-the-art approaches across all reported performance measures.
  • The best method has false-positive and false-negative rates of around 9% and 12% in each class, respectively, which remains large for real-world deployment.The authors characterize the work as a generic proof of concept and suggest application-specific improvements, including learned thresholds and real-world normal states.
Loading 2011.06296v1…