PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Fault Detection in Mechanical Multimodal Time Series via Self-Supervised Cross-Modal Reconstruction

Magnus Munk Jensen, Dorte Hammershøi, Rafał Wiśniewski, Olga Fink

arXiv:2609.16314v1cs.LGstat.AP

TL;DR

Fault detection under changing operating conditions is difficult because shifts can reduce performance in unseen regimes, while heterogeneous modalities are not fully exploited by independent processing or simple fusion. The paper uses cross-modal reconstruction and adaptive test-time thresholding, reporting improved robustness across three industrial datasets, especially in challenging out-of-distribution conditions.

  • Problem

    Operating-condition shifts can degrade fault detection in unseen regimes, and independent modality processing or simple fusion limits use of cross-modal relationships.

    Source: §1 · p. 2, §2.1 · p. 4

  • Method

    The framework reconstructs each heterogeneous time-series modality from the others and applies an adaptive test-time threshold to cross-modal reconstruction error.

    Source: §1 · p. 3, §3.2 · p. 6

  • Results

    Across three datasets, cross-modal reconstruction improves robustness under out-of-distribution conditions, with the strongest gains in the most challenging regimes.

    Source: §7 · p. 13, §8 · p. 14

  • Takeaways & Limitations

    Dynamic per-condition thresholding improves robustness over static thresholding, particularly with heterogeneous operating conditions and incomplete training coverage.

    Source: §7 · p. 13

    The formulation assumes discrete condition changes, and unmeasured sources of domain shift may degrade performance.

    Source: §8 · p. 14

Abstract

from arXiv · show

Fault detection is essential in industrial systems, enabling early identification of abnormal behaviour and improving safety, reliability, and operational efficiency. Modern systems increasingly rely on heterogeneous sensing modalities that capture complementary aspects of the underlying physical process. However, existing data-driven anomaly detection methods often process each modality independently or use simple feature-level fusion, limiting their ability to exploit cross-modal relationships that characterize normal system behaviour. Their performance also commonly assumes similar training and deployment distributions, whereas real-world operation is affected by changing operating conditions, environmental influences, and system degradation that induce distribution shifts and reduce detection performance, especially in unseen regimes. In this work, we propose a multimodal anomaly detection framework based on cross-modal reconstruction of heterogeneous time-series sensor data. Rather than modeling each modality independently, the framework learns system dynamics by reconstructing each modality from the others, thereby exploiting complementary information across modalities. This integrates information across sensing channels without requiring explicit temporal alignment or identical sampling rates, while improving robustness to sensor noise, missing measurements, and modality-specific disturbances. To address distribution shifts during real-world deployment, anomalies are identified using cross-modal reconstruction error and an adaptive test-time thresholding mechanism that adjusts to changing operating conditions. Experiments on three industrial case studies show strong fault detection performance and substantially improved robustness under out-of-distribution conditions, with the largest gains observed in the most challenging operating regimes.

1. Introduction

Fault detection must distinguish fault-induced deviations from changes caused by operating conditions, while deployment shifts can degrade performance in unseen regimes. The paper proposes cross-modal reconstruction to exploit relationships among heterogeneous sensor streams and address this challenge.

1. Introduction

  • Changing operating conditions make healthy sensor measurements vary, complicating the distinction between normal variability and fault-induced deviations.Distribution shifts between training and deployment can substantially degrade detection, particularly in previously unseen conditions.Source: §1 · p. 2
  • Heterogeneous modalities capture complementary aspects of system dynamics, but differing statistical and temporal characteristics complicate effective fusion.Many approaches process modalities independently or fuse at the decision level, limiting their ability to learn consistent cross-modal representations.Source: §1 · pp. 2–3
  • The paper identifies a gap in methods that jointly exploit heterogeneous, asynchronous measurements and remain robust under severe out-of-distribution conditions.Existing anomaly detection methods typically address multimodal representation learning or distribution-shift robustness in isolation.Source: §1 · p. 3
  • The framework reconstructs each sensing modality from the others to learn shared system dynamics from heterogeneous measurements.It combines modality-specific encoders with attention-based latent alignment and uses an adaptive test-time threshold on cross-modal reconstruction error.Source: §1 · p. 3

2. Related work

Related work covers distribution-shift challenges, robust representations, multimodal fusion, and cross-modal self-supervision. The paper situates its approach at their intersection: unsupervised fault detection with asynchronous sensors and severe operating-condition shifts.

2.1. Fault Detection under Distribution Shifts

  • Healthy-only training data are common in industrial fault detection, but shifts in operating conditions can make healthy observations resemble faults and degrade reliability.Fault data are scarce, while changing loads, environments, control strategies, and degradation shift the healthy-data distribution.Source: §2.1 · pp. 3–4

2.2. Learning Robust Representations under Distribution Shifts

  • Domain adaptation and generalization address shifts, but their reliance on available target observations or diverse source domains leaves severe unseen operating regimes challenging.Many robust-representation approaches were developed for supervised tasks and do not directly transfer to unsupervised anomaly detection with only healthy training data.Source: §2.2 · p. 4

2.3. Multimodal Representation Learning for Fault Detection

  • Multimodal methods use fusion or shared representations, yet learned relationships between modalities may change with operating conditions.Asynchronous sensors also make explicit synchronization problematic because it can introduce interpolation errors, delays, and uncertainty.Source: §2.3 · p. 5

2.4. Cross-modal Self-supervised Learning

  • Cross-modal reconstruction predicts one modality from complementary observations, encouraging representations of shared system information rather than sensor-specific artifacts.Its application to industrial multivariate time series remains limited, especially for asynchronous sensors and severe operating-condition shifts.Source: §2.4 · p. 5

3. Preliminaries

The preliminaries model measurements as partial observations of a shared latent system state under changing operating conditions. Fault detection then uses cross-modal reconstruction learned from healthy data, with reconstruction error serving as the anomaly score.

3. Preliminaries

  • The model assumes multiple sensing modalities observe a shared latent state, while each provides only a partial, modality-specific view.No single modality is assumed to fully characterize system dynamics.Source: §3 · p. 6

3.1. Operational Parameters

  • Operating conditions take known values from a finite set of discrete modes, with changes modeled as instantaneous.For unseen-condition evaluation, the formulation assumes a system classified as fault-free before a condition change remains fault-free immediately afterward.Source: §3.1 · p. 6, Figure 1 · p. 6
  • The objective is to learn a health representation that generalizes across conditions while allowing anomaly scores to depend on the operating condition.This separates health representation from condition-dependent scoring.Source: §3.1 · p. 6

3.2. Reconstruction-based fault detection

  • The method reconstructs one modality from complementary modalities using a model trained exclusively on healthy data.At inference, larger reconstruction errors indicate greater likelihood that measurements deviate from normal behavior.Source: §3.2 · p. 6

4. Proposed method

The method learns cross-modal representations through modality-specific encoders, attention-based fusion, and reconstruction objectives, then detects anomalies using reconstruction error and a condition-adaptive threshold. The inference discussion emphasizes that operating conditions shift score calibration and that a single global threshold is impractical.

4. Proposed method

  • At inference, the target modality’s encoder is removed, and its decoder reconstructs that modality from the remaining sensor streams.Reconstruction accuracy depends on the predictive information available in those remaining modalities.Source: §4 · p. 7, Figure 2 · p. 7

4.1. Definitions and training

  • Training combines self-reconstruction, shared-latent reconstruction, and leave-one-out cross-modal reconstruction to preserve modality-specific cues and support reconstruction when modalities are omitted.The self-reconstruction path bypasses fusion; the leave-one-out objective trains each target reconstruction from the other modalities.Source: §4.1 · p. 8

4.2. Inference with Adaptive Test-Time threshold

  • AUC remains high when each operating condition is evaluated separately, but aggregating conditions can degrade performance because shifts primarily affect score calibration.Healthy-score distributions also shift systematically across conditions, making one global decision threshold impractical.Source: §4.2 · p. 8
  • The adaptive test-time threshold maintains condition-specific score statistics and uses a gated, exponentially smoothed update to track gradual drift while excluding large score excursions.The alarm threshold and update-gate parameters are decoupled to control the trade-off between detection sensitivity and adaptation robustness.Source: §4.2 · pp. 8–9

5. Experimental Setup

The evaluation uses three industrial multimodal datasets and tests healthy-only training across in-distribution and shifted operating conditions. Comparisons use early-fusion and unimodal baselines, with model and threshold parameters selected using validation data.

5.1. Datasets

  • Three public datasets span different industrial systems, sensing modalities, fault types, and operating conditions for evaluation under nominal and distribution-shift scenarios.The datasets are AMPERE, KAIST, and IMAD-DS.Source: §5.1 · p. 9

5.1.1. AMPERE

  • AMPERE covers rotor-related faults in a motor test bench; experiments include five health conditions across five load levels, with final evaluation covering all fault types.The dataset contains 25 experiments; initial experiments focus on the least severe fault, one broken rotor bar.Source: §5.1.1 · p. 9

5.1.2. KAIST

  • KAIST measurements used here are vibration, temperature, and motor current, with bearing defects, shaft misalignment, and rotor imbalance across 0, 2, and 4 Nm loads.Rotational speed is excluded because it is constant, and acoustic measurements are omitted because they are incomplete across operating conditions.Source: §5.1.2 · p. 10

5.1.3. IMAD-DS

  • IMAD-DS uses microphone, accelerometer, and gyroscope signals from a brushless motor, with shifts induced by rotational speed and background acoustic noise.Anomalies are controlled mechanical perturbations, and training uses only healthy source-domain data.Source: §5.1.3 · p. 10

5.2. Evaluation Protocol

  • The protocol tests in-distribution data, intermediate conditions withheld during training, and highest conditions withheld during training; target-domain data are excluded from IMAD-DS training.For AMPERE and KAIST, healthy operating data are split into 60%, 20%, and 20% training, validation, and test subsets, while faulty observations are reserved for evaluation.Source: §5.2 · p. 10

5.3. Baselines

  • The proposed framework is compared with an early-fusion reconstruction baseline and with results using each sensing modality individually.The unimodal comparisons quantify the benefit of multimodal representation learning.Source: §5.3 · p. 10

5.4. Implementation Details

  • Model hyperparameters are optimized using leave-one-out cross-modal reconstruction performance on validation data, while adaptive-threshold parameters are selected by grid search using validation F1.Source: §5.4 · p. 10

6. Results

Results cover three industrial datasets, with controlled out-of-distribution comparisons emphasized on AMPERE. There, the proposed method’s advantage is clearest under severe shifts, while performance also remains stable in a restructured test stream.

6. Results

  • Overall AUC pools predictions across operating conditions and is not the average of the per-condition AUC values.It reflects discrimination across the full range of conditions.Source: §6 · p. 10

6.1. Results obtained on AMPERE

  • The AMPERE ablation compares self, shared, and leave-one-out reconstruction components, but its loss-term results precede OPTUNA optimization.The configuration tuple is ordered (Self, Shared, LOO).Source: §6.1 · p. 11, Table 2 · p. 11
  • Under severe OODt shifts, the multimodal method substantially outperforms single-modality and early-fusion approaches; methods are generally comparable under OODm.OODt excludes 50%, 75%, and 100% load from AMPERE training; OODm excludes 25% and 75%.Source: §6.1 · p. 11, Table 1 · p. 11
  • On bearing degradation, performance remains stable and slightly improves in overall accuracy and precision under the restructured test stream.The stream interleaves healthy and faulty samples across conditions after condition-wise healthy initialization; the adaptive-threshold configuration was not individually tuned for this test.Source: §6.1 · p. 12, Table 4 · p. 12
  • Across AMPERE conditions, recall remains 0.995–1.000, while performance variation is primarily in precision, particularly at higher loads.The reported dynamic-threshold results cover different fault types and evaluation settings.Source: Table 3 · p. 11, §6.1 · p. 12

6.2. Results obtained on KAIST

  • KAIST results are reported as AUC scores and overall dynamic-threshold performance.The supplied captions identify the dataset and metric scope but do not provide cell values.Source: Table 5 · p. 12, Table 6 · p. 12

6.3. Results obtained on IMAD-DS

  • IMAD-DS evaluates unseen target speeds of 1000–1400 RPM against higher-speed source domains, reporting AUC and adaptive-threshold metrics.The dataset’s source and target domains are defined by operating speed.Source: §6.3 · pp. 12–13, Table 8 · p. 13, Table 9 · p. 13

7. Discussion

Cross-modal reconstruction improves robustness over unimodal and early-fusion baselines, especially under severe OOD conditions. Dynamic, per-condition thresholding also improves robustness over static thresholding, although OOD AUC degradation remains across case studies.

7. Discussion

  • Cross-modal reconstruction improves robustness over unimodal and early-fusion baselines, particularly under severe OOD conditions; leave-one-out reconstruction primarily drives this robustness.The leave-one-out strategy encourages shared latent structure rather than modality-specific shortcuts.Source: §7 · p. 13
  • Across case studies, OOD conditions still degrade AUC, while online condition-aware threshold adaptation partially compensates at the decision level.The first case study reaches near-perfect in-distribution F1 (approximately 0.99 for 0–50% load), with noticeable accuracy and F1 degradation at 100% load.Source: §7 · p. 13
  • Vibration-based signals can add load-dependent variability that obscures fault-specific patterns and increases overlap between normal and faulty distributions, especially for subtle faults.The paper presents this as a plausible explanation for degradation under extreme conditions.Source: §7 · p. 13
  • Dynamic per-condition thresholding improves robustness over static thresholding, especially with heterogeneous operating conditions and incomplete training coverage.The approach separates representation learning from decision calibration.Source: §7 · p. 13

8. Conclusion

The paper combines multimodal reconstruction with condition-wise, gated exponentially weighted thresholding that adapts at test time. Across three datasets, it reports improved robustness under out-of-distribution operating conditions, especially in the most challenging regimes, while noting limits in the current formulation.

8. Conclusion

  • The proposed approach combines multimodal reconstruction with condition-wise, gated exponentially weighted thresholding that adapts at test time.The conclusion presents these components together as the proposed framework.Source: §8 · p. 14
  • Across three datasets, the method reports improved robustness under out-of-distribution operating conditions, particularly in the most challenging regimes where baseline methods degrade substantially.Source: §8 · p. 14
  • The current formulation assumes discrete condition changes, and unmeasured domain-shift factors such as background noise or latent environmental variations may degrade performance.The authors also identify optimal sensor selection as an area for further investigation.Source: §8 · p. 14

A. Detailed results for the AMPERE case study

The AMPERE appendix reports adaptive-threshold results for four test fault cases. Results are organized by fault type and load condition, with four classification metrics reported.

A. Detailed results for the AMPERE case study

  • Table 10 reports adaptive-threshold AMPERE results for the test case of one broken rotor bar, across load conditions and four metrics.The reported metrics are Accuracy, Precision, Recall, and F1-score.Source: A. Detailed results for the AMPERE… · p. 15, Table 10 · p. 16
  • Table 11 reports adaptive-threshold AMPERE results for three broken rotor bars; this configuration was not tuned separately for that fault type.Results are organized by test fault type and load condition, with Accuracy, Precision, Recall, and F1-score reported.Source: A. Detailed results for the AMPERE… · p. 15, Table 11 · p. 16
  • Table 12 reports adaptive-threshold AMPERE results for four broken rotor bars; this configuration was not tuned separately for that fault type.Results are organized by test fault type and load condition, with Accuracy, Precision, Recall, and F1-score reported.Source: A. Detailed results for the AMPERE… · p. 15, Table 12 · p. 16
  • Table 13 reports adaptive-threshold AMPERE results for bearing degradation; this configuration was not tuned separately for that fault type.Results are organized by test fault type and load condition, with Accuracy, Precision, Recall, and F1-score reported.Source: A. Detailed results for the AMPERE… · p. 15, Table 13 · p. 16

B. Detailed results for the KAIST case study

The KAIST appendix reports adaptive-threshold results across load conditions and distinguishes in-distribution, simple-generalization, and strong-generalization settings. It specifies the test-only torque conditions used for the two out-of-distribution settings.

B. Detailed results for the KAIST case study

C. Detailed results for the IMAD-DS case study

The IMAD-DS appendix reports adaptive-threshold results grouped by domain, distinguishing target-domain from source-domain operating conditions. It reports four detection metrics for each rotational speed.

C. Detailed results for the IMAD-DS case study

D. Hyperparameters

The appendix lists model configurations across AMPERE, KAIST, and IMAD-DS, and records the authors’ contributions.

D. Hyperparameters

CRediT authorship contribution statement

  • Jensen led methodology, investigation, data curation, analysis, validation, and visualization; the other authors contributed supervision, methodology, investigation, or resources.All authors reviewed and edited the manuscript and approved its final version.Source: CRediT authorship contribution statement · p. 17

D. Hyperparameters

Related papers

Loading 2609.16314v1…