PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal Anomaly Detection: A Survey

Xudong Mou, Zexin Wu, Chuan Luo, Shiru Chen, Xudong Liu, Chunming Hu, Renyu Yang

arXiv:2608.24937v1cs.LGcs.AI

TL;DR

MMAD research is fragmented despite the need to detect rare events across heterogeneous modalities, where anomalies may arise from cross-modal interactions. This survey formalizes MMAD and organizes prior work by normality and abnormality assumptions, while examining foundation models and future directions. It concludes with a unified view of the field’s challenges, methods, benchmarks, and open questions.

  • Problem

    MMAD lacks a unified anomaly-centered survey despite heterogeneous modalities creating alignment, semantic, and cross-modal anomaly challenges.

    Source: §I · p. 1

  • Method

    The survey formalizes MMAD and reviews methods through complementary normality-modeling and abnormality-injection paradigms.

    Source: §I · p. 1, A. Definition of Multi-modal Anomaly… · p. 2, §VII · p. 15

  • Results

    The survey provides a structured, assumption-driven synthesis that clarifies the strengths and limitations of MMAD approaches, including foundation-model-based methods.

    Source: §I · p. 2, §VII · p. 15

  • Takeaways & Limitations

    The survey identifies assumption fusion, reliable anomaly hypothesis generation, semantic crossmodal reasoning, and lifelong adaptation as future directions.

    Source: §VII · p. 15

Abstract

from arXiv · show

Multi-Modal Anomaly Detection (MMAD) detects rare abnormal events from heterogeneous data sources and is increasingly used in safety- and reliability-critical applications such as industrial inspection and cybersecurity. Yet the literature is fragmented across domains and modality combinations, and existing surveys usually group methods by architecture rather than by how abnormality is defined and separated in multi-modal settings. We survey MMAD from an assumption-driven perspective. We formalize the problem, identify five intrinsic characteristics underlying its core challenges, and organize prior work into two complementary paradigms. The first, normality-assumption methods, models regularity via representation learning, cross-modal alignment, and knowledge enhancement. The second, anomaly-assumption methods, sharpens decision boundaries through coarse-grained, structural, and semantic anomaly injection. We also investigate how foundation models are reshaping MMAD through scalable pretraining, flexible cross-modal transfer, and emerging reasoning capabilities. Finally, we compile representative benchmarks and evaluation protocols across domains and highlight open problems and future directions for robust, adaptive, and interpretable MMAD systems.

I. INTRODUCTION

MMAD addresses failures of unimodal detection by combining heterogeneous evidence, but this integration introduces distinct alignment and semantic challenges. The survey responds with an assumption-driven taxonomy covering normality modeling, abnormality construction, and foundation-model-enabled directions.

I. INTRODUCTION

  • MMAD combines heterogeneous modalities to detect events that may appear normal in any single modality.Multimodal integration can also improve robustness to noise, missing data, and partial failures.Source: §I · p. 1
  • Existing MMAD surveys are fragmented across domains or focus on general learning tools rather than a unified anomaly-centered perspective.Prior reviews target industrial RGB/3D data, audio–visual benchmarks, time-series signals, or collaboration and fusion methods.Source: §I · p. 1
  • The survey organizes MMAD methods by assumptions about normality and abnormality instead of modality or model family.Normality-based methods model regular patterns implicitly, whereas abnormality-based methods explicitly construct contrasts beyond normality.Source: §I · p. 1
  • The survey analyzes MMAD’s intrinsic nature and fundamental challenges, then reviews methods through complementary normality- and abnormality-centric lenses.This framing summarizes the objectives, capabilities, and limitations of existing approaches.Source: §I · p. 2
  • Foundation-model directions emphasize reliable anomaly hypothesis generation, semantic crossmodal reasoning, and lifelong adaptation.The survey also highlights emerging opportunities for assumption fusion through foundation models.Source: §I · p. 2

A. Definition of Multi-modal Anomaly Detection

MMAD assigns anomaly scores to partially observed multimodal samples by measuring deviation from a joint normal distribution. Its formulation must account for within-modality regularities, cross-modal dependencies, class imbalance, and distinctions among related detection settings.

A. Definition of Multi-modal Anomaly Detection

  • MMAD integrates heterogeneous modalities to identify rare samples that deviate substantially from the normal distribution.Cross-modal complementarity can reveal patterns obscured in unimodal views.Source: A. Definition of Multi-modal Anomaly… · p. 2
  • A multimodal sample X contains K modality observations, while M identifies observed modalities when some inputs are missing.The observed subset is represented as X_M.Source: A. Definition of Multi-modal Anomaly… · p. 2
  • The scoring function f produces anomaly score S by measuring deviation from the joint normal distribution P_normal.The score is computed from the available multimodal observations.Source: A. Definition of Multi-modal Anomaly… · p. 2
  • The decision rule labels a sample anomalous when S > τ and normal when S ≤ τ.τ is typically determined on a validation set.Source: A. Definition of Multi-modal Anomaly… · p. 2
  • MMAD must capture both within-modality regularities and cross-modal dependencies, including inconsistency, imbalance, and missing modalities.The formulation also stresses aligned multimodal normality, extreme class imbalance, and distinctions among AD, ND, OD, and OOD.Source: A. Definition of Multi-modal Anomaly… · p. 2

2) Exacerbation introduced by multimodality:

Multimodality amplifies anomaly-detection difficulty through heterogeneous, asynchronous, and potentially contradictory signals. The survey identifies five defining characteristics and shows why alignment, domain knowledge, and multiscale sensitivity remain central challenges.

2) Exacerbation introduced by multimodality:

  • Multimodal data introduce inconsistent distributions, asynchronous sampling, and contradictory signals across disparate sensors.These issues make scarce abnormal signals harder to reconcile across unaligned feature spaces.Source: 2) Exacerbation introduced by… · p. 2, 1) CH1: Modal discrepancy from… · p. 3
  • An event can appear normal within one stream yet become anomalous through spatial, relational, temporal, or semantic interactions across modalities.Cross-modal anomalies therefore extend anomaly diversity beyond single-stream manifestations.Source: 2) Exacerbation introduced by… · p. 2
  • Concept drift is multidimensional because modalities may evolve at different rates or in opposing directions, making calibration harder.This is one of the multimodal complications identified in the survey roadmap.Source: 2) Exacerbation introduced by… · p. 3
  • MMAD’s five key characteristics are modal heterogeneity, knowledge deficit, anomaly diversity, anomaly scarcity, and concept drift.These characteristics summarize the field’s core multimodal and intrinsic challenges.Source: 2) Exacerbation introduced by… · p. 3
  • Accurate semantic alignment is necessary when modality-specific evidence differs, such as force changes from a visually imperceptible robotic micro-slip.Without alignment, visual bias can cause the model to miss the slip.Source: 1) CH1: Modal discrepancy from… · p. 3
  • Anomaly boundaries are blurred by limited expertise, weak or noisy annotations, legitimate rare states, and contamination of normal baselines by failure patterns.Domain priors such as the 3σ rule can provide constraints when data alone cannot define the boundary.Source: 2) CH2: Boundary ambiguity from… · p. 3
  • Anomalies range from point-level outliers to long-horizon patterns and large structural deviations, requiring sensitivity across temporal and spatial granularities.Brute-force attacks may be high-frequency point anomalies, whereas low-and-slow exfiltration can emerge over weeks.Source: 3) CH3: Sensitivity imbalance from… · p. 3

4) CH4: Representation fragility from anomaly scarcity:

Anomaly scarcity makes direct discriminative learning unreliable, so MMAD methods often characterize predominantly normal data and detect deviations. However, evolving normality can make learned manifolds obsolete without adaptation.

  • Rare anomalies create severe class imbalance, forcing detectors to rely heavily on normality assumptions and potentially reducing recall for unseen patterns.Critical infrastructure may contain black-swan failures absent from observed histories.Source: 4) CH4: Representation fragility from… · p. 3
  • Non-stationary environments can shift healthy data statistics through aging or seasonal changes, causing fixed models to flag legitimate shifts as anomalies.The passage identifies lifelong learning as an adaptation need.Source: 5) CH5: Performance decay from concept… · p. 3
  • Normality-assumption methods model consistent, compact multi-modal association patterns and identify anomalies as deviations from those norms.Source: §III · p. 4
  • Normality-based scoring commonly uses an encoder followed by a scoring head, with reconstruction error, normal-density likelihood, or distance to a compact normal set as criteria.Source: §III · p. 4
  • Normality feature learning maps possibly incomplete observations into embeddings that capture modality-specific regularities and cross-modal consistency.The embedding serves as an implicit normality prior.Source: A. Normality Feature Learning · p. 4

1) Generic Normality Feature Learning:

Generic normality feature learning characterizes normal distributions to detect low-probability inputs, while related boundary and cross-modal methods impose compactness or consistency constraints. These approaches provide explicit or enriched normal representations but remain vulnerable to heterogeneous, non-stationary, and weakly aligned modalities.

Generic Normality Feature Learning

Boundary-constrained Feature Learning

Cross-modal Representation Learning

2) Local Consistency Verification:

Local Consistency Verification shifts MMAD from global semantic mapping to fine-grained correspondence and token-level checks. It improves sensitivity to subtle local deviations but depends on reliable correspondences and can be computationally expensive.

Local correspondence matching

  • Token-level verification directly addresses granularity imbalance and improves anomaly localization and overall detection performance.The approach shifts from global semantic mapping toward patch-, point-, or region-level verification.Source: 2) Local Consistency Verification · p. 6
  • Fine-grained correspondence matching enforces local alignments using geometric or structural priors, revealing anomalies through matching failures or high correspondence errors.Typical settings include RGB–depth, image–point cloud, and 3D industrial inspection.Source: 2) Local Consistency Verification · p. 6

Advantages

  • Local Consistency Verification detects subtle local deviations that global semantic synchronization may miss.Patch-to-token alignment helps prevent high-density information in one modality from being suppressed by lower-density signals in another.Source: 2) Local Consistency Verification · p. 6

Limitations

  • The approach relies on reliable local correspondences or stable attention patterns, limiting performance when modalities are misaligned, noisy, or partially missing.Fine-grained matching and attention operations can also limit scalability for high-resolution or large-scale multimodal inputs.Source: 2) Local Consistency Verification · p. 6

Domain knowledge augmentation

  • Knowledge enhancement incorporates structured priors to build more compact and robust representations of normal patterns.Knowledge constraints guide models toward the semantic logic of normality in heterogeneous multimodal data.Source: C. Domain Knowledge Augmentation · p. 7, 1) Cross-modal Knowledge Distillation · p. 7
  • Cross-modal knowledge distillation transfers discriminative and semantic information from informative modalities to target models, but depends on teacher quality and modality availability.Modality heterogeneity can cause information loss, misalignment, or diluted anomaly signals in fused teacher representations.Source: 1) Cross-modal Knowledge Distillation · p. 7

2) Prototype Memory Mechanism:

Prototype Memory Mechanisms model normality with a finite memory of representative normal patterns, while abnormality-assumption methods construct contrasts outside normal support. These approaches sharpen boundaries but face trade-offs in coverage, purity, and pseudo-anomaly validation.

Prototype Memory Mechanism

  • Prototype memory constrains the discrimination boundary by representing normal samples with a finite set of prototypes in embedding space.The memory acts as a domain prior containing only normal patterns.Source: 2) Prototype Memory Mechanism · p. 7
  • Insufficient prototype capacity limits coverage of normal variation, whereas excessive capacity risks memorizing anomalies or noise.The resulting trade-off is between memory capacity and representational diversity.Source: 2) Prototype Memory Mechanism · pp. 7–8

Abnormality assumption

  • Abnormality-assumption methods explicitly construct contrasts that expose deviations rather than only reconstructing or enforcing self-consistency on normal data.They use domain knowledge to synthesize signals representing the flip side of the data distribution.Source: §IV · p. 8
  • Injected samples are generated by transforming normal observations, and detectors are trained to assign them higher anomaly scores.Ranking or binary classification losses can treat injected samples as pseudo anomalies.Source: §IV · p. 8
  • Pseudo-anomalies probe outside normal support to tighten the decision boundary without being intended to match the true anomaly distribution.This distinguishes abnormality-assumption training from standard self-supervised learning, which primarily learns invariant representations.Source: §IV · p. 8
  • Abnormality-assumption methods progress from coarse-grained to structural and semantic injection, with pseudo-anomaly rationality becoming harder to validate at higher semantic levels.Coarse and structural schemes commonly use heuristic or physical constraints, whereas semantic completion requires different criteria.Source: §IV · p. 8

1) External Exposure:

External Exposure treats data outside the training distribution as proxy anomalies, while internal perturbation creates modality-agnostic corruptions from in-distribution samples. Both provide coarse contrasts that regularize the boundary, but auxiliary-data quality and realism remain constraints.

External Exposure

  • Outlier Exposure samples proxy anomalies from an auxiliary external dataset under an open-world assumption.Exposure to foreign examples is intended to compact the decision boundary around normal data.Source: 1) External Exposure · p. 8
  • Latent Outlier Exposure uses auxiliary outliers as latent-space regularizers under contaminated training data, with extensions to time series, robotic vision, and anomalous sound detection.The supplied passage describes LOE as preventing representation collapse in contaminated settings.Source: 1) External Exposure · p. 8

Injection granularity

  • Coarse-grained injection corrupts low-level statistics through within-modality perturbations or outlier exposure.The figure categorizes this alongside structural disruption and semantic inconsistency synthesis as three pseudo-anomaly granularities.Source: Figure 3 · p. 9
  • Structural injection breaks relational integrity by disturbing links or temporal order within or across modalities.The figure contrasts this with coarse-grained statistical corruption and semantic logical inconsistency.Source: Figure 3 · p. 9

External Exposure

  • Poorly aligned or contaminated auxiliary sets can distort the shared decision boundary and reduce the effectiveness of external exposure.Exposure quality is therefore a central practical constraint in multimodal settings.Source: 1) External Exposure · p. 9

Internal perturbation

  • Internal perturbation synthesizes pseudo-anomalies by locally disrupting continuity or statistics without relying on external data.Masking and shuffling are modality-agnostic primitives that help bypass data heterogeneity.Source: 2) Internal Perturbation · p. 9
  • Internal perturbation offers a simple, computationally efficient, modality-agnostic response to anomaly scarcity by generating diverse pseudo-anomalies from in-distribution data.Reported applications include multimodal fusion, vehicle-damage detection, cloud anomaly classification, and cyber-physical systems.Source: 2) Internal Perturbation · p. 9

1) Temporal Structure Disruption:

Anomaly injection methods create pseudo-anomalies by disrupting temporal order, relational topology, or semantic plausibility. These strategies improve granularity, cross-modal inconsistency modeling, and explainability, but depend on calibrated perturbations and suitable structures.

Temporal Structure Disruption:

  • Temporal disruption methods violate sequence order, speed, or predictive continuity to simulate contextual deviations more finely than coarse masking.Their applicability is limited for non-sequential data, while aggressive or poorly calibrated disruptions may introduce unrealistic artifacts.Source: 1) Temporal Structure Disruption · p. 10

Relational Structure Disruption:

  • Relational disruption methods perturb edges, attributes, relations, or subgraphs to model network-level inconsistencies across multiple relational scales.Multi-modal variants disrupt dependencies between modalities, including visual-textual relations and point cloud-image alignments.Source: 2) Relational Structure Disruption · p. 10
  • Cross-modal relational disruption exposes mismatched predicates, desynchronized sensor graphs, and other relational voids between modalities.These pseudo-anomalies support unified representations across fused relational spaces, but require calibrated disruptions and well-defined structures.Source: 2) Relational Structure Disruption · p. 10

Semantic Anomaly Injection:

  • Semantic injection creates knowledge-guided pseudo-anomalies that violate domain logic or contextual plausibility, addressing ambiguity and limited anomaly examples.It uses semantic priors, prompts, and generative or multi-modal foundation models without explicitly modeling structural relations.Source: C. Semantic Injection · p. 11, 1) Knowledge-driven Prompting · p. 11
  • MLLM-based methods extend semantic anomaly detection toward spatially aware zero-shot detection and explainable justifications for video anomalies.Knowledge-driven prompting supplies semantically rich supervision and supports precise localization through explicit contrastive evidence rather than opaque reconstruction error.Source: 1) Knowledge-driven Prompting · p. 11

2) Generative Data Completion:

Generative data completion synthesizes semantically coherent pseudo-anomalies from normal data by violating logical or contextual consistency. It offers high-fidelity anomaly modeling but remains constrained by computational cost, conditioning requirements, and generator exposure.

1) Muliti-granularity evaluation metrics.:

MMAD evaluation requires metrics and protocols matched to spatial, temporal, and modality-specific behavior rather than a single unified measure. Deployment-oriented testing increasingly examines open-set generalization, anomaly-injection fairness, missing-modality robustness, and computational scalability.

Multi-granularity evaluation metrics

Protocol-level testing

  • Protocol-level testing evaluates zero-shot detection, fairness of pseudo-anomaly injection, and performance degradation when modalities are missing at inference.Source: 2) Emerging evaluation protocols · p. 14

Computational complexity and scalability

VI. OPEN PROBLEMS AND FUTURE DIRECTIONS

The survey frames future MMAD research around foundation-model-driven assumption fusion, combining normality alignment with targeted anomaly generation. It identifies reliable anomaly hypotheses, semantic reasoning under cross-modal conflict, and continual adaptation as central open problems.

Unifying paradigm: Foundation-model-driven assumption fusion

  • Future systems should couple foundation-model priors for normal representation alignment with targeted semantic edits or prompts as hard negatives.Source: §VI · p. 15

RQ1: Reliable anomaly hypotheses

  • Reliable anomaly hypotheses require reference-free validation because pseudo-anomaly realism is difficult to measure and FID may overfit when real anomalies are scarce.Source: §VI · p. 15

RQ2: Semantic reasoning under crossmodal conflicts

  • Semantic reasoning under cross-modal conflicts calls for uncertainty-aware fusion, semantic grounding, and rationales linked to violated cross-modal constraints.Source: §VI · p. 15

RQ3: Continual adaptation under evolving environments

  • Continual adaptation must accommodate evolving distributions while preserving stable decision boundaries under constrained resources.Source: §VI · p. 15

Conclusions

  • The survey positions MMAD as shifting toward holistic systems that combine representation learning, anomaly simulation, semantic reasoning, and deployment-aware adaptation.Source: §VII · p. 15
  • The intended roadmap targets adaptive, interpretable, and reliable real-world MMAD systems.Source: §VII · p. 15

Related papers

Loading 2608.24937v1…