PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal Anomaly Detection: A Survey

Xudong Mou, Zexin Wu, Chuan Luo, Shiru Chen, Xudong Liu, Chunming Hu, Renyu Yang

arXiv:2608.24937v1cs.LGcs.AI

TL;DR

MMAD research is fragmented despite the need to detect rare events across heterogeneous modalities, where anomalies may arise from cross-modal interactions. This survey formalizes MMAD and organizes prior work by normality and abnormality assumptions, while examining foundation models and future directions. It concludes with a unified view of the field’s challenges, methods, benchmarks, and open questions.

  • Problem

    MMAD lacks a unified anomaly-centered survey despite heterogeneous modalities creating alignment, semantic, and cross-modal anomaly challenges.

  • Method

    The survey formalizes MMAD and reviews methods through complementary normality-modeling and abnormality-injection paradigms.

  • Results

    The survey provides a structured, assumption-driven synthesis that clarifies the strengths and limitations of MMAD approaches, including foundation-model-based methods.

  • Takeaways & Limitations

    The survey identifies assumption fusion, reliable anomaly hypothesis generation, semantic crossmodal reasoning, and lifelong adaptation as future directions.

Abstract

from arXiv · show

Multi-Modal Anomaly Detection (MMAD) detects rare abnormal events from heterogeneous data sources and is increasingly used in safety- and reliability-critical applications such as industrial inspection and cybersecurity. Yet the literature is fragmented across domains and modality combinations, and existing surveys usually group methods by architecture rather than by how abnormality is defined and separated in multi-modal settings. We survey MMAD from an assumption-driven perspective. We formalize the problem, identify five intrinsic characteristics underlying its core challenges, and organize prior work into two complementary paradigms. The first, normality-assumption methods, models regularity via representation learning, cross-modal alignment, and knowledge enhancement. The second, anomaly-assumption methods, sharpens decision boundaries through coarse-grained, structural, and semantic anomaly injection. We also investigate how foundation models are reshaping MMAD through scalable pretraining, flexible cross-modal transfer, and emerging reasoning capabilities. Finally, we compile representative benchmarks and evaluation protocols across domains and highlight open problems and future directions for robust, adaptive, and interpretable MMAD systems.

I. INTRODUCTION

MMAD addresses failures of unimodal detection by combining heterogeneous evidence, but this integration introduces distinct alignment and semantic challenges. The survey responds with an assumption-driven taxonomy covering normality modeling, abnormality construction, and foundation-model-enabled directions.

  • I. INTRODUCTION: MMAD combines heterogeneous modalities to detect events that may appear normal in any single modality.Multimodal integration can also improve robustness to noise, missing data, and partial failures.
  • I. INTRODUCTION: Existing MMAD surveys are fragmented across domains or focus on general learning tools rather than a unified anomaly-centered perspective.Prior reviews target industrial RGB/3D data, audio–visual benchmarks, time-series signals, or collaboration and fusion methods.
  • I. INTRODUCTION: The survey organizes MMAD methods by assumptions about normality and abnormality instead of modality or model family.Normality-based methods model regular patterns implicitly, whereas abnormality-based methods explicitly construct contrasts beyond normality.
  • I. INTRODUCTION: The survey analyzes MMAD’s intrinsic nature and fundamental challenges, then reviews methods through complementary normality- and abnormality-centric lenses.This framing summarizes the objectives, capabilities, and limitations of existing approaches.
  • I. INTRODUCTION: Foundation-model directions emphasize reliable anomaly hypothesis generation, semantic crossmodal reasoning, and lifelong adaptation.The survey also highlights emerging opportunities for assumption fusion through foundation models.

A. Definition of Multi-modal Anomaly Detection

MMAD assigns anomaly scores to partially observed multimodal samples by measuring deviation from a joint normal distribution. Its formulation must account for within-modality regularities, cross-modal dependencies, class imbalance, and distinctions among related detection settings.

  • A. Definition of Multi-modal Anomaly Detection: MMAD integrates heterogeneous modalities to identify rare samples that deviate substantially from the normal distribution.Cross-modal complementarity can reveal patterns obscured in unimodal views.
  • A. Definition of Multi-modal Anomaly Detection: A multimodal sample X contains K modality observations, while M identifies observed modalities when some inputs are missing.The observed subset is represented as X_M.
  • A. Definition of Multi-modal Anomaly Detection: The scoring function f produces anomaly score S by measuring deviation from the joint normal distribution P_normal.The score is computed from the available multimodal observations.
  • A. Definition of Multi-modal Anomaly Detection: The decision rule labels a sample anomalous when S > τ and normal when S ≤ τ.τ is typically determined on a validation set.
  • A. Definition of Multi-modal Anomaly Detection: MMAD must capture both within-modality regularities and cross-modal dependencies, including inconsistency, imbalance, and missing modalities.The formulation also stresses aligned multimodal normality, extreme class imbalance, and distinctions among AD, ND, OD, and OOD.

2) Exacerbation introduced by multimodality:

Multimodality amplifies anomaly-detection difficulty through heterogeneous, asynchronous, and potentially contradictory signals. The survey identifies five defining characteristics and shows why alignment, domain knowledge, and multiscale sensitivity remain central challenges.

  • 2) Exacerbation introduced by multimodality:: Multimodal data introduce inconsistent distributions, asynchronous sampling, and contradictory signals across disparate sensors.These issues make scarce abnormal signals harder to reconcile across unaligned feature spaces.
  • 2) Exacerbation introduced by multimodality:: An event can appear normal within one stream yet become anomalous through spatial, relational, temporal, or semantic interactions across modalities.Cross-modal anomalies therefore extend anomaly diversity beyond single-stream manifestations.
  • 2) Exacerbation introduced by multimodality:: Concept drift is multidimensional because modalities may evolve at different rates or in opposing directions, making calibration harder.This is one of the multimodal complications identified in the survey roadmap.
  • 2) Exacerbation introduced by multimodality:: MMAD’s five key characteristics are modal heterogeneity, knowledge deficit, anomaly diversity, anomaly scarcity, and concept drift.These characteristics summarize the field’s core multimodal and intrinsic challenges.
  • 2) Exacerbation introduced by multimodality:: Accurate semantic alignment is necessary when modality-specific evidence differs, such as force changes from a visually imperceptible robotic micro-slip.Without alignment, visual bias can cause the model to miss the slip.
  • 2) Exacerbation introduced by multimodality:: Anomaly boundaries are blurred by limited expertise, weak or noisy annotations, legitimate rare states, and contamination of normal baselines by failure patterns.Domain priors such as the 3σ rule can provide constraints when data alone cannot define the boundary.
  • 2) Exacerbation introduced by multimodality:: Anomalies range from point-level outliers to long-horizon patterns and large structural deviations, requiring sensitivity across temporal and spatial granularities.Brute-force attacks may be high-frequency point anomalies, whereas low-and-slow exfiltration can emerge over weeks.

4) CH4: Representation fragility from anomaly scarcity:

Anomaly scarcity makes direct discriminative learning unreliable, so MMAD methods often characterize predominantly normal data and detect deviations. However, evolving normality can make learned manifolds obsolete without adaptation.

  • Rare anomalies create severe class imbalance, forcing detectors to rely heavily on normality assumptions and potentially reducing recall for unseen patterns.Critical infrastructure may contain black-swan failures absent from observed histories.
  • Non-stationary environments can shift healthy data statistics through aging or seasonal changes, causing fixed models to flag legitimate shifts as anomalies.The passage identifies lifelong learning as an adaptation need.
  • Normality-assumption methods model consistent, compact multi-modal association patterns and identify anomalies as deviations from those norms.
  • Normality-based scoring commonly uses an encoder followed by a scoring head, with reconstruction error, normal-density likelihood, or distance to a compact normal set as criteria.
  • Normality feature learning maps possibly incomplete observations into embeddings that capture modality-specific regularities and cross-modal consistency.The embedding serves as an implicit normality prior.

1) Generic Normality Feature Learning:

Generic normality feature learning characterizes normal distributions to detect low-probability inputs, while related boundary and cross-modal methods impose compactness or consistency constraints. These approaches provide explicit or enriched normal representations but remain vulnerable to heterogeneous, non-stationary, and weakly aligned modalities.

  • Generic Normality Feature Learning: Learning normal-pattern features can detect unknown anomalies by identifying inputs in low-probability density regions of the modeled normal distribution.
  • Generic Normality Feature Learning: Audio-visual sequence modeling and mixture-of-experts systems illustrate normality learning that captures temporal synchronization or combines complementary modality-specific detectors.
  • Boundary-constrained Feature Learning: Boundary-constrained methods enclose normal samples with explicit feature-space boundaries, directly addressing ambiguous decision boundaries and producing distance-based anomaly scores.
  • Boundary-constrained Feature Learning: Compactness constraints can yield loose or unstable boundaries on heterogeneous or non-stationary data, while encoder capacity gaps can cause modality dominance and collapse.
  • Cross-modal Representation Learning: Cross-modal representation learning models normal inter-modality consistency and detects anomalies through disagreement or deviations from a learned normal region.
  • Cross-modal Representation Learning: Cross-modal fusion spans early, late, and hybrid designs, with hybrid intermediate fusion often effective for complex inter-modal dependencies despite higher computational demand.
  • Cross-modal Representation Learning: Global synchronization bridges heterogeneity and leverages complementary information, but may smooth local details and reduce sensitivity to fine-grained anomalies.

2) Local Consistency Verification:

Local Consistency Verification shifts MMAD from global semantic mapping to fine-grained correspondence and token-level checks. It improves sensitivity to subtle local deviations but depends on reliable correspondences and can be computationally expensive.

  • Local correspondence matching: Token-level verification directly addresses granularity imbalance and improves anomaly localization and overall detection performance.The approach shifts from global semantic mapping toward patch-, point-, or region-level verification.
  • Local correspondence matching: Fine-grained correspondence matching enforces local alignments using geometric or structural priors, revealing anomalies through matching failures or high correspondence errors.Typical settings include RGB–depth, image–point cloud, and 3D industrial inspection.
  • Advantages: Local Consistency Verification detects subtle local deviations that global semantic synchronization may miss.Patch-to-token alignment helps prevent high-density information in one modality from being suppressed by lower-density signals in another.
  • Limitations: The approach relies on reliable local correspondences or stable attention patterns, limiting performance when modalities are misaligned, noisy, or partially missing.Fine-grained matching and attention operations can also limit scalability for high-resolution or large-scale multimodal inputs.
  • Domain knowledge augmentation: Knowledge enhancement incorporates structured priors to build more compact and robust representations of normal patterns.Knowledge constraints guide models toward the semantic logic of normality in heterogeneous multimodal data.
  • Domain knowledge augmentation: Cross-modal knowledge distillation transfers discriminative and semantic information from informative modalities to target models, but depends on teacher quality and modality availability.Modality heterogeneity can cause information loss, misalignment, or diluted anomaly signals in fused teacher representations.

2) Prototype Memory Mechanism:

Prototype Memory Mechanisms model normality with a finite memory of representative normal patterns, while abnormality-assumption methods construct contrasts outside normal support. These approaches sharpen boundaries but face trade-offs in coverage, purity, and pseudo-anomaly validation.

  • Prototype Memory Mechanism: Prototype memory constrains the discrimination boundary by representing normal samples with a finite set of prototypes in embedding space.The memory acts as a domain prior containing only normal patterns.
  • Prototype Memory Mechanism: Insufficient prototype capacity limits coverage of normal variation, whereas excessive capacity risks memorizing anomalies or noise.The resulting trade-off is between memory capacity and representational diversity.
  • Abnormality assumption: Abnormality-assumption methods explicitly construct contrasts that expose deviations rather than only reconstructing or enforcing self-consistency on normal data.They use domain knowledge to synthesize signals representing the flip side of the data distribution.
  • Abnormality assumption: Injected samples are generated by transforming normal observations, and detectors are trained to assign them higher anomaly scores.Ranking or binary classification losses can treat injected samples as pseudo anomalies.
  • Abnormality assumption: Pseudo-anomalies probe outside normal support to tighten the decision boundary without being intended to match the true anomaly distribution.This distinguishes abnormality-assumption training from standard self-supervised learning, which primarily learns invariant representations.
  • Abnormality assumption: Abnormality-assumption methods progress from coarse-grained to structural and semantic injection, with pseudo-anomaly rationality becoming harder to validate at higher semantic levels.Coarse and structural schemes commonly use heuristic or physical constraints, whereas semantic completion requires different criteria.

1) External Exposure:

External Exposure treats data outside the training distribution as proxy anomalies, while internal perturbation creates modality-agnostic corruptions from in-distribution samples. Both provide coarse contrasts that regularize the boundary, but auxiliary-data quality and realism remain constraints.

  • External Exposure: Outlier Exposure samples proxy anomalies from an auxiliary external dataset under an open-world assumption.Exposure to foreign examples is intended to compact the decision boundary around normal data.
  • External Exposure: Latent Outlier Exposure uses auxiliary outliers as latent-space regularizers under contaminated training data, with extensions to time series, robotic vision, and anomalous sound detection.The supplied passage describes LOE as preventing representation collapse in contaminated settings.
  • Injection granularity: Coarse-grained injection corrupts low-level statistics through within-modality perturbations or outlier exposure.The figure categorizes this alongside structural disruption and semantic inconsistency synthesis as three pseudo-anomaly granularities.
  • Injection granularity: Structural injection breaks relational integrity by disturbing links or temporal order within or across modalities.The figure contrasts this with coarse-grained statistical corruption and semantic logical inconsistency.
  • External Exposure: Poorly aligned or contaminated auxiliary sets can distort the shared decision boundary and reduce the effectiveness of external exposure.Exposure quality is therefore a central practical constraint in multimodal settings.
  • Internal perturbation: Internal perturbation synthesizes pseudo-anomalies by locally disrupting continuity or statistics without relying on external data.Masking and shuffling are modality-agnostic primitives that help bypass data heterogeneity.
  • Internal perturbation: Internal perturbation offers a simple, computationally efficient, modality-agnostic response to anomaly scarcity by generating diverse pseudo-anomalies from in-distribution data.Reported applications include multimodal fusion, vehicle-damage detection, cloud anomaly classification, and cyber-physical systems.

1) Temporal Structure Disruption:

Anomaly injection methods create pseudo-anomalies by disrupting temporal order, relational topology, or semantic plausibility. These strategies improve granularity, cross-modal inconsistency modeling, and explainability, but depend on calibrated perturbations and suitable structures.

  • Temporal Structure Disruption:: Temporal disruption methods violate sequence order, speed, or predictive continuity to simulate contextual deviations more finely than coarse masking.Their applicability is limited for non-sequential data, while aggressive or poorly calibrated disruptions may introduce unrealistic artifacts.
  • Relational Structure Disruption:: Relational disruption methods perturb edges, attributes, relations, or subgraphs to model network-level inconsistencies across multiple relational scales.Multi-modal variants disrupt dependencies between modalities, including visual-textual relations and point cloud-image alignments.
  • Relational Structure Disruption:: Cross-modal relational disruption exposes mismatched predicates, desynchronized sensor graphs, and other relational voids between modalities.These pseudo-anomalies support unified representations across fused relational spaces, but require calibrated disruptions and well-defined structures.
  • Semantic Anomaly Injection:: Semantic injection creates knowledge-guided pseudo-anomalies that violate domain logic or contextual plausibility, addressing ambiguity and limited anomaly examples.It uses semantic priors, prompts, and generative or multi-modal foundation models without explicitly modeling structural relations.
  • Semantic Anomaly Injection:: MLLM-based methods extend semantic anomaly detection toward spatially aware zero-shot detection and explainable justifications for video anomalies.Knowledge-driven prompting supplies semantically rich supervision and supports precise localization through explicit contrastive evidence rather than opaque reconstruction error.

2) Generative Data Completion:

Generative data completion synthesizes semantically coherent pseudo-anomalies from normal data by violating logical or contextual consistency. It offers high-fidelity anomaly modeling but remains constrained by computational cost, conditioning requirements, and generator exposure.

  • Generative architectures create semantically implausible continuations or completions by introducing controlled violations into normal samples.
  • High-fidelity pseudo-anomalies support precise modeling of complex irregularities and help address anomaly rarity through unlimited diverse synthesis.
  • Diffusion models are computationally expensive because iterative sampling limits real-time use, while GANs face instability and mode collapse.
  • Foundation models act as semantic generators and reasoners, supporting promptable pseudo-anomaly synthesis and natural-language rationales for violated cross-modal constraints.
  • MMAD benchmarks span industrial, medical, surveillance, and cyber-physical settings with differing modality compositions, scales, and environmental dynamics.

1) Muliti-granularity evaluation metrics.:

MMAD evaluation requires metrics and protocols matched to spatial, temporal, and modality-specific behavior rather than a single unified measure. Deployment-oriented testing increasingly examines open-set generalization, anomaly-injection fairness, missing-modality robustness, and computational scalability.

  • Multi-granularity evaluation metrics: Metrics should match annotation granularity, including image-level AUROC for sample decisions and Pixel-level AUROC or AUPRO for defect localization.
  • Multi-granularity evaluation metrics: Video and audio evaluation commonly uses Frame-level AUROC, Average Precision under class imbalance, and Detection Latency after anomaly onset.
  • Multi-granularity evaluation metrics: Sequential metrics remain fragmented: Point-Wise measures can undervalue shifted predictions, whereas Point-Adjusted F1-score can overestimate partially detected segments.
  • Protocol-level testing: Protocol-level testing evaluates zero-shot detection, fairness of pseudo-anomaly injection, and performance degradation when modalities are missing at inference.
  • Computational complexity and scalability: Standard self-attention can require O(N^2) computation and memory, while diffusion synthesis costs approximately O(T · Cstep), constraining large-scale deployment.
  • Computational complexity and scalability: Compression and parameter-efficient adaptation are proposed to reduce overhead while preserving cross-modal alignment capability.

VI. OPEN PROBLEMS AND FUTURE DIRECTIONS

The survey frames future MMAD research around foundation-model-driven assumption fusion, combining normality alignment with targeted anomaly generation. It identifies reliable anomaly hypotheses, semantic reasoning under cross-modal conflict, and continual adaptation as central open problems.

  • Unifying paradigm: Foundation-model-driven assumption fusion: Future systems should couple foundation-model priors for normal representation alignment with targeted semantic edits or prompts as hard negatives.
  • RQ1: Reliable anomaly hypotheses: Reliable anomaly hypotheses require reference-free validation because pseudo-anomaly realism is difficult to measure and FID may overfit when real anomalies are scarce.
  • RQ2: Semantic reasoning under crossmodal conflicts: Semantic reasoning under cross-modal conflicts calls for uncertainty-aware fusion, semantic grounding, and rationales linked to violated cross-modal constraints.
  • RQ3: Continual adaptation under evolving environments: Continual adaptation must accommodate evolving distributions while preserving stable decision boundaries under constrained resources.
  • Conclusions: The survey positions MMAD as shifting toward holistic systems that combine representation learning, anomaly simulation, semantic reasoning, and deployment-aware adaptation.
  • Conclusions: The intended roadmap targets adaptive, interpretable, and reliable real-world MMAD systems.
Loading 2608.24937v1…