PaperlayerView exact revision on arXiv

Source-linked AI summary

Bidirectional Adaptive Transformers for Multimodal Anomaly Detection

Yuxin Jiang, Yunkang Cao, Weiming Shen

arXiv:2610.14598v1cs.CV

TL;DR

Existing multimodal anomaly detection methods often fuse RGB and point-cloud features through static spatial alignment, which may not establish semantic correspondence. BAT adapts pre-trained encoders and uses bidirectional feature prediction to align modalities; it reports state-of-the-art results on MVTec 3D-AD and Eyecandies, including 97.0% image-level AUROC on MVTec 3D-AD.

  • Problem

    Existing multimodal anomaly detection methods often rely on static fusion and spatial alignment without establishing latent semantic correspondence between RGB and point-cloud features.

    Source: §1 · pp. 1–2

  • Method

    BAT uses lightweight adapters to fine-tune pre-trained image and point-cloud models and a bidirectional feature-alignment module for mutual prediction across modalities.

    Source: §1 · p. 2

  • Results

    BAT reports state-of-the-art performance on MVTec 3D-AD and Eyecandies, including 97.0% image-level AUROC on MVTec 3D-AD.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    BAT's bidirectional alignment refines point-cloud texture detail and RGB spatial context for semantically aligned multimodal representations.

    Source: §1 · p. 2

    BAT requires paired RGB and point-cloud training data, which can be challenging to collect in industrial settings.

    Source: §4.4 · p. 11

Abstract

from arXiv · show

The effective integration of features derived from RGB images and 3D point clouds presents a substantial challenge in the field of multimodal anomaly detection (MAD). Existing MAD methods typically rely on static fusion paradigms built upon frozen pre-trained encoders. While these frameworks facilitate spatial alignment, they lack the mechanism to establish latent semantic correspondence, thereby inducing cross-modal interference that degrades anomaly discriminability. To mitigate these limitations, this study proposes a Bidirectional Adaptive Transformers (BAT) framework, which constructs comprehensive and semantically aligned multimodal representations by leveraging complementary attributes across modalities. The BAT framework introduces a Bidirectional Feature Alignment (BFA) module, which employs a mutual predictive mechanism to bridge the gap between 2D textural patterns and 3D geometric structures. This bidirectional interaction explicitly enhances the textural granularity of point cloud descriptors while augmenting the spatial contextual awareness of RGB features. To ensure that the latent representations possess sufficient informational density for alignment, BAT incorporates a parameter-efficient adaptation architecture. By utilizing lightweight image and point cloud adapters, the framework facilitates specialized fine-tuning of pre-trained models, mitigating the information loss inherent in frozen encoders. Extensive experiments conducted on MVTec 3D-AD and Eyecandies confirm that BAT achieves state-of-the-art anomaly detection performance, significantly outperforming existing MAD methods.

1. Introduction

RGB and point clouds provide complementary cues for anomaly detection, but spatial fusion alone may not establish semantic correspondence. BAT combines adaptive modality-specific features with bidirectional prediction, and reports strong results on two benchmarks.

1. Introduction

  • RGB-only methods rely on 2D textural cues, while simplistic multimodal concatenation and spatial alignment may fail to exploit complementary information or establish semantic correspondence.The paper describes this limitation in prior spatially aligned fusion, including M3DM.Source: §1 · pp. 1–2
  • BAT uses lightweight image and point cloud adapters to fine-tune pre-trained models, then applies mutual feature prediction to align the modalities semantically.This bidirectional alignment refines point-cloud textural granularity and RGB spatial context.Source: §1 · p. 2
  • BAT reports 97.0% image-level AUROC on MVTec 3D-AD and superior performance to previous state-of-the-art approaches on MVTec 3D-AD and Eyecandies.The paper also reports enhanced localization accuracy compared with M3DM.Source: §1 · p. 2

2. Related work

RGB anomaly detection uses several approaches to characterize normal patterns, while multimodal methods combine image and point-cloud information. Prior fusion and mapping strategies face limits in exploiting complementary features and establishing semantic correspondence.

2.1. RGB Anomaly Detection

  • RGB anomaly detection methods use reconstruction, feature statistics, or teacher–student divergence to distinguish anomalous patterns from normal features.The reviewed methods include autoencoders, density estimation, memory banks, and knowledge distillation.Source: §2.1 · p. 3

2.2. Multimodal Anomaly Detection

  • Multimodal methods progress from concatenation and spatial fusion to cross-modal mapping, but semantic gaps and modality-specific information can limit correspondence.BAT instead uses trainable adapters and bidirectional alignment to refine RGB spatial awareness and point-cloud texture detail.Source: §2.2 · pp. 3–4

3. Method

BAT adapts pre-trained image and point transformers, predicts features across modalities for semantic alignment, and combines modality-specific and fused features for anomaly detection. At test time, it compares features with normal-feature memory banks and combines scores using trainable OCSVMs.

3.1. Overview

  • The overview adds trainable adapters and dual predictors to pre-trained extractors, stores RGB, point-cloud, and fused features in memory banks, and detects anomalies using nearest distances to normal features.The resulting adaptive features are F_rgb and F_pc, and the fused representation is F_f.Source: §3.1 · p. 4

3.2. Adaptive Transformers

  • AIT and the point adapter fine-tune pre-trained transformers; the point adapter combines linear transformation with nonlinear refinement while preserving the original FFN branch.PCR uses projection and pooling operations with GELU; point features from layers 3, 7, and 11 are concatenated.Source: §3.2 · pp. 4–5

3.3. Bidirectional Feature Alignment

  • BFA uses modality-specific predictors for mutual feature prediction and jointly minimizes cosine distance, with stop-gradient to prevent feature collapse and preserve modality-specific characteristics.RGB features predict point-cloud features and point-cloud features predict RGB features.Source: §3.3 · p. 5

3.4. Anomaly Detection

  • The detection stage uses RGB, point-cloud, and fused-feature memory banks, then trainable OCSVMs balance modality scores to produce image-level and pixel-level anomaly scores.The memory banks store each modality's features separately.Source: §3.4 · pp. 5–6

4. Experiments and analyses

BAT is evaluated on MVTec 3D-AD and Eyecandies, with comparisons spanning full-data, few-shot, qualitative, and component analyses. Results favor BAT’s anomaly localization and show contributions from its adapters and bidirectional alignment, while training requires paired RGB and point-cloud data.

4.1. Datasets and Evaluation Metrics

  • Evaluation uses MVTec 3D-AD and Eyecandies, reporting I-AUROC for detection and P-AUROC plus AUPRO at 0.3 and 0.01 FPR thresholds for localization.MVTec 3D-AD has 2,656 training and 1,197 test samples; Eyecandies has 10,000 training and 500 test samples.Source: §4.1 · p. 6

4.2. Anomaly Detection Results

  • BAT improves MVTec 3D-AD I-AUROC by 2.5% over M3DM and 0.7% over UCF, while leading localization there and on Eyecandies.On Eyecandies, BAT matches M3DM’s I-AUROC and outperforms competing methods in P-AUROC, AUPRO@30%, and AUPRO@1%.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • Across 5-, 10-, and 50-pair settings, BAT surpasses BTF, AST, and M3DM in anomaly detection and localization.The few-shot experiments use randomly selected image-point cloud pairs per category from MVTec 3D-AD.Source: §4.2 · p. 7, Table 5 · p. 8

4.3. Qualitative Analysis

  • On MVTec 3D-AD, BAT produces compact, accurate anomaly maps, whereas M3DM often produces broader regions and emphasizes object outlines.The comparison is qualitative and concerns localization maps.Source: §4.3 · p. 8

4.4. Ablation Study

  • AIT raises I-AUROC by 1.9% over the frozen image backbone, while convolutional adaptors and CoordConv improve image-feature adaptation.AIT increases parameters from 327MB to 343MB; the reported adaptor comparisons favor convolutional over linear layers.Source: §4.4 · p. 8, Table 6 · p. 8
  • APT improves I-AUROC by 4.6% and AUPRO by 2.7% over the point-backbone baseline with only 2MB of additional tunable parameters.Full transformer fine-tuning is reported as suboptimal and carries catastrophic-forgetting risks.Source: §4.4 · pp. 8–9, Table 7 · p. 9
  • With AIT, APT, BFA, a memory bank, and OCSVM, BAT achieves the best overall alignment-strategy performance; unaligned concatenation performs worst.Bidirectional alignment unifies complementary modalities; multimodal BAT localization also exceeds single-modality counterparts. Across tested backbones, BAT outperforms M3DM.Source: §4.4 · pp. 9–10, Table 8 · p. 9
  • BAT requires paired RGB and point-cloud data during training, a constraint the authors connect to the difficulty of collecting large-scale paired data in industry.The authors suggest zero-shot multimodal alignment using synthetic or public multimodal data as a future direction.Source: §4.4 · pp. 10–11

5. Conclusion and future work

BAT combines bidirectional feature alignment with lightweight adapters to create semantically aligned RGB and point-cloud representations for multimodal anomaly detection. Experiments on MVTec 3D-AD and Eyecandies report superior performance, while future work could extend the framework to additional modalities and image settings.

5. Conclusion and future work

  • BAT uses mutual feature prediction to align RGB and point-cloud representations, while lightweight adapters fine-tune pre-trained transformer networks.The alignment enhances point-cloud textural granularity and RGB spatial contextual awareness.Source: §5 · p. 11
  • Experiments on MVTec 3D-AD and Eyecandies report BAT surpassing existing multimodal anomaly detection methods.Source: §5 · p. 11
  • Future work could add linguistic or depth data and extend BAT to multi-illumination or multi-view RGB alongside point clouds.Source: §5 · p. 11
Loading 2610.14598v1…