PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Industrial Anomaly Detection via Hybrid Fusion

Yue Wang, Jinlong Peng, Jiangning Zhang, Ran Yi, Yabiao Wang, Chengjie Wang

arXiv:2303.00601v2cs.CV

TL;DR

Multimodal anomaly detection with RGB images and 3D point clouds remains underexplored, and direct feature concatenation can cause feature disturbance. M3DM uses point-feature alignment, contrastive unsupervised fusion, and decision-layer fusion with multiple memory banks; it outperforms SOTA methods on MVTec-3D AD detection and segmentation precision, though label bias leaves a prediction-map gap.

  • Problem

    Multimodal industrial anomaly detection remains underexplored, while direct concatenation of RGB and 3D features can cause strong disturbance and reduce performance.

  • Method

    M3DM aligns point and RGB features, fuses corresponding patches with unsupervised patch-wise contrastive learning, and combines separate memory-bank outputs through decision-layer fusion.

  • Results

    M3DM outperforms state-of-the-art methods on detection and segmentation precision on the MVTec-3D AD dataset.

  • Takeaways & Limitations

    Hybrid fusion with aligned multimodal features and multiple memory banks provides a reported approach for multimodal industrial anomaly detection.

  • Takeaways & Limitations

    Bias between 3D point clouds and 2D ground-truth labels leaves a gap between multimodal prediction maps and the ground truth.

Abstract

from arXiv · show

2D-based Industrial Anomaly Detection has been widely discussed, however, multimodal industrial anomaly detection based on 3D point clouds and RGB images still has many untouched fields. Existing multimodal industrial anomaly detection methods directly concatenate the multimodal features, which leads to a strong disturbance between features and harms the detection performance. In this paper, we propose Multi-3D-Memory (M3DM), a novel multimodal anomaly detection method with hybrid fusion scheme: firstly, we design an unsupervised feature fusion with patch-wise contrastive learning to encourage the interaction of different modal features; secondly, we use a decision layer fusion with multiple memory banks to avoid loss of information and additional novelty classifiers to make the final decision. We further propose a point feature alignment operation to better align the point cloud and RGB features. Extensive experiments show that our multimodal industrial anomaly detection model outperforms the state-of-the-art (SOTA) methods on both detection and segmentation precision on MVTec-3D AD dataset. Code is available at https://github.com/nomewang/M3DM.

1. Introduction

Industrial anomaly detection commonly trains unsupervised models on normal examples, but multimodal RGB and 3D methods face feature disturbance and alignment challenges. M3DM addresses these issues with hybrid fusion, contrastive learning, multiple memory banks, and point-feature alignment, outperforming SOTA detection and segmentation methods on MVTec-3D AD.

  • Motivation: Industrial quality inspection benefits from both 3D shape and RGB color information, while defect examples are scarce and normal examples are abundant.Most current industrial anomaly detection methods therefore use unsupervised training with normal examples only.
  • Problem: Directly concatenating high-dimensional multimodal features can create strong feature disturbance and reduce performance.Pretrained feature extractors avoid the difficult design of multimodal reconstruction targets, motivating feature-based fusion.
  • Method: M3DM uses Unsupervised Feature Fusion with patch-wise contrastive loss to encourage interaction and learn relations between corresponding multimodal patches.The method fuses RGB and point-cloud features rather than directly concatenating them.
  • Method: M3DM uses Decision Layer Fusion with separate RGB, 3D, and fused-feature memory banks for final anomaly detection and segmentation decisions.Two learnable decision modules combine the outputs of the memory banks.
  • Result: M3DM outperforms state-of-the-art methods on detection and segmentation precision on the MVTec-3D AD dataset.This is the paper's reported benchmark conclusion.

2. Related Works

Related work spans unsupervised 2D and 3D industrial anomaly detection, including reconstruction, pretrained-feature, and memory-bank approaches. MVTec-3D AD extends the setting by pairing RGB images with point-position information for the same instance.

  • Traditional Anomaly Detection: Traditional anomaly detection uses probabilistic models, representation-space neighbors, clustering, and outlier ensembles to identify abnormal data.These approaches seek deviations from representations of normal data.
  • 2D Industrial Anomaly Detection: 2D industrial anomaly detection commonly uses unsupervised image reconstruction or pretrained features, with MVTec AD containing only good training cases.Testing includes both good and bad cases.
  • Multimodal Direction: M3DM's related methodological context combines Point Feature Alignment, Unsupervised Feature Fusion, and Decision Layer Fusion for multimodal anomaly detection.These components address position mismatch, multimodal interaction, and information retention through memory banks.
  • 3D Industrial Anomaly Detection: MVTec-3D AD provides RGB information and point-position information for the same instance, establishing a public multimodal 3D anomaly-detection benchmark.Earlier 3D methods included reconstruction and student-teacher approaches but could lose spatial structure or require extra pretraining data.

3. Method

M3DM combines aligned RGB and point-cloud representations through feature-level interaction and decision-level fusion. Point Feature Alignment projects interpolated 3D features into RGB-image space, while multiple memory banks preserve modality-specific and fused information for final decisions.

  • Overall architecture: M3DM uses RGB and 3D point-cloud inputs with pretrained DINO and PointMAE feature extractors.The extracted color and point-cloud representations support subsequent alignment and fusion.
  • Point Feature Alignment: Point Feature Alignment interpolates point features back to the original cloud, projects them onto the RGB plane, and pools projected features into patches.Inverse-distance weighting uses sampled group-center features; empty projected positions are set to zero.
  • Unsupervised Feature Fusion: UFF trains multimodal patch features so corresponding RGB and point-cloud patches share more information than patches at different positions.The module uses MLPs and fully connected layers with an InfoNCE-based patch-wise contrastive loss.
  • Unsupervised Feature Fusion: During inference, UFF concatenates the modality-specific MLP outputs into fused patch features.The fused representation is formed from the processed RGB and point-cloud patch features.
  • Decision Layer Fusion: The method addresses information loss during feature fusion by retaining original modality features alongside the fused representation.The separate memory banks are built from color, position, and fusion features for inference-time decisions.
  • Decision Layer Fusion: DLF stores RGB, point-cloud, and fused features in separate memory banks, then uses two learnable OCSVMs for anomaly scores and segmentation maps.Each memory bank independently produces an anomaly score and segmentation map before decision-layer fusion.

4. Experiments

Experiments evaluate M3DM on MVTec-3D AD using transformer-based feature extraction, multimodal fusion ablations, feature-distribution analysis, few-shot settings, and dataset-property discussions. Results show strong detection and segmentation performance, while label bias remains a segmentation limitation.

  • Experimental Details: MVTec-3D AD contains 10 categories, with 2656 training samples and 1137 testing samples, using structured-light 3D scans paired with RGB information.The dataset provides aligned position tensors representing x, y and z coordinates and corresponding RGB information.
  • Experimental Details: The evaluation uses I-AUROC for image-level anomaly detection and AUPRO for segmentation, with higher I-AUROC indicating better detection.AUPRO measures average relative overlap with connected ground-truth components.
  • Main Results: 0.874 mean I-AUROC is achieved in pure 3D, while 0.945 mean I-AUROC is achieved in the 3D + RGB setting.The table reports these values across all MVTec-3D AD categories.
  • Main Results: 0.942 mean AUPRO is achieved for RGB segmentation, and 0.964 mean AUPRO is achieved for RGB + 3D segmentation.The table states that M3DM outperforms other methods in both reported settings.
  • Few-shot Evaluation: In 5-shot and 10-shot settings, M3DM retains better segmentation performance than some non-few-shot methods.The few-shot experiments randomly select 5 or 10 images from each category for training and test on the full testing dataset.
  • Dataset Discussion: The dataset’s 3D information helps detect anomalies missed by RGB, but point-cloud predictions can be misaligned with 2D ground-truth labels.This label bias lowers 3D AUPRO relative to RGB in the reported discussion and remains a target for future work.

5. Conclusion

The conclusion presents M3DM as a multimodal anomaly detector for point clouds and RGB images that combines multiple memory banks with hybrid feature fusion. It incorporates UFF, DLF, transformer backbones, and PFA, and reports SOTA performance on MVTec-3D AD.

  • Conclusion: M3DM processes point clouds and RGB images with multiple memory banks and a hybrid feature-fusion scheme.The approach is designed for multimodal industrial anomaly detection.
  • Conclusion: UFF uses patch-wise contrastive loss to promote multimodal interaction and unify feature distributions, while DLF combines multiple memory-bank outputs.These modules form the method’s hybrid fusion pipeline.
  • Conclusion: Point Feature Alignment converts 3D features to a 2D plane to align point-cloud and RGB features spatially.The alignment operates between pretrained Point Transformer and Vision Transformer feature extractors.
  • Conclusion: M3DM outperforms SOTA results on the MVTec-3D AD dataset.The conclusion reports this outcome without specifying a single metric value.

Overview

The supplementary material provides implementation details, additional metrics and ablations, backbone analysis, few-shot and cross-dataset results, and visualization results.

  • Supplementary Contents: Appendix A provides hardware and software implementation details.
  • Supplementary Contents: Appendices B–E provide P-AUROC scores, category-level ablation results, Point Transformer experiments, and few-shot results.
  • Supplementary Contents: Appendix F discusses backbone choices, while Appendix G reports results for all Eyecandies categories.
  • Supplementary Contents: Appendix H presents visualization results for all MVTec-3D AD categories.

A. Implementation Details

M3DM uses PyTorch-based feature extractors and memory banks, Scikit-Learn for OCSVM and AUROC, and runs on a single Tesla V100.

  • M3DM is implemented with PyTorch for feature extractors and memory-bank algorithms.
  • Scikit-Learn is used for OCSVM and AUROC calculation.
  • All experiments run on a single Nvidia Tesla V100 with at most 50 GB of memory in the full setting.
  • P-AUROC is reported as an additional segmentation metric, with smaller inter-method differences than AUPRO.

C. Detailed Results of Ablation Study

The ablation study examines UFF, DLF, and multiple memory banks across all MVTec-3D AD categories using detailed anomaly-detection results.

  • The study reports detailed ablation results for every MVTec-3D AD category.
  • The ablations investigate Unsupervised Feature Fusion, decision-layer fusion, and multiple memory banks.
  • Compared settings include point-cloud-only and RGB-only information.
  • A single-memory-bank baseline directly concatenates Point Transformer and RGB features.

D. Detailed Results of PFA Analysis

The PFA analysis compares point-group and projected point-feature representations, showing that smaller projected features can better preserve small-defect information.

  • PFA settings: The PFA experiments report category-level results for the settings evaluated in the main paper’s Table 5.
  • PFA settings: Point Transformer hyper-parameters include the number of groups and group size during farthest point sampling.
  • PFA settings: Original point groups are stored in a memory bank as patches, then projected onto a 2D plane for segmentation interpolation.
  • Results: Direct anomaly calculation on point groups benefits Bagel, Cable Gland, Foam, and Rope because FPS preserves more small-defect information.
  • Results: As the patch becomes smaller, the 2D-plane point feature performs better at detecting small defects.

E. Detailed Results of Few-shot Setting

Few-shot experiments evaluate M3DM with 10 or 5 training images per category and compare backbone choices, including self-supervised extractors.

  • Few-shot setting: The few-shot evaluation randomly selects 10 or 5 training images from each category and tests on the full test set.
  • Few-shot setting: In 10-shot and 5-shot settings, M3DM achieves better segmentation performance than some non-few-shot methods.
  • Few-shot setting: Some categories, including Bagel and Potato, perform better with 50 shots than with the full-dataset version.
  • Backbone analysis: Self-supervised pretrained backbones outperform supervised ones, while smaller self-supervised backbones can outperform larger supervised backbones.
  • Backbone analysis: Point-MAE performs better than Point-Bert in the evaluated point-cloud transformer comparison.

G. Eyecandies Results

The Eyecandies evaluation adapts the dataset to M3DM and reports segmentation performance, while additional ablations examine point-group settings and multimodal configurations.

  • Eyecandies dataset: Eyecandies contains 10 candy categories, with 1,000 training images and 50 public test images per category.Each sample provides six RGB images under different lighting conditions, a depth map, and a normal map.
  • Dataset preparation: M3DM uses the environment-light image as RGB input when converting Eyecandies to its supported format.
  • Segmentation ablation: The ablation section includes detailed AUPRO results for anomaly segmentation across all MVTec-3D AD categories.
  • Point Transformer setting: The point-group ablation reports best performance with 1,024 point groups per sample and 128 points per group.This configuration is stated for the explored Point Transformer setting.
  • Eyecandies evaluation: M3DM reports AUPRO segmentation performance for Eyecandies because previous work did not provide that metric.The reported score is presented for reference in further study.

H. Visualization Results

The visualization and supplementary experiments compare M3DM across segmentation maps, few-shot settings, feature backbones, and Eyecandies metrics.

  • Visualization Results: M3DM produces better multimodal anomaly segmentation maps than PatchCore + FPFH in the visualized MVTec-3D AD results.The comparison is shown through heatmap visualizations across all MVTec-3D AD categories.
  • Few-shot evaluation: M3DM retains good anomaly detection performance on few-shot MVTec-3D AD settings.
  • Few-shot evaluation: M3DM achieves better few-shot anomaly segmentation performance than most non-few-shot methods on MVTec-3D AD.
  • Backbone comparison: Self-supervised RGB feature extractors outperform supervised backbones for MVTec-3D AD anomaly detection and segmentation.This pattern is reported for both I-AUROC and AUPRO evaluations.
  • Eyecandies evaluation: Eyecandies evaluations report I-AUROC, P-AUROC, and AUPRO results across all dataset categories.The baseline results for I-AUROC and P-AUROC are taken from prior work.
Loading 2303.00601v2…