PaperlayerView exact revision on arXiv

Source-linked AI summary

Complementary Pseudo Multimodal Feature for Point Cloud Anomaly Detection

Yunkang Cao, Xiaohao Xu, Weiming Shen

arXiv:2303.13194v1cs.CVcs.AI

TL;DR

Point-cloud anomaly detection remains an emerging task whose performance is described as far from practical requirements, while handcrafted descriptors lack global semantic context. CPMF combines handcrafted 3D geometry with pretrained 2D features from rendered multiview images. The paper reports that CPMF outperforms existing point-cloud anomaly-detection methods by a large margin and that combining the modalities improves image- and pixel-level results over single feature types.

  • Problem

    Point-cloud anomaly detection remains an emerging task with performance described as far from practical requirements, and handcrafted descriptors are limited to local structure rather than global semantic context.

    Source: §I · p. 1

  • Method

    CPMF combines handcrafted 3D descriptors for local geometry with pretrained 2D-network features from rendered multiview images, aligning and aggregating the modalities.

    Source: §I · p. 2

  • Results

    CPMF is reported to outperform existing point-cloud anomaly-detection methods by a large margin, while combined features yield better image- and pixel-level performance than single feature types.

    Source: §I · p. 2, C. Ablation Studies · p. 9

  • Takeaways & Limitations

    The reported feature comparisons support combining 3D geometry and 2D semantics rather than relying on either modality alone.

    Source: C. Ablation Studies · p. 9

    The selection of suitable views remains an area for further investigation; the passage notes that fixed views may underperform adaptive views.

    Source: C. Ablation Studies · p. 9

Abstract

from arXiv · show

Point cloud (PCD) anomaly detection steadily emerges as a promising research area. This study aims to improve PCD anomaly detection performance by combining handcrafted PCD descriptions with powerful pre-trained 2D neural networks. To this end, this study proposes Complementary Pseudo Multimodal Feature (CPMF) that incorporates local geometrical information in 3D modality using handcrafted PCD descriptors and global semantic information in the generated pseudo 2D modality using pre-trained 2D neural networks. For global semantics extraction, CPMF projects the origin PCD into a pseudo 2D modality containing multi-view images. These images are delivered to pre-trained 2D neural networks for informative 2D modality feature extraction. The 3D and 2D modality features are aggregated to obtain the CPMF for PCD anomaly detection. Extensive experiments demonstrate the complementary capacity between 2D and 3D modality features and the effectiveness of CPMF, with 95.15% image-level AU-ROC and 92.93% pixel-level PRO on the MVTec3D benchmark. Code is available on https://github.com/caoyunkang/CPMF.

I. INTRODUCTION

Point-cloud anomaly detection is emerging, but its performance remains far from practical requirements, and handcrafted descriptors lack global semantic context. CPMF combines local 3D geometry with global semantics from pretrained 2D networks to improve point-cloud representations and anomaly detection.

I. INTRODUCTION

  • Handcrafted point-cloud descriptors capture local structure but miss global semantic context, which matters for detecting semantic abnormalities.The introduction motivates combining semantic context with handcrafted geometric modeling.Source: §I · p. 1
  • CPMF unifies local geometric and global semantic cues by extracting point-cloud features in direct 3D and generated pseudo-2D modalities.The paper contrasts this with prior methods –[3], which use only 3D features.Source: Figure 1 · p. 1, §I · p. 2
  • The pseudo-2D module projects and renders point clouds into multiview images, then uses pretrained 2D networks to extract semantic feature maps aligned back to points.The aligned point-wise features are aggregated across views and combined with 3D features.Source: §I · p. 2
  • CPMF demonstrates complementary 2D and 3D features and is reported to outperform existing point-cloud anomaly-detection methods by a large margin.The introduction states the performance claim without reporting metric values.Source: §I · p. 2

II. RELATED WORK

The related work reviews 2D image anomaly detection, 3D point-cloud anomaly detection, and point-cloud feature learning. It traces a shift toward pretrained or learned representations while noting that point-cloud features still face a semantic-context gap.

A. 2D Image Anomaly Detection

  • 2D image anomaly detection typically combines feature extraction, which seeks distinguishable normal and abnormal representations, with modeling of normal-feature distributions.The review describes both from-scratch approaches and methods using pretrained representations.Source: A. 2D Image Anomaly Detection · p. 2

B. 3D PCD Anomaly Detection

  • Point-cloud anomaly detection has received less study; handcrafted features can perform comparably but rarely capture semantic information because of their heuristic designs.The review describes learned methods including 3D-ST and BTF, which reports strong handcrafted-descriptor performance and limited transfer of pretrained features.Source: B. 3D PCD Anomaly Detection · p. 3

C. PCD Feature Learning

  • Most point-cloud representations are extracted directly in 3D, using handcrafted descriptors or learned point-level features and self-supervised methods.Examples include FPFH, PointNet, PointNet++ [28], and FCGF.Source: C. PCD Feature Learning · p. 3
  • Prior work also renders point clouds into multiview images for instance- or point-wise features, while CPMF unifies direct 3D and pseudo-2D modalities to represent local structure and global semantics.The review notes work on view selection and aggregation,.Source: C. PCD Feature Learning · p. 3

III. PROPOSED METHOD

The method assigns larger anomaly scores to abnormal points and uses those scores to localize anomalous regions. CPMF extracts local geometry from handcrafted 3D descriptors and global semantics from rendered multiview images, aligns and aggregates the features, and uses them for anomaly detection.

A. Problem definition

  • The task is to score each point in a test point cloud, assigning larger scores to abnormal points so anomalous regions can be localized.Training point-clouds are assumed to contain only normal data.Source: A. Problem definition · p. 3

B. Pipeline Overview

  • CPMF extracts local structures with handcrafted 3D descriptors and semantic feature maps from pretrained 2D networks applied to rendered multiview images.The point-cloud features are extracted in parallel from the two modalities.Source: B. Pipeline Overview · p. 3, Figure 2 · p. 4
  • The 2D feature maps are aligned to 3D space as point-wise features, then aggregated with 3D features before PatchCore nearest-neighbor anomaly detection.The two modalities provide complementary local and semantic information.Source: B. Pipeline Overview · pp. 3–4

C. 3D Modality Feature Extractor

CPMF uses handcrafted descriptors for local 3D geometry and a rendered pseudo-2D pathway to extract point-wise features from pre-trained networks. The 2D features are aligned to points and aggregated across views.

C. 3D Modality Feature Extractor

  • Handcrafted descriptors provide the 3D modality features for representing local PCD geometry.The 3D feature extractor outputs a feature vector for each point.Source: C. 3D Modality Feature Extractor · p. 4

D. 2D Modality Feature Extractor

E. Aggregation

The aggregation stage normalizes the 2D and 3D modality features to unit vectors before concentrating them into a joint CPMF representation.

E. Aggregation

  • The 2D and 3D feature vectors are normalized to unit length so they have the same magnitude and contribute similarly to anomaly detection.The normalized modality features are then concentrated into the combined representation.Source: E. Aggregation · pp. 5–6
  • The method aggregates multi-view 2D point-wise features and combines the 2D and 3D features to form CPMF.The algorithm describes writing the resulting CPMF features to a memory bank.Source: E. Aggregation · p. 6

F. PCD Anomaly Detection

CPMF anomaly detection follows PatchCore: it stores training-point features in a memory bank and uses nearest-neighbor search to score test points.

F. PCD Anomaly Detection

  • The training phase constructs a memory bank containing CPMF features for points in the training point clouds.The method follows PatchCore for anomaly detection.Source: F. PCD Anomaly Detection · p. 6
  • At test time, CPMF features are extracted for the test point cloud and nearest-neighbor search is used to calculate anomaly scores.The scores are collected for the testing set.Source: F. PCD Anomaly Detection · p. 6
  • The method summarizes point-level scores into an anomaly-score list for the testing set.The described process is presented as the overall CPMF anomaly-detection framework.Source: F. PCD Anomaly Detection · p. 6

IV. EXPERIMENTS

The experiments evaluate CPMF performance and examine how its components influence anomaly detection.

IV. EXPERIMENTS

  • The experiments assess CPMF performance and the influence of its components on anomaly detection.Source: §IV · p. 6

A. Experiment Settings

Experiments use MVTec 3D scans across ten categories, with preprocessing and handcrafted and pretrained-network features supporting anomaly detection and localization.

A. Experiment Settings

  • Experiments use 3D scans from the ten-category MVTec 3D dataset, including rigid and deformable objects and some natural variation.Although the dataset includes RGB images, the experiments focus on point-cloud scans.Source: A. Experiment Settings · p. 6
  • Background planes are filtered after boundary-strip plane approximation and NaN removal; point clouds are downsampled before FPFH extraction.The 3D feature dimension is 33.Source: A. Experiment Settings · p. 6
  • Multiview images are rendered at 224 × 224 and passed through the first three blocks of ImageNet-pretrained ResNet18 for 448-dimensional features.The default uses 27 views selected from small-angle rotations.Source: A. Experiment Settings · p. 7
  • Image-level detection is evaluated with AU-ROC, while anomaly segmentation is evaluated with pixel-level PRO, reported per class.The study abbreviates these as I-ROC and P-PRO, respectively.Source: A. Experiment Settings · p. 7

B. Comparisons with State-of-the-art Methods

On MVTec3D, CPMF outperforms existing methods on image-level anomaly detection and improves pixel-level localization over BTF.

B. Comparisons with State-of-the-art Methods

C. Ablation Studies

The ablations show that 2D and 3D features have different strengths, while their combination improves anomaly detection across feature types and backbones. More rendering views generally improve performance, although some views can produce lower-quality features.

Influence of the number of rendering views

  • I-ROC and P-PRO improve moderately with more views, with the largest rise from one to three views and a slight drop around 12–18 views.Compared with one view, 27 views yield about 10% higher I-ROC and 4% higher P-PRO for 2D-only ResNet34, and about 5% and 2% higher, respectively, for the combined features.Source: C. Ablation Studies · p. 8
  • Some views make abnormalities more visually apparent than others, and low-quality features from specific views may damage detection performance.The authors identify adaptive view selection as a direction for further investigation.Source: C. Ablation Studies · pp. 8–9

Influence of 2D and 3D modality features

  • Using only 3D features achieves average I-ROC of 82.04% and average P-PRO of 92.30%; 2D-only results improve significantly as the number of views increases.The 3D modality captures geometrical information, while 2D features exploit more semantics.Source: C. Ablation Studies · p. 9
  • Combining 2D and 3D features outperforms either alone in I-ROC; with ResNet18, the combination improves I-ROC by about 6%.For P-PRO, 3D features outperform 2D features in nearly all tested situations, while their combination exceeds a single feature type by about 0.6% with ResNet18.Source: C. Ablation Studies · p. 9
  • Different feature types localize anomalies in different categories, while their combination localizes anomalies across all categories.The feature combination also has a more distinguishable distribution than either single feature type.Source: C. Ablation Studies · p. 9

Influence of backbones

  • With ResNet18, CPMF achieves 95.15% I-ROC and 92.93% P-PRO, and the feature combination improves both image- and pixel-level performance across tested backbones.Backbone type does not greatly affect performance; ResNet18 gives the best overall result.Source: C. Ablation Studies · p. 9, Table III · p. 9

Limitation analysis

  • Noise during point-cloud acquisition can produce poor-quality rendered images that may affect feature capability.The paper also notes that P-PRO covers anomaly types that may require RGB information, limiting the interpretation of CPMF’s localization performance.Source: C. Ablation Studies · pp. 9–10

V. CONCLUSION

CPMF combines handcrafted 3D descriptors for local geometry with pre-trained 2D features from rendered multiview images for global semantics. Experiments show that CPMF outperforms previous methods and that its 2D and 3D features are complementary; end-to-end development and adaptive view selection and aggregation remain future directions.

V. CONCLUSION

  • CPMF combines handcrafted 3D descriptors with pre-trained 2D features from rendered multiview images to represent local geometry and global semantics.The 2D feature maps are aligned and aggregated before being combined with the 3D features.Source: §V · p. 10
  • Experiments show that CPMF outperforms previous methods by a large margin and that its 2D and 3D features are complementary.Source: §V · p. 10
  • End-to-end development is proposed to improve efficiency and reduce memory consumption, while adaptive view selection and aggregation are expected to improve detection performance further.Source: §V · p. 10
Loading 2303.13194v1…