Source-linked AI summary
Back to the Feature: Classical 3D Features are (Almost) All You Need for 3D Anomaly Detection
Eliahu Horwitz, Yedid Hoshen
TL;DR
The paper investigates whether 3D information is underused in anomaly detection and segmentation. It evaluates shape representations, identifies rotation invariance as important, and proposes BTF, which combines handcrafted 3D and color features. BTF achieves state-of-the-art performance on MVTec 3D-AD, while image-level detection remains imperfect.
Problem
The study addresses limited use of 3D information in anomaly detection and segmentation and asks whether 3D information provides benefits beyond color-only methods.
Method
BTF combines handcrafted 3D representations with a deep color-based method, after evaluating hand-crafted and learned shape representations.
Results
BTF sets a new state-of-the-art on MVTec 3D-AD, achieving 99.3% Pixel-wise ROCAUC, 96.4% PRO, and 87.3% Image ROCAUC.
Takeaways & Limitations
Rotation-invariant 3D representations are highly effective, and combining shape with color outperforms using either modality alone in the reported study.
Takeaways & Limitations
BTF's image-level detection accuracy remains imperfect, reaching an I-ROC of 86.5%.
Abstract
from arXiv · showhide
Despite significant advances in image anomaly detection and segmentation, few methods use 3D information. We utilize a recently introduced 3D anomaly detection dataset to evaluate whether or not using 3D information is a lost opportunity. First, we present a surprising finding: standard color-only methods outperform all current methods that are explicitly designed to exploit 3D information. This is counter-intuitive as even a simple inspection of the dataset shows that color-only methods are insufficient for images containing geometric anomalies. This motivates the question: how can anomaly detection methods effectively use 3D information? We investigate a range of shape representations including hand-crafted and deep-learning-based; we demonstrate that rotation invariance plays the leading role in the performance. We uncover a simple 3D-only method that beats all recent approaches while not using deep learning, external pre-training datasets, or color information. As the 3D-only method cannot detect color and texture anomalies, we combine it with color-based features, significantly outperforming previous state-of-the-art. Our method, dubbed BTF (Back to the Feature) achieves pixel-wise ROCAUC: 99.3% and PRO: 96.4% on MVTec 3D-AD.
1 Introduction
The paper asks whether 3D information improves anomaly detection and segmentation, then identifies effective representations and combines complementary shape and color cues.
- Research questions: The study evaluates whether 3D methods outperform state-of-the-art 2D methods, whether 3D information helps, and which representations and modalities are complementary.
- Motivation: Color-only methods outperform current 3D anomaly detection and segmentation methods, despite missing several geometric anomalies that 3D views reveal.The study frames this mismatch as evidence that 3D information can be useful but is not being used effectively by existing methods.
- Representations: Rotation-invariant representations are identified as key, and a handcrafted 3D point-cloud descriptor outperforms current methods, including learning-based representations.
- Final method: 99.3% Pixel-wise ROCAUC, 96.4% PRO, and 87.3% Image ROCAUC are achieved by BTF, which combines 3D and color features.
2 Related Work
The related work spans classical and deep anomaly detection, prior 3D approaches, and the dataset context motivating point-cloud anomaly segmentation.
- Anomaly detection: Classical anomaly detection includes kNN, KDE, GMM, PCA, one-class SVM, and isolation forests, while deep methods extend ideas such as PCA and OCSVM.
- 3D information: The paper notes that 3D information can distinguish cases that color alone cannot, such as a hole versus chocolate.
- 3D anomaly detection: 3D anomaly detection has received less attention than 2D methods, with prior work adapting anomaly detection to voxel data and introducing point-cloud datasets.
3 Problem Definition
The paper defines sample- and pixel-level anomaly scoring, represents local regions across RGB and 3D modalities, and evaluates methods with image- and pixel-level metrics.
- Problem definition: Anomaly detection scores samples, whereas anomaly segmentation scores individual pixels or corresponding 3D elements as anomalous or normal.
- Scoring pipeline: Methods extract local-region representations and estimate normality, often using nearest-neighbor distances to training representations.
- 3D representations: RGB lacks explicit 3D information, while depth maps, organized and unorganized point clouds, and voxels encode 3D structure differently.
- Dataset: MVTec 3D-AD contains over 4000 high-resolution scans across 10 categories, pairing organized point clouds with corresponding RGB images.
- Evaluation metrics: PRO measures average relative overlap with ground-truth components, while I-ROC and P-ROC evaluate image- and pixel-level anomaly performance.
- Evaluation setup: Figure 4 compares recent 3D methods with PatchCore and illustrates a nuisance artifact in fabric addressed through 3D-aware preprocessing.
4 An Empirical Investigation of 3D AD&S
The study finds that color-only methods can outperform existing 3D approaches, yet 3D information is necessary for some geometric and background anomalies. Rotation-invariant handcrafted 3D features, especially FPFH, and their fusion with color in BTF produce the strongest results.
- 4.1 Do current 3D methods beat 2D methods?: Color-only state-of-the-art methods outperform existing methods using 3D or 3D plus color information on MVTec 3D-AD.The comparison evaluates PatchCore against Voxel GAN variants and 3D-ST.
- 4.2 Is 3D information potentially useful for AD&S?: 3D information reveals geometric anomalies hidden in color and helps distinguish cluttered backgrounds from anomalous objects.The study highlights ambiguous geometry and background variation as two cases where 3D is useful.
- 4.3 What are the key properties of successful 3D AD&S representations?: D-SIFT surpasses previous depth-based results on all three metrics, while HoG improves pixel-level accuracy over raw and learning-based features.These results show that classical descriptors can be competitive despite not being designed specifically for 3D.
- 4.3 What are the key properties of successful 3D AD&S representations?: Rotation-invariant handcrafted representations are highly effective: FPFH outperforms methods using color, depth, or both.The authors attribute part of the advantage to retaining dense local point information for fine-grained anomalies.
- 4.4 Are there complimentary benefits from using both 3D and color modalities?: BTF concatenates ImageNet-based color features with FPFH and improves over the previous best combined method by 32.5% PRO and 33.6% I-ROC.It also achieves 99.3% P-ROC, improving 1.3% over FPFH.
- 4.6 Limitations.: BTF’s limitations include fused features underperforming color-only features for cable gland and foam, and image-level I-ROC remaining 86.5%.The paper also notes sampling noise, missing information, and noisy backgrounds in 3D sensing and MVTec 3D-AD.
5 Conclusion
The study finds that rotation-invariant 3D representations perform best for 3D anomaly detection and introduces BTF, which combines 3D and color features as a new state-of-the-art method.
- BTF combines 3D and color features to set a new state-of-the-art on MVTec 3D-AD.The method combines handcrafted 3D representations with color-based features.
- Rotation-invariant representations achieve the best performance among the investigated approaches for 3D anomaly detection.
- The study was motivated by color-only approaches outperforming existing 3D methods on MVTec 3D-AD.
A Detailed I-ROC Results
Table 3 provides detailed I-ROCAUC results, separating current state-of-the-art methods from methods investigated by the authors. It reports that many author-investigated methods outperform current methods by a wide margin.
- Table 3 reports detailed I-ROCAUC results for current state-of-the-art methods and methods investigated by the authors.
- The table is divided into a top half containing current state-of-the-art methods and a bottom half containing methods investigated by the authors.
- Many of the authors’ methods outperform all current methods by a wide margin.
B Detailed P-ROC Results
Table 4 provides detailed P-ROC results and notes that previous methods did not report results. It also identifies ImageNet-pre-trained methods with the label “iNet.”
- Table 4 reports detailed P-ROC results.
- Results are not reported for previous methods in the table.
- The label “iNet” indicates ImageNet-pre-trained methods.
C Detailed PointNext Results
Tables 5–7 contain the full breakdown of PointNext results on S3DIS, while Tables 5 and 6 specifically report PRO and I-ROC results. The PointNext results use PointNext-XL, with separate models trained for each of S3DIS’s six areas.
- Tables 5–7 contain the full breakdown of PointNext results on the S3DIS dataset.
- Table 5 reports detailed PointNext PRO results.
- Table 6 reports detailed PointNext I-ROC results, and all results use the PointNext-XL model with separate models for S3DIS’s six areas.
D Additional Method Combinations
Additional results evaluate combinations of depth and RGB using other methods. These results are reported in supplementary tables rather than the main paper.
- Supplementary Tables 8, 9, and 10 report results for combining depth and RGB from other methods.
- The additional combinations use both depth and RGB information.
- These combination results are not shown in the main paper.
E Method Specific Implementation Details
The supplementary implementation details specify feature extraction and input preprocessing for the evaluated anomaly-detection methods. They also provide detailed results for PointNeXt and additional method combinations.
- Supplementary tables report detailed PointNeXt P-ROC results and additional combinations for PRO, I-ROC, and P-ROC.The cited tables are labeled Tables 7–10.
- PatchCore extracts RGB features with an ImageNet-pretrained WideResNet50 and depth features from ImageNet-normalized depth maps.RGB patch features come from aggregated blocks 2 and 3, while depth images are processed separately.
- The evaluated methods use varied depth representations, including flattened patches, histogram features, Dense SIFT, and FPFH descriptors.The implementations use patch-level or point-level representations with method-specific dimensionalities.
- NSA is used instead of CutPaste because the official CutPaste implementation was unavailable and unofficial versions lagged behind reported figures.The NSA implementation was modified to handle depth images.
- PointNeXt features use an S3DIS-pretrained PointNeXt-XL model with RGB+XYZ inputs and densely sampled point clouds.Samples are downsampled to 224×224 and reshaped into unorganized point clouds.
F Preprocessing Implementation Details
Preprocessing removes nuisance geometry from organized point clouds using boundary-based plane estimation and connected-component filtering. The supplementary material also reports numeric preprocessing results.
- The preprocessing assumes centered objects and treats image-boundary edges as lying on a common plane.A 10-pixel-wide strip around the image boundary is used to estimate that plane.
- Table 11 reports average numeric preprocessing metrics across all classes.The table corresponds to the preprocessing results shown in Figure 7-top of the main paper.
- RANSAC estimates the boundary plane, and points within a 0.005 distance threshold are removed by zeroing their XYZ and RGB values.Zeroing preserves the original image resolution.
- DB-Scan removes residual artifacts by retaining the largest connected component and discarding other clusters.This addresses planes that are not planar and therefore escape the RANSAC removal step.