Source-linked AI summary
The MVTec 3D-AD Dataset for Unsupervised 3D Anomaly Detection and Localization
Paul Bergmann, Xin Jin, David Sattlegger, Carsten Steger
TL;DR
Unsupervised 3D anomaly detection lacks comprehensive real-world datasets, despite its relevance to detecting manufacturing defects from anomaly-free training data. The paper introduces MVTec 3D-AD, a ten-category dataset with precise annotations and an initial benchmark showing substantial room for improvement.
Problem
Unsupervised 3D anomaly detection lacks suitable comprehensive datasets, although manufacturing inspection requires detecting defects when training data are anomaly-free.
Method
The paper constructs a real-world dataset of high-resolution 3D scans with anomaly-free training and validation sets, anomalous tests, precise annotations, and pixel-level evaluation scores.
Results
Existing unsupervised 3D anomaly-localization methods perform insufficiently on the dataset, leaving considerable room for improvement.
Takeaways & Limitations
MVTec 3D-AD provides a benchmark for developing and evaluating methods for unsupervised 3D anomaly detection and localization.
Takeaways & Limitations
The evaluated methods are not yet accurate enough for real-world industrial applications.
Abstract
from arXiv · showhide
We introduce the first comprehensive 3D dataset for the task of unsupervised anomaly detection and localization. It is inspired by real-world visual inspection scenarios in which a model has to detect various types of defects on manufactured products, even if it is trained only on anomaly-free data. There are defects that manifest themselves as anomalies in the geometric structure of an object. These cause significant deviations in a 3D representation of the data. We employed a high-resolution industrial 3D sensor to acquire depth scans of 10 different object categories. For all object categories, we present a training and validation set, each of which solely consists of scans of anomaly-free samples. The corresponding test sets contain samples showing various defects such as scratches, dents, holes, contaminations, or deformations. Precise ground-truth annotations are provided for every anomalous test sample. An initial benchmark of 3D anomaly detection methods on our dataset indicates a considerable room for improvement.
1 INTRODUCTION
The paper addresses the lack of suitable datasets for unsupervised 3D anomaly detection by introducing a real-world industrial dataset with anomaly-free training data and annotated anomalous tests. Its initial benchmark shows that existing methods leave considerable room for improvement.
- Unsupervised 3D anomaly detection is comparatively unexplored, partly because suitable datasets are unavailable.
- The dataset targets industrial inspection, where unknown defect types and costly defective-data collection make anomaly-free training useful.
- Training and validation sets contain only anomaly-free samples, while test samples contain varied anomalies with precise ground-truth annotations.
- Existing unsupervised 3D anomaly-localization methods perform poorly on the dataset, leaving considerable room for improvement.
2 RELATED WORK
Prior anomaly-detection datasets and methods largely focus on 2D images or medical voxel scans. The paper positions MVTec 3D-AD as a comprehensive dataset explicitly designed for unsupervised 3D anomaly detection and localization.
- Existing 2D industrial datasets evaluate defect detection and localization using anomaly-free training images and anomalous test images.
- Methods developed for 2D color images are ill-suited to 3D data because pretrained networks expect RGB images as input.
- Before MVTec 3D-AD, no comprehensive 3D dataset was designed explicitly for unsupervised anomaly detection and localization.
- Existing 3D anomaly-detection methods were evaluated mainly on BRATS and ATLAS, medical MRI benchmarks originally introduced for supervised disease or lesion detection.
- Only a few 3D methods existed, including voxel-based extensions of f-AnoGAN and autoencoder approaches that derive anomaly scores per voxel.
3 DESCRIPTION OF THE DATASET
MVTec 3D-AD provides high-resolution, real-world 3D scans across 10 object categories, with anomaly-free training and validation data, anomalous tests, and precise annotations. Its acquisition and evaluation design supports detecting defects in both valid and invalid 3D sensor pixels while accounting for highly variable anomaly sizes.
- Dataset composition: The dataset contains scans from 10 object categories, with anomaly-free training and validation sets and test samples containing real-world defects.Defects include scratches, dents, and contaminations, and were devised and fabricated for industrial inspection scenarios.
- Dataset composition: The categories span naturally variable, deformable, and rigid objects, including bagel, carrot, cookie, peach, potato, foam, rope, tire, cable gland, and dowel.The dataset includes 41 prototypical anomaly types, such as cracks, deformations, holes, contaminations, and cut-off parts.
- Data acquisition and preprocessing: Scans were acquired with a high-resolution structured-light sensor as 1920×1200 three-channel images encoding x, y, and z coordinates, with complementary RGB values.The coordinate image has a one-to-one mapping to the corresponding point cloud, and objects were viewed from a fixed category-specific angle.
- Data acquisition and preprocessing: The data are cropped to fixed rectangular domains around objects, preserving a margin for augmentation and approximating inspection setups with defined object positions and task-suited illumination.The objects were recorded against dark backgrounds.
- Ground-truth annotations: Each anomalous test sample includes precise annotations mapped from 3D point clouds to 2D image regions, including invalid sensor pixels and relevant RGB-visible pixels.This captures defects that cause missing points or 3D reconstruction failures.
- Performance evaluation: Localization requires a real-valued anomaly score for every (x,y,z) pixel and evaluates thresholded predictions with PRO, whose curve is integrated over false-positive rates.The metric is useful when anomaly sizes vary substantially, but integration should not exceed 0.3 because anomalous regions are small relative to the images.
4 INITIAL BENCHMARK
The benchmark evaluates existing unsupervised 3D anomaly-localization methods across voxel and depth representations, with and without color. Voxel methods generally outperform depth methods, but false positives and reconstruction errors limit practical performance.
- Benchmark setup: The benchmark evaluates voxel-based, depth-based, and variation-model methods on the dataset, including comparisons with and without RGB information.All methods are assessed for anomaly localization, using representations derived from 3D data or depth images.
- Benchmark setup: Voxel grids use 64×64×64 resolution, and voxel-based anomaly scores are computed by comparing each input voxel with its reconstruction.The voxel representation is converted into pixel-level anomaly maps for evaluation.
- Results: At an integration limit of 0.3, Voxel f-AnoGAN performs best on average and on most categories without color, while Voxel AE performs worst among voxel methods.Voxel AE’s weaker performance is attributed to blurry and inaccurate reconstructions.
- Results: Each voxel-based method outperforms its depth-based counterpart on average, while Depth Variation Model is the strongest depth-based method.Depth AE and Depth f-AnoGAN produce many false positives around invalid input pixels.
- Results: Adding RGB improves every method except the Variation Models, although voxel-based methods still outperform depth-based methods and Voxel f-AnoGAN remains best overall.Depth AE and Depth f-AnoGAN benefit most because RGB images do not contain invalid pixels.
- Evaluation considerations: AU-PRO increases monotonically with the integration limit, but high limits can make results meaningless because defects occupy far fewer pixels than normal regions.The benchmark therefore reports relative performance at an integration limit of 0.3 rather than using large false-positive rates.
5 CONCLUSION
The paper concludes by presenting a comprehensive 3D anomaly-detection dataset motivated by industrial visual inspection. Its initial benchmark shows substantial room for improving existing methods and their applicability to industrial use.
- Dataset: The dataset contains over 4000 point clouds from ten object categories acquired with a high-resolution structured-light 3D sensor.The data was designed around real-world visual inspection tasks.
- Dataset: About 1000 samples contain varied anomalies, and every anomalous sample has precise ground-truth annotations.The dataset supports both anomaly detection and localization.
- Benchmark conclusion: The initial benchmark of the few existing methods shows significant room for improvement.The authors expect the dataset to encourage development of better methods.
- Benchmark conclusion: The evaluated methods are insufficient for use in real-world industrial applications.This conclusion is stated for the initial benchmarked methods.
Details on Training Parameters.
The benchmark uses specified architectures and training settings for deep learning methods, with latent dimensions selected through ablation for compression-based models.
- Training parameters: Deep learning-based methods are trained using documented architectures, optimizers, batch sizes, learning rates, and epoch counts.The appendix provides the detailed training parameters and model architectures.
- Voxel methods: Voxel f-AnoGAN and Voxel Autoencoder are trained for 50 epochs with batch size 2 using Adam optimization.Their initial learning rates are 0.0002 and 0.0001, respectively.
- Depth methods: Depth f-AnoGAN uses an encoder with ten convolution blocks that maps 256×256 inputs to a d-dimensional latent vector.The convolution blocks use instance normalization and LeakyReLU except for the last block.
- Model selection: Latent dimensions for compression-based methods are selected using an ablation study over mean performance across object categories.The best-performing latent dimension from the ablation is used in the main experiments.
Results for Anomaly Classification.
The paper also evaluates sample-level anomaly classification by reducing each anomaly map to a single maximum score and measuring area under the ROC curve.
- Classification evaluation: Sample-level classification uses the maximum anomaly score from each anomaly map as the sample’s anomaly score.Samples are classified as anomalous or anomaly-free.
- Classification evaluation: Performance is measured using the area under the ROC curve, with results listed in Table 5.These classification results are separate from the localization results in Table 2.
Quality of Reconstructions.
Figure 7 compares reconstruction quality across voxel- and depth-based methods, revealing distinct failures around object surfaces, missing parts, and noisy regions.
- Reconstruction quality is important for anomaly detection with autoencoder- and GAN-based methods.The evaluated methods are illustrated through reconstruction examples.
- Voxel AE produces blurry reconstructions around object surfaces.
- Voxel f-AnoGAN avoids surface blurring but sometimes fails to reconstruct parts of the input.
- Depth f-AnoGAN and Depth AE have difficulty reconstructing noisy areas with many invalid pixels.Depth inputs and reconstructions are visualized as depth images.