Source-linked AI summary
Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection
Chengjie Wang, Wenbing Zhu, Bin-Bin Gao, Zhenye Gan, Jianning Zhang, Zhihao Gu, Shuguang Qian, Mingang Chen, Lizhuang Ma
TL;DR
Industrial anomaly detection research is constrained by datasets whose limited scale, diversity, realism, and near-saturated benchmark performance make methods difficult to distinguish. The paper introduces Real-IAD, a large multi-view dataset with a fully unsupervised setting and reports a challenging benchmark for popular methods. Its key benchmark result is a drop from 97.9% I-AUROC on MVTec to 85% on Real-IAD.
Problem
Existing IAD datasets have limited scale, object diversity, defect coverage, and practical realism, while methods exceed 99% AUROC on MVTec, making comparisons difficult.
Method
The paper constructs Real-IAD with high-resolution multi-view images and introduces a fully unsupervised setting by adding 0∼40% anomaly images to anomaly-free training data.
Results
97.9% I-AUROC on MVTec decreases to 85% on single-view and multi-view Real-IAD, indicating a more challenging benchmark.
Takeaways & Limitations
Real-IAD provides a highly challenging benchmark for evaluating IAD methods across broader object, defect, and viewing conditions.
Abstract
from arXiv · showhide
Industrial anomaly detection (IAD) has garnered significant attention and experienced rapid development. However, the recent development of IAD approach has encountered certain difficulties due to dataset limitations. On the one hand, most of the state-of-the-art methods have achieved saturation (over 99% in AUROC) on mainstream datasets such as MVTec, and the differences of methods cannot be well distinguished, leading to a significant gap between public datasets and actual application scenarios. On the other hand, the research on various new practical anomaly detection settings is limited by the scale of the dataset, posing a risk of overfitting in evaluation results. Therefore, we propose a large-scale, Real-world, and multi-view Industrial Anomaly Detection dataset, named Real-IAD, which contains 150K high-resolution images of 30 different objects, an order of magnitude larger than existing datasets. It has a larger range of defect area and ratio proportions, making it more challenging than previous datasets. To make the dataset closer to real application scenarios, we adopted a multi-view shooting method and proposed sample-level evaluation metrics. In addition, beyond the general unsupervised anomaly detection setting, we propose a new setting for Fully Unsupervised Industrial Anomaly Detection (FUIAD) based on the observation that the yield rate in industrial production is usually greater than 60%, which has more practical application value. Finally, we report the results of popular IAD methods on the Real-IAD dataset, providing a highly challenging benchmark to promote the development of the IAD field.
1. Introduction
Existing industrial anomaly detection datasets remain limited in scale, diversity, realism, and defect difficulty, constraining reliable evaluation. Real-IAD addresses these gaps with a large, multi-view benchmark and a fully unsupervised setting closer to industrial practice.
- Current anomaly detection datasets have limited data volume, restricting further development of industrial anomaly detection.
- MVTec AD and VisA expose shortcomings including small defect ranges, simple scenarios, few test defects, limited object diversity, and near-saturated performance.Recent methods exceed 99% on MVTec image- and pixel-level AUROC, making methods difficult to distinguish.
- Real-IAD provides a larger, real-world dataset intended to address existing benchmark limitations and support practical industrial anomaly detection research.
- Real-IAD introduces fully unsupervised anomaly detection by adding 0∼40% quality-inspection anomaly images to anomaly-free training data, based on production yield rates above 60%.
- The dataset uses multi-angle image acquisition to address defects that a single view may not cover.
- The benchmark contains 30 object classes, five shooting angles per class, and 150K high-resolution images with broader defect-area and defect-ratio ranges.
2. Related Work
Industrial anomaly detection datasets have expanded from small, single-category collections toward broader and more challenging settings, but existing visual benchmarks remain limited in scale and coverage. Real-IAD extends this line with approximately 150K high-definition, multi-view images across 30 categories.
- Early datasets such as KolektorSDD, MTD, MPDD, and BTAD were limited in category count or total image scale.
- MVTec AD established conventional visual industrial anomaly detection as a mainstream research task, while VisA expanded object coverage and scenario complexity.
- Recent work explores synthetic pose-agnostic data, 3D information, few-shot learning, noisy training data, video detection, and semi-supervised anomaly detection.
- Existing visual IAD datasets remain small in scale and category and cover limited industrial scenarios, leaving no ImageNet-1K-equivalent benchmark.
- Real-IAD addresses these gaps with approximately 150K high-definition images across 30 categories and five shooting angles, alongside elaborate labels and more challenging defects.
3. Real-IAD Dataset Description
Real-IAD is a large, high-resolution, multi-view dataset designed to represent diverse and challenging industrial anomaly detection scenarios. It covers 30 object classes, five viewpoints per object, varied materials and defects, and broad defect-scale distributions.
- Dataset Scale and Diversity: The collection covers metal, plastic, wood, ceramics, and mixed materials, with defects including pits, deformation, abrasion, scratches, damage, missing parts, foreign objects, and contamination.Material preparation involved manually creating multiple defect types on 30 collected objects.
- Multi-View Acquisition: Five synchronized viewpoints use one top-down camera and four cameras arranged uniformly at 45 degrees to capture each object.The multi-view design addresses defects that may be invisible from a single viewpoint.
- Challenge and Statistics: Defect proportions range from 0.01% to 6.75%, while defect ratios range from 1:1 to 1:10, increasing dataset difficulty.The dataset statistics show larger defect-area and defect-ratio ranges than contrastive datasets.
- Dataset Scale and Diversity: 30 object classes and 150K high-resolution images make Real-IAD substantially larger than mainstream industrial anomaly detection datasets.Each object has five images captured from different angles, with resolutions up to 2,000∼5,000.
- Dataset Advantages: Real-IAD provides richer categories, over 150K images, multi-view images, and pixel-level annotations to support robust training and fairer evaluation.The authors characterize these properties as advantages in diversity, scale, and challenge.
4. Benchmark
The benchmark evaluates Real-IAD under conventional Unsupervised IAD and Fully Unsupervised IAD settings. It reports image-, pixel-, and sample-level performance, with multi-view aggregation aligning evaluation more closely with industrial production lines.
- Evaluation Settings: Real-IAD establishes Unsupervised IAD and Fully Unsupervised IAD evaluation protocols across all 30 categories.Unsupervised IAD trains only on normal samples, whereas Fully Unsupervised IAD introduces noise into training.
- Evaluation Settings: Fully Unsupervised IAD models the practical setting in which anomalous samples are unavailable or mixed into training data.The setup keeps training-set size fixed while adaptively changing normal and abnormal sample counts according to a noise ratio.
- Evaluation Metrics: The benchmark uses AUROC for image-level and pixel-level detection and AUPRO as an additional pixel-level metric.AUPRO is computed from normalized Per-Region Overlap between segmentation and ground truth.
- Evaluation Metrics: Sample-level evaluation aggregates multiple views because each Real-IAD sample contains several images from different viewpoints.This extends evaluation beyond the image and pixel levels commonly used by existing IAD methods.
5. Comparisons with IAD Benchmarks
The benchmark comparisons show that Real-IAD is substantially more challenging than MVTec AD and supports more reliable evaluation of fully unsupervised IAD settings. Its design preserves evaluation scale across noisy training conditions, unlike adapting existing datasets by moving test anomalies into training.
- Evaluation setup: The comparison includes embedded-based, data-augmentation-based, and reconstruction-based IAD methods across existing and Real-IAD benchmarks.
- Unsupervised IAD: 97.9% in I-AUROC on MVTec falls to 85% on single-view and multi-view Real-IAD, indicating a substantially more challenging benchmark.Most methods achieve about 90% I-AUROC on Real-IAD, compared with about 98%-99% on MVTec.
- Unsupervised IAD: Real-IAD challenges pixel-level localization, with P-PRO comparable to VisA but noticeably lower than MVTec AD.
- FUIAD benchmark design: Existing benchmarks lose valid testing categories and normal training samples when anomaly noise is injected for FUIAD evaluation.Randomly moving anomaly samples from testing into training can make challenging high-noise settings impossible to evaluate.
- FUIAD benchmark design: Real-IAD maintains consistent valid testing categories and stable normal-training scale for noisy ratios from 0.1 to 0.4 and 50 to 150 anomalous testing samples.This supports a more reliable and comprehensive assessment of FUIAD algorithms.
6. Comparisons with Fully Unsupervised IAD
Under standard unsupervised evaluation, methods perform similarly for sample- and image-level classification, whereas fully unsupervised training causes severe degradation. PatchCore is relatively robust, and SoftPatch achieves near-optimal performance but with limited gains over PatchCore.
- Unsupervised IAD: Most state-of-the-art methods show almost similar sample- and image-level anomaly-classification performance when the noisy ratio is zero.PatchCore, SoftPatch, SimpleNet, and DeSTSeg are among the methods showing similar performance.
- Fully unsupervised IAD: Fully unsupervised training causes severe performance degradation on all metrics for almost all methods, especially PaDim, SimpleNet, DeSTSeg, and RD.
- Fully unsupervised IAD: Unsupervised PatchCore remains robust because its patch-level memory bank stores normal and abnormal features and can detect anomalies with a different inference distribution.The low proportion of anomalous pixels can preserve the quality of normal patch features in the memory bank.
- Fully unsupervised IAD: SoftPatch achieves almost optimal performance across methods by filtering noisy features before constructing its memory bank, but improves only modestly over unsupervised PatchCore.
7. Conclusion
The paper concludes that existing IAD datasets and clean-data assumptions limit progress toward practical inspection. Real-IAD addresses these gaps with a large, real-world, multi-view benchmark and unsupervised and fully unsupervised evaluation settings.
- Conclusion: Real-IAD contains 150k high-resolution images covering 30 metal, plastic, wood, ceramic, and mixed-material objects across five views and eight common defects.It provides pixel-, image-, and sample-level annotations.
- Conclusion: The paper identifies performance saturation, a gap between clean training assumptions and noisy production data, and insufficient dataset scale for fully unsupervised IAD.
- Conclusion: Real-IAD establishes unsupervised and fully unsupervised settings to better represent practical industrial anomaly detection scenarios.