VISION Datasets: A Benchmark for Vision-based InduStrial InspectiON

1 Introduction

• VISION comprises 14 industrial inspection datasets spanning manufacturing processes, materials, and industries, addressing challenges often missing from existing benchmarks.The paper describes the collection as a benchmarking opportunity for vision-based industrial inspection.Source: §1 · p. 1
• Annotation masks across training, validation, and testing splits support unsupervised, weakly supervised, semi-supervised, and supervised defect detection.The paper contrasts this flexibility with prior datasets focused solely on self-supervised settings [7].Source: §1 · pp. 1–2
• Instance-segmentation labels distinguish defects of the same type within an image, supporting training and evaluation of models that identify individual defects.The paper contrasts these labels with semantic-segmentation masks that do not make this distinction.Source: §1 · p. 2
• The collection contains 18k images and 44 defect types across manufacturing processes, materials, and industries.The paper presents this breadth as representative of real-world production scenarios.Source: §1 · p. 2
• Two challenge competitions accompany the datasets and bring together researchers and practitioners from academia and industry.The paper presents the combination as a catalyst for developing solutions to the challenges represented by the datasets.Source: §1 · p. 2

2.1 Related Datasets and Benchmarks

• Unlike datasets focused on a single product, VISION provides masks across training, validation, and testing splits for four defect-detection supervision settings.The paper notes that BeanTech and MVTec AD feature multiple products but are mainly designed for unsupervised algorithms and have limited test masks.Source: §2.1 · p. 3
• Existing datasets use semantic masks that do not distinguish same-type defects in one image, limiting supervision and evaluation for identifying individual defects.VISION's instance-segmentation labels support training and evaluation on individual defects and their specifications.Source: §2.1 · p. 3

3.1 Overview

• VISION contains 18k images across 14 Roboflow-sourced datasets; V1 provides 10k defect-segmentation annotations on 4k images across 44 defect types, with the remaining 14k images unlabeled.The unlabeled images are intended to represent data-rich, annotation-rare production settings and encourage self-supervised and semi-supervised methods; full V2 annotations are planned for a subsequent challenge.Source: §3.1 · pp. 3–4

3.2 Curation Process and Rationale

• The team screened over 1,800 Roboflow manufacturing datasets and selected the top 14 by consensus on relevance, annotation quality, and realistic production challenges.Original defect decisions were retained, while refined segmentation masks were provided for defects indicated by original bounding boxes.Source: §3.2 · p. 4

3.3 Annotating Defect Instance Segmentation Labels

• Experienced annotators practiced on dataset subsets, flagged ambiguous defect boundaries for consensus review, and contributed 3,200 working hours to annotation.The team used practice rounds to reach satisfactory annotation quality and inter-rater reliability before full annotation.Source: §3.3 · p. 4

3.4 Building Dataset Splits

• To reduce leakage, the split process compares defect regions, groups images of the same product unit, and applies connected-component analysis to the similarity graph.The process first removes only byte-identical duplicates; images without defect annotations are set aside. It keeps at most 200 components per defect type and assigns components to train, validation, and test with capacities of 30, 30, and 140, respectively. Discarded images accompany the test data as inference data, but evaluation uses only the test split.Source: §3.4 · pp. 4–5

4 VISION Challenges

• The two tracks take model-centric and data-centric approaches to industrial inspection, and both had participation from companies and universities.At the time of the paper, Track 1 had over 30 registered teams; Track 2 featured participation from 11 companies and 10 universities.Source: §4 · p. 5

4.1 Track 1: Data-Efficient Defect Detection

• Track 1 evaluates data efficiency and adaptation across diverse products, surface tasks, and defect types.The competition uses limited training data to test data efficiency.Source: §4.1 · p. 5
• Track 1 scores the equally weighted average of 0.5 ∗mAP and 0.5 ∗ mARmax=100 across all 14 datasets.The metric combines precision and recall-oriented measures.Source: §4.1 · pp. 5–6

4.2 Track 2: Data-Generation for Defect Detection

• Track 2 targets data limitations, including the imbalance between normal and defective samples and rare long-tailed defects with too few training examples.The paper notes that rare defects can be costly to ignore because they may lead to product failures or safety issues.Source: §4.2 · p. 6
• Track 2 fixes the model and initial dataset, inviting participants to clean and augment the data or generate synthetic samples.Examples of augmentation include flipping, rotating, zooming, and cropping; synthetic samples may come from generative models or rendering engines.Source: §4.2 · p. 6
• Track 2 evaluates altered datasets rather than detector changes, focusing the competition on data enhancement for defect detection.Participants submit only the altered datasets for evaluation.Source: §4.2 · p. 6

5.1 Track 1

• The first-place Track 1 team combined EVA [15] and MaskDino with Dice loss and reported better performance from joint training on all 14 datasets than separate training.The second-place team used a multi-stage HTC [17] and CBNetV2 [19] solution, refining outputs with Mask2Former and other models to improve recall; the third-place team used 28 models trained on combinations of all 14 datasets.Source: §5.1 · p. 7

General Insights

• Top-performing teams commonly used ensembles, augmentation, and strong backbones; two of the top three Track 1 teams trained on all 14 datasets simultaneously.The paper reports that training on all datasets improved performance over training on each separately for the first-place team.Source: General Insights · p. 7, §5.1 · p. 7
• High-resolution inputs, cut-paste augmentation, and Test Time Augmentation were among the reported Track 1 strategies, though augmentation effectiveness depended on the dataset.The Swin Transformer was a popular backbone, and EVA and MaskDino showed strong transfer performance on the datasets.Source: General Insights · p. 7

5.2 Track 2

• The first-place Track 2 solution combined StyleGANv2 [22] and ControlNet [23] generated defects, with an ablation reporting higher downstream detection mAP when both generated sources were combined.Traditional augmentation, including rotation and reflective padding, was also applied during generation.Source: §5.2 · p. 7
• The second-place Track 2 solution used data augmentation, data generation, and Bayesian optimization to select samples using validation feedback.The method selected from original, augmented, and generated datasets.Source: §5.2 · p. 7
• The third-place Track 2 ablation reported increasing mAP after adding StyleGAN2-ADA [24] and further improvement with DM-CP, while generative methods remained difficult to assess against traditional augmentation.The paper notes that small training sets can be a problem for training generative models and that their additional benefits still need serious discussion.Source: §5.2 · p. 7, General Insights · p. 8

Algorithms for working under Data and Annotation Limitations

• Transfer learning and self-supervised, semi-supervised, few-shot, and weakly supervised methods are proposed for applications with rich data but limited labels.Competition participants’ observation of positive transfer across datasets is described as a first step toward generalizable representations.Source: Algorithms for working under Data and… · p. 8

Data Generation Techniques that can counteract real-world data challenges

• Data generation directions include VAE, GAN, and diffusion models, 3D-rendering synthetic pipelines, and inverse rendering from real defect images.Source: Data Generation Techniques that can… · p. 8

Data-centric Tools and Methodologies for more suitable data collection procedures, higher data quality, and efficiency

• Data-centric directions include active learning for selecting examples to label, semi-supervised label-quality assessment, automatic labeling tools, and dataset and annotation assessment.Source: Data-centric Tools and Methodologies… · p. 8

• VISION cannot exhaustively reproduce real-world lighting, camera angles, scale, or inspection-image quality, limiting applicability in scenarios that significantly deviate from captured conditions.Source: §7 · p. 8
• The 44 defect types do not exhaust industrial possibilities, so models trained solely on VISION may not generalize well to uncommon or unrepresented defects.Source: §7 · p. 8
• Instance-segmentation defect boundaries can be subjective, potentially causing annotation inconsistencies.Source: §7 · p. 8
• Future work could broaden real-world conditions and defect coverage, reduce annotation inconsistencies, and investigate transfer learning or domain adaptation for new defects and inspection scenarios.Source: §7 · pp. 8–9

• VISION aims to address the academic–industrial gap with a benchmark for inspection, precise defect localization and segmentation, and competitions evaluating algorithms across diverse datasets.Source: §8 · p. 9

• Table 3 reports instance-segmentation label statistics for 14 datasets spanning different domains and defect types.Source: Table 3 · p. 13

A.2 License

• Provided polygon annotations are licensed under CC BY-NC 4.0.Source: §A.2 · p. 13
• Original dataset assets remain under their original dataset licenses.Source: §A.2 · p. 13

A.3 Disclaimer

• The authors make no representation about original dataset license terms and tell users to follow those terms.Source: §A.3 · p. 13