Towards Open-Vocabulary Industrial Defect Understanding with a Large-Scale Multimodal Dataset

Motivation

• Automated optical inspection remains limited by false alarms, poor adaptation to novel defects, and weak generalization across manufacturing contexts.Manual inspection also involves high labor costs, subjective judgment, and limited throughput.Source: §1 · p. 2
• Natural-image VLMs lack industrial visual-semantic knowledge for subtle, localized defects and specialized terms such as delamination and solder void.Specialized detectors also lack unified multitask capabilities, require extensive pixel-level annotations, and can be black-box discriminators.Source: §1 · p. 2

Key contributions

• IMDD-1M spans 421 defect types across 63 manufacturing domains, with expert-verified image-text annotations at roughly two orders of magnitude beyond existing benchmarks.The domains include automotive, electronics, metals, textiles, and packaging.Source: §1 · p. 2
• A hybrid annotation pipeline combines expert verification with LLM-assisted captions for technical accuracy and linguistic consistency across industrial terminology.Source: §1 · p. 2
• The diffusion-based foundation model combines defect generation, segmentation, detection, and semantic grounding in one framework.It is trained on IMDD-1M and integrates discriminative and generative capabilities.Source: §1 · p. 2
• The paper provides evaluation protocols and benchmarks for defect detection, segmentation, and synthesis using standardized metrics for practical industrial assessment.Source: §1 · p. 2

Industrial defect datasets

• Industrial defect datasets progressed from limited synthetic or single-domain data to more realistic and diverse collections, but remain limited in scale and multimodal annotations.The progression includes DAGM and KolektorSDD, MVTec AD, then BTAD, VisA, and Real-IAD, alongside domain-specific datasets.Source: §2 · p. 2

Vision-language datasets

• Natural-image datasets such as ImageNet, COCO, and LAION advanced VLMs, but industrial defects require specialized terminology and pixel-level precision for subtle, localized features.IMDD-1M is presented as the first large-scale industrial image-text dataset for multimodal defect analysis.Source: §2 · p. 2

3.1. Data Collection

• Public benchmarks, multilingual web mining, and industrial partnerships supply data from varied sources and manufacturing sectors.Partner imagery comes from petrochemical, metal processing, and powder metallurgy; samples are anonymized, stratified, and screened for low resolution or ambiguity over an 18-month collection period.Source: §3.1 · pp. 2–3

3.2. Annotation Framework

• Each image is paired with expert-written descriptions of product identity, defect morphology, and manufacturing causes.A controlled vocabulary of over 500 terms and structured fields capture material, defect type, location, root cause, orientation, scale, and pattern; the authors report greater multimodal alignment precision than crowdsourcing.Source: §3.2 · p. 3

3.3. Dataset Statistics

• Figure 3 presents the log-scaled sample distribution among the top 100 defect categories and a three-step dataset-construction workflow.Source: Figure 3 · p. 3
• Figure 4 depicts the normal-versus-anomaly distribution and the dataset’s composition across domains.Source: Figure 4 · p. 3
• The corpus contains over 1.24 million high-resolution image-text pairs across 63 product categories and 421 defect types, including 285,451 normal and 954,928 anomaly samples.Images are standardized at 512×512 pixels; descriptions average 42 words, and industrial samples add variability such as uneven illumination and complex backgrounds.Source: §3.3 · p. 4
• Figure 5 compares normalized anomaly and normal proportions by dataset, illustrating substantial imbalance across industrial domains.Source: Figure 5 · p. 4, §3.3 · p. 4
• Figure 6 ranks the top 10 datasets by object-class and defect-type counts, with MVTec AD and VisA standing out for broad coverage.Source: Figure 6 · p. 4

4. Method

• The framework trains a text-conditioned diffusion model on IMDD-1M and transfers its features to downstream tasks, using an implicit captioner and Mask2Former generator.The U-Net has 860M parameters, the implicit captioner 0.3M, and the Mask2Former generator 45M.Source: §4 · p. 4, §4.2 · p. 4

4.1. Problem Formulation

• At test time, images from disjoint novel categories are classified using only category names, with the model predicting a semantic mask for each image.Source: §4.1 · p. 4

4.2. Architecture

• An implicit captioner generates pseudo text embeddings from visual features when captions are unavailable, and a mask generator uses diffusion features to predict masks and embeddings.Source: §4.2 · p. 4, Figure 7 · p. 5

4.3. Industrial Diffusion Model

• The industrial diffusion model uses a randomly initialized Stable Diffusion v1.5 U-Net with cross-attention text conditioning and a frozen VAE encoder.The U-Net has four encoder blocks and a mirrored decoder with skip connections.Source: §4.3 · p. 5

4.4. Implicit Captioner

• The implicit captioner maps frozen CLIP image embeddings from 512 to 768 dimensions and is jointly trained with the diffusion U-Net using stochastic caption conditioning.Training selects ground-truth captions or implicit embeddings with equal probability and adds an auxiliary cosine-similarity loss.Source: §4.4 · pp. 5–6

4.5. Feature Extraction

• For feature extraction, the frozen model adds noise at timestep 50, encodes the image and implicit caption, then extracts multi-scale features in one forward pass.Feature resolutions are h, h/2, h/4, and h/8.Source: §4.5 · p. 6

4.6. Mask Generation and Classification

• A Mask2Former pixel and transformer decoder generates 100 mask queries, which are trained with binary cross-entropy and classified using text embeddings or caption-derived pseudo-labels.Grounding similarity and bidirectional contrastive loss support classification when captions supply nouns as pseudo-labels.Source: §4.6 · p. 6

4.7. Training Protocol

• Stage 1 trains the diffusion model and implicit captioner from scratch, while Stage 2 freezes diffusion and trains the mask generator on downstream data.At inference, masks and embeddings are generated and classified against test-category names.Source: §4.7 · p. 6

5.1. Implementation Details

• The model has 890M parameters and takes 0.35 seconds per image on an A100 GPU; generative quality is evaluated with FID and IS.Source: §5.1 · p. 6

5.2. Text-Guided Defect Generation

• 100.29 IS and 5.5–13.6 FID are reported; compared with SDXL, the model has higher IS and lower FID, with material-specific texture reproduction in generated samples.Captioning evaluation is deferred to future work.Source: §5.2 · p. 6, Figure 8 · p. 7, Figure 9 · p. 7

5.3. Unified Framework for Downstream Tasks

• The evaluation covers classification, object detection, and segmentation as downstream tasks for transferred representations.Source: §5.3 · p. 6

5.3.1. Defect Classification

• 96.7% average accuracy is achieved across four datasets without task-specific modifications.Source: §5.3.1 · p. 7, Table 3 · p. 7

5.3.2. Object-Level Defect Detection

• 74.6% mAP@0.5 and 58.9% mAP@0.75 are achieved with 200 samples per class, approaching YOLOv8’s 78.3% and 62.1%, respectively.Bounding boxes are derived from segmentation masks, so explicit box annotations are not required.Source: §5.3.2 · p. 7, Table 4 · p. 7

5.3.3. Pixel-Level Defect Segmentation

• 52.9% average IoU is reported, while the 200-sample-per-class setting reaches 96.1% P-AUC-ROC and 90.2% AUC-PRO, about 2% below full-data methods.Figure 10 shows localized segmentation masks and defect bounding boxes across material domains.Source: §5.3.3 · p. 7, Table 5 · p. 7, Figure 10 · p. 8

5.4. Ablation Study and Data Efficiency

• Removing diffusion conditioning drops accuracy by 7.0%, removing implicit text embedding drops accuracy by 4.8%, and removing grounding loss degrades IoU by 3.1%.Source: §5.4 · p. 7, Table 7 · p. 8
• 96.1% accuracy is reached with 200 samples per class; conventional methods use approximately 4,000 samples per class for comparable performance, and gains plateau beyond 200.The 200-sample setting uses less than 5% of the training data required by conventional approaches.Source: §5.4 · p. 8, Figure 11 · p. 8

• The work introduces IMDD-1M with aligned image-text annotations and a unified diffusion framework that reports competitive zero-shot performance using less than 5% of supervised training data.Source: §6 · p. 8
• The framework is presented as a foundation for scalable, language-driven industrial inspection systems.Source: §6 · p. 9
• Planned extensions include temporal and multi-view data for video tracking and 3D reasoning, cross-sector generalization, and integration of multimodal reasoning with physical simulation.Source: §6 · p. 9

7. Reproducibility and Code Release

• The release includes code, lightweight pretrained model snapshots, and documentation intended to support reproduction of the experiments.Source: §7 · p. 10

7.1. Repository Structure

• The repository includes implementations for the Industrial Diffusion U-Net, Implicit Captioner, and Mask2Former generator, plus task scripts and setup documentation.Source: §7.1 · p. 10

7.2. Model and Dataset Release

• Full pretrained models and datasets are not in the repository; qualified researchers may request complete resources under institutional data-sharing policies.Source: §7.2 · p. 10

8.1. Training Limitations.

• Stage 1 pre-training takes 72 hours on 8 H100 GPUs (576 GPU-hours), and the two-stage training process adds complexity and multiplies tuning costs.Peak memory reaches 76GB per GPU; researchers must manage frozen and trainable parameters across stages.Source: §8.1 · p. 10

8.2. Inference Limitations.

• At 0.35 seconds per image on an A100, inference may be too slow for scenarios requiring 50–200 frames per second.Inference also uses 18.7GB of memory, exceeding the capacity of common industrial edge devices.Source: §8.2 · p. 10

8.3. Application Limitations.

• Effective fine-tuning requires at least 200 samples per class, which may be impractical for extremely rare defects.The framework is limited to 2D analysis and visible-light RGB imaging, without temporal or 3D reasoning or non-RGB modalities.Source: §8.3 · p. 10

9.1. Positive Impacts.

• The framework may improve quality control, reduce annotation needs, and help small and medium-sized enterprises adopt AI inspection.The authors also describe potential knowledge transfer across manufacturing domains and reduced exposure to hazardous work.Source: §9.1 · pp. 10–11

9.2. Potential Concerns.

• Deployment may affect inspection jobs and encourage overreliance on predictions; the authors call for retraining and human-in-the-loop workflows in safety-critical settings.They also flag intellectual-property concerns around proprietary imagery and possible misuse of computer vision.Source: §9.2 · p. 11

10.1.1. Forward Diffusion Process; 10.1.2. Reverse Denoising Process

• Forward diffusion progressively adds noise, with the reparameterization enabling training without iterating through timesteps; reverse denoising predicts added noise and samples from xT drawn from a standard Gaussian.Source: §10.1.1 · p. 11, §10.1.2 · p. 11

10.1.3. Conditional Generation; 10.2. Latent Diffusion Models

• Text conditioning enters noise prediction through CLIP embeddings and cross-attention, while classifier-free guidance strengthens conditioning and latent diffusion is reported to accelerate training 16–32× overall.Stable Diffusion uses a VAE latent space with downsampling factor 8 and four latent channels; the reported attention-layer speedup is 64×.Source: §10.1.3 · p. 11, §10.2 · p. 11

10.3. U-Net Architecture; 11.3. Training Details

• The model follows a randomly initialized 860M-parameter Stable Diffusion v1.5 U-Net, with four resolution stages and text cross-attention; training proceeds through pre-training and mask-generator fine-tuning.Stage 1 requires 72 hours on 8 H100 80GB GPUs; Stage 2 fine-tuning is reported as 4 hours on MVTec AD and 5.5 hours on VisA.Source: §10.3 · p. 12, §11.3 · p. 15

11.1. Dataset Details; 12.3. Dataset Statistics

• Collection integrated 20 public benchmarks, multilingual web mining, and industrial partnerships with 12 companies; expert annotators used peer review and consensus, with κ = 0.87 classification and κ = 0.81 segmentation agreement (IoU¿0.75).Source: §11.1 · p. 12

11.2. Compared Model Settings; 11.1. Dataset Details; 12.4. Real and Generated Visual Comparison

• The reported comparisons configure detection and segmentation baselines with different architectures, training regimes, and data quantities, including full-MVTec training for the listed anomaly-detection baselines.Qualitative samples cover aluminum and other industrial materials, while the generated examples pair real images on the left with corresponding generations on the right.Source: §11.2 · pp. 12, 15, §11.1 · p. 12, Figure 12 · p. 13

12.1. Extended Quantitative Analysis

• Zero-shot transfer IoU is 52.9%–54.7%, an 11%–15% gain over single-dataset baselines; on MVTec AD, average Acc is 91.99% and mean IoU is 55.71%.The transfer comparison is summarized in Table 12; per-category results use 200 samples per class in Table 11.Source: §12.1 · p. 15, Table 12 · p. 16, Table 11 · p. 16

12.2. Ablation Studies

• Random initialization achieves 82.7% mIoU versus 74.5% for fine-tuned Stable Diffusion, a reported 8.2% advantage; IMDD-1M needs 150 samples for 95% accuracy, and t = 50 balances semantic context with spatial precision.The sample-efficiency comparison reports 1,850 samples for random initialization and 520 for ImageNet pre-training; at t = 50, accuracy is 91.0% and IoU is 52.9%.Source: §12.2 · pp. 15–16, Table 13 · p. 16