PaperlayerView exact revision on arXiv

Source-linked AI summary

IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing

Guoyang Xie, Jinbao Wang, Jiaqi Liu, Jiayi Lyu, Yong Liu, Chengjie Wang, Feng Zheng, Yaochu Jin

arXiv:2301.13359v5cs.CVcs.AI

TL;DR

Industrial image anomaly detection lacks a uniform benchmark that reflects manufacturing needs, including varied supervision, learning paradigms, and efficiency. IM-IAD evaluates methods across five settings and reports results that bear on algorithm selection and future research.

  • Problem

    Existing IAD benchmarks and research do not adequately cover industrial manufacturing needs, including varied learning settings and accuracy, memory, and inference-speed considerations.

    Source: §I · p. 1

  • Method

    IM-IAD provides a uniform benchmark spanning 19 algorithms, seven datasets, and five settings, with standardized implementation and evaluation details.

    Source: §I · p. 2, §III · p. 3, §V · p. 11

  • Results

    Experiments report no statistically significant unsupervised winner across accuracy, memory usage, and inference speed, while fully supervised methods outperform unsupervised methods at pixel level.

    Source: §I · p. 2, C. Abnormal Data for Fully Supervised… · p. 7

  • Takeaways & Limitations

    The benchmark points to selecting unsupervised algorithms with anomaly types in mind and highlights partial labels, noise, continual learning, and global features as research considerations.

    Source: A. Overall Comparisons · p. 6, §V · p. 11

Abstract

from arXiv · show

Image anomaly detection (IAD) is an emerging and vital computer vision task in industrial manufacturing (IM). Recently, many advanced algorithms have been reported, but their performance deviates considerably with various IM settings. We realize that the lack of a uniform IM benchmark is hindering the development and usage of IAD methods in real-world applications. In addition, it is difficult for researchers to analyze IAD algorithms without a uniform benchmark. To solve this problem, we propose a uniform IM benchmark, for the first time, to assess how well these algorithms perform, which includes various levels of supervision (unsupervised versus fully supervised), learning paradigms (few-shot, continual and noisy label), and efficiency (memory usage and inference speed). Then, we construct a comprehensive image anomaly detection benchmark (IM-IAD), which includes 19 algorithms on seven major datasets with a uniform setting. Extensive experiments (17,017 total) on IM-IAD provide in-depth insights into IAD algorithm redesign or selection. Moreover, the proposed IM-IAD benchmark challenges existing algorithms and suggests future research directions. To foster reproducibility and accessibility, the source code of IM-IAD is uploaded on the website, https://github.com/M-3LAB/IM-IAD.

I. INTRODUCTION

Industrial image anomaly detection is important in manufacturing, but existing research and benchmarks do not adequately represent industrial requirements. IM-IAD benchmarks methods across manufacturing-relevant settings and examines their results to inform algorithm selection and redesign.

Motivation

  • Existing benchmarks largely omit industrial needs such as few-shot, continual, and noisy-label learning, as well as accuracy–efficiency trade-offs.The paper notes that these gaps matter for factory use, where memory, inference speed, limited normal training samples, forgetting, and annotation errors are practical concerns.Source: §I · pp. 1–2

Findings

  • Experiments report no statistically significant unsupervised winner across accuracy, memory usage, and inference speed, while supervised methods outperform unsupervised methods at pixel level.Other reported findings include few-shot feature-embedding performance reaching 95% of vanilla IAD with four augmented rotated samples, and improved noise resilience from importance reweighting above a 10% noise ratio.Source: §I · p. 2

Contribution

  • The benchmark evaluates IAD methods across manufacturing-derived problems to bridge academic research and industrial practice.The paper presents a standardized, uniform benchmark and a plug-and-play, modular implementation for fair evaluation.Source: §I · p. 2
  • The authors frame IM-IAD as a way to expose the gap between academic research and industrial practice and to guide future improvements.Their contribution includes examining 16 methods on seven datasets in 17,017 instances.Source: §I · p. 2

Benchmark focus

  • IM-IAD examines four aspects: few-shot representational ability, accuracy–efficiency trade-offs, catastrophic forgetting, and robustness to noisy labels.The authors state that these aspects are drawn from research and manufacturing-process requirements.Source: §I · p. 2

II. RELATED WORK

The related work reviews IAD methods and learning settings, describing their approaches and remaining challenges. The paper then introduces IM-IAD’s five settings and benchmark implementation.

II. RELATED WORK

  • Unsupervised IAD includes feature-embedding and reconstruction methods; teacher–student and memory-bank approaches compare test features with learned normal features, while reconstruction methods compare inputs with reconstructions.The review notes that teacher–student and memory-bank methods depend on teacher capacity or bank size, while reconstruction methods require substantial training time and lag feature-embedding methods in performance.Source: §II · p. 2
  • Fully supervised IAD trains with abnormal images, often using few anomalies to distinguish abnormal from normal features, but some approaches underperform unsupervised methods.The review identifies effective use of abnormal training data as an area needing improvement.Source: §II · p. 2
  • Few-shot and zero-shot methods reduce training-data demands, but the review describes the area as early-stage and notes that some visual-language models are too large for production lines.It identifies few-shot learning as a continuing focus for industrial anomaly detection.Source: §II · pp. 2–3
  • Noisy IAD addresses mislabeled or shifted training data; SoftPatch denoises at patch level, while InReaCh learns internal image relationships and performs near state of the art on artificially corrupted data.The review says noisy-data IAD needs further research.Source: §II · p. 3
  • Continual IAD work includes a benchmark that ignores large domain gaps between datasets and a method focused within classes that neglects forgetting between classes.The review calls for more work in this setting.Source: §II · p. 3

III. IM-IAD

  • IM-IAD defines five settings and summarizes baseline implementations, datasets, metrics, and hyperparameters within a uniform benchmark framework.Table II presents representative algorithms and marks the reimplemented methods in purple.Source: §III · p. 3, Table II · p. 4

A. Problem Definition

IAD predicts whether a target-category image is anomalous and localizes an anomaly region when one is detected. IM-IAD defines five training and evaluation settings that vary the available samples and task sequence.

Unsupervised IAD

  • In unsupervised IAD, training uses only m normal samples per category, while testing includes normal and abnormal samples.Source: A. Problem Definition · p. 4

Fully supervised IAD

  • In fully supervised IAD, training uses m normal samples and n abnormal samples, with n much smaller than m and set to 10 in this benchmark.Source: A. Problem Definition · p. 4

Few-shot IAD

  • Few-shot IAD trains on m normal samples, where m is at most 8 and takes values 1, 2, 4, or 8 for the target category.Source: A. Problem Definition · p. 4

Noisy IAD

  • Noisy IAD includes abnormal samples in training but labels them as normal; the abnormal-sample count is described as up to 20% of the combined training samples.Source: A. Problem Definition · p. 4

Continual IAD

  • Continual IAD trains sequentially on category-specific normal-sample datasets and evaluates the updated model on previously seen categories.Source: A. Problem Definition · p. 4

B. Implementation Datails

IM-IAD standardizes algorithm coverage, datasets, metrics, and experimental settings across five manufacturing-relevant IAD regimes.

1) Baseline Methods:

  • IM-IAD benchmarks 16 representative algorithms spanning unsupervised, fully supervised, few-shot, noise-resilient, and continual-learning capabilities.The selection criteria prioritize methods representative of different supervision levels and industrial adaptation needs.Source: 1) Baseline Methods · p. 4

2) Datasets:

  • Seven datasets span structural, logical, and multi-instance anomalies, with pixel-level annotations available across them.The collection includes MVTec AD, MVTec LOCO-AD, MPDD, BTAD, VisA, MTD, and DAGM; DAGM is synthetic.Source: 2) Datasets · p. 4

3) Evaluation Metrics:

  • IM-IAD uses AU-ROC/AUC, AUPR/AP, and PRO for structural anomalies, sPRO for logical anomalies, and FM for catastrophic forgetting.The metrics evaluate anomaly localization, logical-defect detection, and resistance to forgetting.Source: 3) Evaluation Metrics · p. 4

C. Uniform Perspective

  • The five settings differ mainly in training-sample type and number; few-shot IAD uses at most eight normal training samples.Unsupervised training uses normal samples, while fully supervised and noisy settings include limited abnormal samples.Source: C. Uniform Perspective · pp. 4–5

4) Hyperparameters:

  • Table V records the experimental settings used in IM-IAD.The supplied table caption identifies it as a summary of experimental settings.Source: Table V · p. 5

D. Open Challenges

The open challenges concern how IAD handles limited anomaly supervision, scarce few-shot data, contaminated labels, and continual learning.

D. Open Challenges

  • Fully supervised IAD must use limited abnormal data alongside many normal samples to detect anomalies effectively.The challenge is how to use the guidance from the limited abnormal examples.Source: D. Open Challenges · p. 5
  • Few-shot IAD must detect anomalies with fewer than eight training samples per category, often without image- or pixel-level annotations.The described setting may use a small number of normal or abnormal images, but its training categories contain only normal samples.Source: D. Open Challenges · p. 5
  • Noisy IAD tests resilience when 5%–20% of training samples are anomalous but mislabeled as normal.Contaminated samples can confuse the algorithms’ decision threshold.Source: D. Open Challenges · p. 5
  • Continual IAD must limit catastrophic forgetting as models learn new categories in recycling assembly lines with many workpieces.Maintaining numerous separate models on one line is described as unfeasible because of maintenance costs.Source: D. Open Challenges · p. 5

IV. RESULTS AND DISCUSSIONS

The results show trade-offs among unsupervised methods and anomaly types, while global-feature approaches show promise for logical anomaly detection.

A. Overall Comparisons

  • No unsupervised method dominates across datasets, accuracy, inference speed, and memory; PatchCore’s strong image AUC comes without an efficiency advantage.The comparisons cover seven datasets, and the figure encodes performance, per-image inference time, and GPU memory use.Source: A. Overall Comparisons · p. 6, Figure 2 · p. 6, Table VI · p. 7
  • On logical screw-bag defects, PatchCore’s heatmap misses the extra pushpin while RD4AD identifies it.MVTec LOCO-AD includes logical anomalies such as displacement or missing parts, unlike MVTec AD’s structural anomalies.Source: A. Overall Comparisons · p. 6
  • Image-level metrics assess whole-product abnormality, while pixel-level metrics assess localization; methods can perform differently across the two.The degree of a localized defect is associated with product price.Source: A. Overall Comparisons · p. 6
  • Knowing anomaly types in advance is presented as useful supervision for selecting and designing unsupervised algorithms.The text notes that product-line specialists can provide information about anomaly types.Source: A. Overall Comparisons · p. 6

B. Role of Global Features in Logical IAD

C. Abnormal Data for Fully Supervised IAD

The fully supervised comparison uses ten abnormal training samples and reports stronger pixel-level performance than unsupervised methods, alongside the cost of anomaly labeling.

C. Abnormal Data for Fully Supervised IAD

  • The fully supervised benchmark compares six methods using ten abnormal training samples.The compared methods are PRN, BGAD, DevNet, DRA, SemiREST, and PatchCore.Source: C. Abnormal Data for Fully Supervised… · p. 7
  • Fully supervised methods outperform unsupervised methods such as PatchCore by a large margin at the pixel level.The paper attributes this result to incorporating abnormal samples during training.Source: C. Abnormal Data for Fully Supervised… · p. 7, Table VIII · p. 8
  • Figure 3 presents heatmaps from PatchCore, RD4AD, and DRAEM alongside normal training images and abnormal test examples.Its left columns show structural anomalies and its right columns logical anomalies.Source: Figure 3 · p. 8
  • Pixel-labeling costs motivate reducing the number of abnormal training samples while maintaining IAD accuracy.The cited average is three hours per worker to complete pixel labels for each image.Source: C. Abnormal Data for Fully Supervised… · pp. 7–8

D. Rotation Augmentation for Feature-Embedding based FewShot IAD

Few-shot results favor rotation augmentation, while noisy-label experiments find feature-embedding methods more effective than IGD at limited noise levels. The section also highlights abnormal-sample synthesis as a challenge.

D. Rotation Augmentation for Feature-Embedding based FewShot IAD

  • Most augmentation methods improve few-shot IAD, and rotation is optimal because industrial images such as metal nuts and screws can be transformed by rotation.The comparison uses vanilla methods at 1, 2, 4, and 8 shots and includes RegAD.Source: D. Rotation Augmentation for… · p. 8, Table IX · p. 9
  • Collecting large numbers of abnormal samples is difficult on fault-free production lines, motivating further work on abnormal-sample synthesis.CutPaste and DRAEM are cited as examples of synthesis methods.Source: D. Rotation Augmentation for… · p. 8

E. Importance Re-weighting for Noisy IAD

  • At noise ratios ≤0.15, feature-embedding methods are more effective than IGD.The noisy-label setting uses noise ratios from 5% to 20% in 5% steps, with training samples labeled normal.Source: E. Importance Re-weighting for Noisy IAD · p. 8

Algorithm 1: PatchCore Pipeline

The PatchCore pipeline builds a memory bank from normal patch features and scores test patches against that bank. Re-weighting improves resilience when noise exceeds 10%.

Algorithm 1: PatchCore Pipeline

  • PatchCore constructs a memory bank of neighborhood-aware features from normal samples, using a patch feature extractor, target memory size, and random linear projection.WideResNet-50 is the default feature extractor, and coreset sampling balances bank size with IAD performance.Source: Algorithm 1: PatchCore Pipeline · p. 9
  • Re-weighting improves PatchCore's resilience when the noise ratio is larger than 10%.The ablation varies noise ratios and neighborhood numbers and reports image AUROC.Source: Algorithm 1: PatchCore Pipeline · p. 10, Table XII · p. 10
  • Sample selection may improve robustness and integrate with current IAD methods; one cited approach separates true-label identification from anomaly detection.The cited example is Song et al..Source: Algorithm 1: PatchCore Pipeline · p. 10

F. Memory Bank-based Methods for Continual IAD

Memory bank-based methods perform satisfactorily against catastrophic forgetting in continual IAD, although deployment must balance storage, inference speed, and task interference.

F. Memory Bank-based Methods for Continual IAD

  • The benchmark compares vanilla methods with the continual baseline DNE under the paper's continual setting.Source: F. Memory Bank-based Methods for… · p. 10
  • Memory bank-based methods provide satisfactory performance against catastrophic forgetting, although they were not explicitly designed for continual IAD.Replay fits memory-based methods because new-task memories can be added, and nearest-memory matching can support test-time decisions.Source: F. Memory Bank-based Methods for… · p. 10
  • Adding memory for new tasks can increase storage use and slow inference, while overlap with prior-task memory may reduce performance.The section identifies limiting memory size and interference from previous tasks as central challenges.Source: F. Memory Bank-based Methods for… · p. 10

G. Uniform View on IM-IAD

In the uniform benchmark, PatchCore achieves the SOTA result but faces memory and inference-time disadvantages. The paper concludes that IM-IAD exposes design insights across multiple industrial IAD settings.

G. Uniform View on IM-IAD

  • PatchCore achieves the SOTA result in IM-IAD, but has inferior GPU memory usage and inference time relative to other algorithms.Maintaining its memory bank becomes a critical issue when the number of object classes exceeds 100.Source: G. Uniform View on IM-IAD · p. 10, Table XIII · p. 11
  • Practical IAD must balance fast inference, low memory use, and continual learning, while handling sequentially streamed test samples.The paper points to multi-objective neural architecture search as a promising direction, noting edge-device memory constraints.Source: G. Uniform View on IM-IAD · pp. 10–11

V. CONCLUSION

  • IM-IAD comprises 19 algorithms, 7 datasets, and 5 settings, with reported insights on partial labels, noise ratios, continual-learning design, and global features for logical anomalies.The conclusion also identifies future research directions for industrial IAD.Source: §V · p. 11
Loading 2301.13359v5…