PaperlayerView exact revision on arXiv

Source-linked AI summary

AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models

Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, Jinqiao Wang

arXiv:2308.15366v4cs.CV

TL;DR

Industrial anomaly detection requires localization from normal-only training data, while LVLMs lack domain-specific and fine-grained anomaly knowledge and existing methods require manual thresholds. AnomalyGPT combines simulated anomaly visual-textual data, an image decoder, and prompt tuning to address these gaps. With one normal shot, it reports 86.1% accuracy, 94.1% image-level AUC, and 95.3% pixel-level AUC on MVTec-AD.

  • Problem

    IAD is hindered by LVLMs’ limited domain-specific and local-detail understanding and by existing methods’ reliance on anomaly scores with manually set thresholds.

  • Method

    AnomalyGPT fine-tunes an LVLM with synthesized anomalous visual-textual data, an image decoder, and prompt embeddings.

  • Results

    86.1% accuracy, 94.1% image-level AUC, and 95.3% pixel-level AUC are achieved on MVTec-AD with one-shot transfer.

  • Takeaways & Limitations

    AnomalyGPT directly detects and localizes anomalies without manual thresholds and supports multi-turn dialogue and few-shot in-context learning.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) such as MiniGPT-4 and LLaVA have demonstrated the capability of understanding images and achieved remarkable performance in various visual tasks. Despite their strong abilities in recognizing common objects due to extensive training datasets, they lack specific domain knowledge and have a weaker understanding of localized details within objects, which hinders their effectiveness in the Industrial Anomaly Detection (IAD) task. On the other hand, most existing IAD methods only provide anomaly scores and necessitate the manual setting of thresholds to distinguish between normal and abnormal samples, which restricts their practical implementation. In this paper, we explore the utilization of LVLM to address the IAD problem and propose AnomalyGPT, a novel IAD approach based on LVLM. We generate training data by simulating anomalous images and producing corresponding textual descriptions for each image. We also employ an image decoder to provide fine-grained semantic and design a prompt learner to fine-tune the LVLM using prompt embeddings. Our AnomalyGPT eliminates the need for manual threshold adjustments, thus directly assesses the presence and locations of anomalies. Additionally, AnomalyGPT supports multi-turn dialogues and exhibits impressive few-shot in-context learning capabilities. With only one normal shot, AnomalyGPT achieves the state-of-the-art performance with an accuracy of 86.1%, an image-level AUC of 94.1%, and a pixel-level AUC of 95.3% on the MVTec-AD dataset. Code is available at https://github.com/CASIA-IVA-Lab/AnomalyGPT.

1. Introduction

AnomalyGPT applies an LVLM to industrial anomaly detection, addressing limited domain knowledge and local-detail sensitivity alongside existing methods’ reliance on manual thresholds. It uses synthesized anomalous visual-textual data and prompt-based fine-tuning to detect and localize anomalies while supporting interaction and few-shot transfer.

  • Problem: LVLMs have limited domain-specific knowledge and weak sensitivity to local object details, restricting their effectiveness on industrial anomaly detection.
  • Problem: IAD detects and localizes anomalies in industrial product images, but real anomalies are rare and unpredictable, so training uses only normal samples.Existing methods typically output anomaly scores and require manually specified class-specific thresholds to distinguish normal from anomalous instances.
  • Approach: AnomalyGPT detects anomaly presence and location without manual threshold settings, while also providing image information and supporting interactive follow-up questions.The model is designed for multi-turn dialogue focused on anomaly detection.
  • Approach: Prompt embeddings fine-tune the LVLM with synthesized anomalous visual-textual data, addressing data scarcity and reducing risks of overfitting and catastrophic forgetting.The method trains with data also used during LVLM pre-training to preserve inherent capabilities.
  • Results: 93.3% accuracy, 97.4% image-level AUC, and 93.1% pixel-level AUC are achieved with unsupervised training on MVTec-AD.One-shot transfer produces 77.4% accuracy, 87.4% image-level AUC, and 96.2% pixel-level AUC on VisA; the reverse transfer produces 86.1%, 94.1%, and 95.3%, respectively.
  • Results: The method combines anomaly detection and localization with multi-round dialogue and few-shot in-context learning on new datasets.The contributions describe robust transferability using only a small number of normal samples.

2. Related Work

Prior industrial anomaly detection methods rely on reconstruction or normal-feature modeling and commonly follow a one-class-one-model paradigm. AnomalyGPT instead uses simulated anomaly data, an image decoder, and prompt embeddings to support threshold-free detection and few-shot learning for novel object categories.

  • Industrial Anomaly Detection: Reconstruction-based methods detect anomalies by comparing reconstructed samples with corresponding normal counterparts using reconstruction error.Representative architectures include autoencoders, GANs, Transformers, and diffusion models.
  • Industrial Anomaly Detection: Feature embedding-based methods model normal-sample embeddings using hyperspheres, normalizing flows, or memory banks and nearest-neighbor distances.
  • Industrial Anomaly Detection: Traditional methods often require plentiful normal samples for each object class under a one-class-one-model paradigm, limiting use on novel categories and dynamic production environments.
  • Zero-/Few-shot Industrial Anomaly Detection: AnomalyGPT enables inference on novel object categories with only a few normal samples through in-context learning.
  • AnomalyGPT: AnomalyGPT uses simulated anomaly data, an image decoder, and prompt embeddings to address domain-specific IAD while avoiding manually specified thresholds.Table 1 compares the method with existing approaches across functions such as anomaly scores and direct anomaly judgement.

3. Method

AnomalyGPT combines a frozen image encoder, feature-matching decoder, prompt learner, and LLM to detect and localize industrial anomalies. It trains with simulated anomalous image-text data and supports both unsupervised and few-shot inference.

  • Overall Architecture: AnomalyGPT aligns industrial images and textual descriptions using a pre-trained image encoder, LLM, simulated anomaly data, decoder, and prompt learner.Prompt tuning and alternate training preserve transferability and prevent catastrophic forgetting.
  • Decoder: The decoder projects intermediate patch features into text-feature space and compares them with normal and abnormal semantics to produce pixel-level localization.Additional linear layers address the lack of final image-text alignment in intermediate features.
  • Few-Shot Inference: In few-shot IAD, normal patch features are stored in stage-specific memory banks, and query patches are localized by distance to their most similar normal counterparts.This memory-bank pathway uses only normal samples for comparison.
  • Prompt Learner: The prompt learner converts localization results into prompt embeddings and combines them with image embeddings and user text inputs for the LLM.It uses learnable base embeddings and decoder-derived embeddings before LLM processing.
  • Anomaly Simulation: Anomaly data are simulated by Poisson image editing, which makes pasted anomalous regions more natural than ordinary cut-paste results.Cut-paste can create noticeable discontinuities, whereas Poisson editing is designed to clone objects seamlessly.
  • Image-Text Alignment: Generated questions describe the object and ask whether anomalies exist, while a 3 × 3 image grid lets the LLM verbally report anomalous locations.Training uses textual responses that indicate anomaly presence, count, and grid position.
  • Loss Functions: Training uses cross-entropy, focal, and dice losses, with focal and dice losses primarily improving pixel-level decoder localization.Focal loss addresses class imbalance because most image regions remain normal; loss coefficients default to 1.

4. Experiments

Experiments evaluate AnomalyGPT on MVTec-AD and VisA under unsupervised and few-shot settings, with ablations examining its modules and inference choices. Results show strong anomaly detection, localization, transfer, and dialogue capabilities, while one-shot localization is slightly lower than unsupervised performance.

  • Experimental Setup: Experiments use MVTec-AD and VisA, including unsupervised training, one-shot transfer, and ablation settings.MVTec-AD contains normal-only training images and mixed normal/anomalous test images; Table 4 evaluates decoder, prompt learner, LLM inference, and LoRA choices.
  • Quantitative Results: Few-shot results outperform prior methods on image-level AUC across both datasets while achieving competitive pixel-level AUC and good accuracy.The comparison uses SPADE, PaDiM, PatchCore, and WinCLIP as baselines.
  • Qualitative Examples: AnomalyGPT detects anomaly presence, pinpoints locations, provides pixel-level localization, and supports questions about image content.In one-shot in-context learning, localization performance is slightly lower because parameter training is absent.
  • Ablation Studies: The decoder improves pixel-level localization, LLM inference outperforms manually set thresholds in accuracy, and prompt tuning outperforms LoRA in accuracy and transferability.

5. Conclusion

AnomalyGPT applies a conversational LVLM to industrial anomaly detection, determining anomaly presence and locations without manual thresholds while supporting dialogue and few-shot in-context learning. Its effectiveness is validated on two common datasets, suggesting applications of LVLMs in industrial anomaly detection.

  • AnomalyGPT is a conversational vision-language model that determines whether industrial images contain anomalies and pinpoints their locations without manually specified thresholds.
  • The model enables multi-turn dialogues focused on anomaly detection and demonstrates few-shot in-context learning.
  • Its effectiveness is validated on the MVTec-AD and VisA datasets.

A. More Experimental Results of Existing IAD Methods

Experiments show that threshold selection strongly affects existing IAD performance, with no single threshold working reliably across object categories.

  • A. More Experimental Results of Existing IAD Methods: Existing IAD methods require normal and anomalous samples with anomaly scores to identify an appropriate threshold for each object category.Anomaly scores alone do not determine whether a sample is normal or abnormal without a threshold.
  • A. More Experimental Results of Existing IAD Methods: A threshold based on maximum normal-sample scores is suited only to one-class-one-model methods and remains uncertain for unseen categories.This limitation makes threshold selection difficult when the applicable object category is unknown.
  • A. More Experimental Results of Existing IAD Methods: Threshold choice significantly influences PatchCore and WinCLIP accuracy across MVTec-AD categories.The evaluation varies thresholds for each category and compares the resulting accuracies.
  • A. More Experimental Results of Existing IAD Methods: The optimal threshold varies considerably across object categories for both PatchCore and WinCLIP.Figures 7 and 8 visualize category-specific results under different threshold settings.

B. Normal and Abnormal Texts

The method represents normal and abnormal semantics with compositional prompts assembled from state-level descriptions and template-level text. Object names are inserted into these prompts, with “object” used when no item name is available.

  • Compositional prompt ensembles represent both normality and abnormality through multi-level text.The complete text combines state-level descriptions with template-level text.
  • The token [c] is replaced by a state-level text and [o] by the object name; “object” substitutes when the name is unavailable.
  • State-level normal prompts include flawless, perfect, and unblemished object descriptions.They also include objects without flaw, defect, or damage.
  • State-level anomaly prompts describe damaged or broken objects and objects with flaws, defects, or damage.

C. Detailed Image Description

The prompting scheme can include a structured image description that identifies objects and expected attributes, giving the LVLM foundational knowledge about the input image. These descriptions are supplied for categories in the MVTec-AD and VisA datasets and may be omitted by users.

  • Prompts use an image-description field alongside the image embedding and user prompt.The illustrated prompt asks whether any anomaly is present in the image.
  • Image descriptions provide the objects present and their expected attributes to support comprehension of image contents.
  • Detailed category descriptions are provided for every category in the MVTec-AD dataset.They are added to corresponding prompts during training as foundational input-image knowledge.
  • Detailed category descriptions are also provided for every category in the VisA dataset.Users can omit the descriptive input when using the prompting scheme.

D. More Qualitative Examples

Qualitative comparisons across normal and anomalous industrial images show AnomalyGPT handling anomaly detection together with image-content questions. The examples cover bottles, wood, pills, fabric, metal grids, and cables, including pixel-level localization.

  • AnomalyGPT is the only compared method reported as proficient in both anomaly-detection questions and image-content questions.The comparison includes PandaGPT, MiniGPT-4, and LLaVA across normal and anomalous samples.
  • The qualitative examples include a normal bottle viewed from above, with anomaly detection, localization, pixel-level results, and image-question answering.
  • Examples also cover cutting wood, a normal pill, fabric with a hole, a normal metal grid, and a defective cable.Across these examples, AnomalyGPT is described as detecting anomalies, pinpointing locations, producing pixel-level localization, and answering image questions.
Loading 2308.15366v4…