PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection

Xiaofan Li, Zhizhong Zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yuan Xie, Lizhuang Ma

arXiv:2404.05231v2cs.CV

TL;DR

Few-shot industrial anomaly detection needs automated prompt learning, but conventional many-class prompt learning is poorly suited to one-class training with only normal samples. PromptAD constructs anomaly prompts by concatenating anomaly suffixes with normal prompts and introduces an explicit anomaly margin. It achieves first place in 11/12 few-shot image-level or pixel-level settings on MVTec and VisA.

  • Problem

    Conventional prompt learning does not work well for one-class anomaly detection because anomaly samples are unavailable for contrastive learning and margin control.

  • Method

    PromptAD uses semantic concatenation to construct anomaly prompts from normal prompts and an explicit anomaly margin to control normal–anomaly prompt-feature distance.

  • Results

    PromptAD achieves first place in 11/12 few-shot image-level or pixel-level settings on MVTec and VisA.

  • Takeaways & Limitations

    PromptAD demonstrates one-class prompt learning with only normal samples for few-shot anomaly detection.

Abstract

from arXiv · show

The vision-language model has brought great improvement to few-shot industrial anomaly detection, which usually needs to design of hundreds of prompts through prompt engineering. For automated scenarios, we first use conventional prompt learning with many-class paradigm as the baseline to automatically learn prompts but found that it can not work well in one-class anomaly detection. To address the above problem, this paper proposes a one-class prompt learning method for few-shot anomaly detection, termed PromptAD. First, we propose semantic concatenation which can transpose normal prompts into anomaly prompts by concatenating normal prompts with anomaly suffixes, thus constructing a large number of negative samples used to guide prompt learning in one-class setting. Furthermore, to mitigate the training challenge caused by the absence of anomaly images, we introduce the concept of explicit anomaly margin, which is used to explicitly control the margin between normal prompt features and anomaly prompt features through a hyper-parameter. For image-level/pixel-level anomaly detection, PromptAD achieves first place in 11/12 few-shot settings on MVTec and VisA.

1. Introduction

PromptAD addresses the difficulty of automated prompt learning for one-class industrial anomaly detection, where training uses only normal samples. It constructs anomaly prompts from normal prompts and uses an explicit margin, achieving strong few-shot results with few prompts.

  • Motivation: One-class industrial anomaly detection trains with only normal samples but must identify anomalous samples during testing.Few-shot training is valuable because industrial production lines often require rapidly customized models.
  • Motivation: Manual prompt engineering improves anomaly detection but requires human-designed prompts and can saturate at around 1000 prompts.This dependence conflicts with automation requirements in industrial scenarios.
  • Challenges: Conventional many-class prompt learning performs poorly in the one-class setting because contrastive learning lacks anomaly samples and an explicit normal–anomaly margin.These are the two challenges PromptAD is designed to address.
  • PromptAD: Semantic concatenation converts normal prompts into anomaly prompts by adding manually annotated or learnable anomaly suffixes, creating negative prompts for contrastive learning.Learnable anomaly prompts are aligned with manual anomaly prompts to acquire anomaly information despite limited annotations.
  • PromptAD: Explicit Anomaly Margin uses a hyper-parameter to make normal features closer to normal prompts than to anomaly prompts.This explicitly controls the distance between normal and anomaly prompt features.
  • Results: 91.3%/92.5% image-level/pixel-level results are achieved with only 10∼20 prompts, outperforming the baseline and WinCLIP on MVTec.The reported gains are 1.2%/7.7% over WinCLIP and 9.8%/3.7% over the baseline for image-level/pixel-level detection, respectively.
  • Results: PromptAD ranks first in 11/12 few-shot image-level or pixel-level settings on MVTec and VisA.This is the paper’s stated benchmark-level contribution.

2. Related Work

The related work situates PromptAD among vision-language models, conventional anomaly-detection paradigms, and few-shot anomaly-detection methods. It also highlights prior use of CLIP and visual feature approaches relevant to PromptAD.

  • Vision-Language Models: CLIP is a widely used vision-language model whose web-scale image-text pretraining provides strong zero-shot classification ability.Prompt engineering and prompt learning have enabled improvements on downstream tasks.
  • Anomaly Detection: Traditional anomaly-detection methods commonly follow feature embedding, knowledge distillation, or reconstruction-based paradigms.These paradigms respectively use feature differences, teacher–student knowledge transfer, or reconstructed-image differences.
  • PromptAD Architecture: PromptAD’s visual encoder uses V-V attention, with the original branch extracting CLS features and the attention branch extracting feature maps.This architecture is part of the method’s visual processing design.
  • Few-Shot Anomaly Detection: Few-shot anomaly-detection research includes methods such as TDG, RegAD, PatchCore, DifferNet, WinCLIP, RWDA, and FastRecon.CLIP-based methods improved few-shot performance, while FastRecon reconstructs anomaly features with distribution regularization.

3. Preliminaries

CLIP predicts image–prompt associations from visual and text embeddings, while prompt learning replaces fixed text descriptions with trainable tokens. CLIP Surgery modifies visual attention to improve local feature extraction for localization.

  • CLIP: CLIP predicts an image’s distribution over K text prompts using visual–text cosine similarities and a temperature parameter.The visual encoder produces f(i), the text encoder produces g(s_k), and τ controls the softmax scale.
  • Prompt Learning: Prompt learning automatically learns efficient CLIP prompts rather than relying on frozen textual descriptions.CoOp introduces this paradigm for few-shot classification.
  • Prompt Learning: A learned prompt combines trainable context tokens with a non-trainable class name.The trainable tokens are represented as [P1][P2] . . . [PEP ], while [classk] identifies the class.
  • CLIP Surgery: CLIP’s global feature extraction through Q-K self-attention limits its adaptation to image localization without fine-tuning.Prior explainability studies analyze this mechanism to explain CLIP’s localization weakness.
  • CLIP Surgery: V-V attention enhances local feature attention while preserving the original CLIP structure.PromptAD uses the resulting original and local-aware outputs for global and local anomaly features.

4. Methodology

PromptAD learns normal and anomaly prompts using only normal training samples, then combines prompt-guided and vision-guided scores for anomaly detection.

  • Semantic Concatenation: PromptAD uses semantic concatenation to transpose learnable normal prompts into anomaly prompts by appending manual or learnable anomaly suffixes.Manual suffixes include dataset anomaly labels, while learnable anomaly suffixes provide additional prompt representations.
  • Semantic Concatenation: Normal prompts are trained toward normal visual features, whereas manual and learnable anomaly prompts are trained away from them.Prompt parameters concatenated by the same normal prefix or anomaly suffix are shared.
  • Explicit Anomaly Margin: Explicit Anomaly Margin adds a margin-based regularization loss to control distances between normal prompt features and anomaly prompt features.The margin hyper-parameter ensures normal samples are closer to the normal prototype than to the anomaly prototype.
  • Anomaly Detection: PromptAD uses normal and anomaly prompt prototypes to calculate image-level and pixel-level anomaly scores from global or local image features.The score uses similarities to the two prototypes, with z_t representing the relevant global or local feature.
  • Anomaly Detection: Vision-guided and prompt-guided maps are fused for pixel-level detection, while the maximum vision-guided score and prompt score are fused for image-level detection.The fusion uses a harmonic mean, which is more sensitive to smaller values.

5. Experiments

PromptAD is evaluated on MVTec and VisA using AUROC across image-level and pixel-level few-shot anomaly detection, with comparisons, ablations, and CLIP transformation studies. Results show strong few-shot performance, especially through semantic concatenation, explicit anomaly margin, and complementary prompt- and vision-guided detection.

  • Image-level comparison: PromptAD achieves significant image-level improvement over WinCLIP+ and RWDA across 1-, 2-, and 4-shot settings on MVTec and VisA.Improvements are 1.3%, 1.3%, and 1.4% on MVTec, and 2.9%, 2.7%, and 1.8% on VisA, respectively.
  • Pixel-level comparison: PromptAD ranks first in 1- and 2-shot pixel-level detection on both MVTec and VisA, while narrowly missing first place on 4-shot MVTec.It exceeds WinCLIP+ by 0.7%/0.3% at 1-shot and 0.2%/0.3% at 2-shot on MVTec/VisA; FastRecon leads 4-shot MVTec by 0.5%.
  • Comparison with many-shot methods: PromptAD achieves competitive few-shot results against many-shot methods, outperforming early full-shot methods but remaining behind the latest PatchCore and SimpleNet methods.The comparison covers image- and pixel-level results on MVTec.
  • Ablation study: Removing semantic concatenation reduces image- and pixel-level performance, while adding it improves MVTec results by 8.9%/3.9% and VisA results by 8.9%/5.0%.The paired values correspond to image-level/pixel-level results.
  • Ablation study: Adding explicit anomaly margin improves image-level/pixel-level results by 0.9%/0.8% on MVTec and 1.9%/0.7% on VisA.EAM uses a hyper-parameter to control the margin between normal and anomaly prompt features.
  • Ablation study: PAD provides better image-level results, VAD provides better pixel-level results, and their fusion reaches 94.6%/95.9% under the 1-shot setting.PAD uses high-level semantic information, whereas VAD emphasizes local detail through normal feature memory.

6. Conclusion

PromptAD learns anomaly-detection prompts using only normal samples in few-shot settings. It combines semantic concatenation with an explicit anomaly margin and ranks first in 11/12 few-shot tasks.

  • PromptAD automatically learns prompts with only normal samples for few-shot anomaly detection.
  • Semantic concatenation constructs anomaly prompts by appending anomaly suffixes to normal prompts.
  • Explicit anomaly margin controls the distance between normal and anomaly prompt features through a hyper-parameter.
  • PromptAD achieves first place in 11/12 few-shot image-level and pixel-level anomaly-detection tasks.

A. Experimental details

The experiments evaluate PromptAD across few-shot datasets, metrics, qualitative settings, and prompt-learning comparisons. Results include qualitative localization, comparisons with contextual prompt engineering, and reported AUROC measurements.

  • Experimental details: The evaluation supplements image-level AUPR and pixel-level PRO alongside AUROC.
  • Experimental details: Experiments report mean and standard deviation over 5 random seeds because few-shot performance depends on random sampling.
  • Qualitative results: PromptAD accurately locates both large-area and small-area surface defects in 1-shot pixel-level results on MVTec and VisA.
  • Comparison with prompt-learning methods: Classical prompt-learning methods perform relatively poorly in 1-shot anomaly detection, with several results below contextual prompt engineering.
  • Comparison with prompt-learning methods: PromptAD improves over contextual prompt engineering by 0.8%/5.8% on MVTec and 5.0%/8.9% on VisA for image-level/pixel-level results.

D. Ablation Study

Ablations examine anomaly suffixes, visual backbones, and benchmark configurations. Semantic concatenation and explicit anomaly margins improve performance, while different backbones produce different image- and pixel-level strengths.

  • Anomaly prompt variants: Removing manual anomaly prompts causes more pronounced degradation, while PromptAD remains better than contextual prompt engineering.
  • Experimental configurations: PromptAD’s backbone and feature-memory configurations are evaluated with AUROC on image-level and pixel-level tasks.
  • Visual backbones: ViT backbones outperform ResNet overall, and ViT-L/14 provides better pixel-wise anomaly detection than ViT-B/16+.

E. Results on Other Benchmarks

PromptAD is additionally evaluated on MPDD and LOCO, with comparisons against PatchCore and WinCLIP+. The paper also reports detailed subset-level and supplementary metric results.

  • Benchmark comparison: PromptAD achieves first place on both MPDD and LOCO in few-shot settings compared with PatchCore and WinCLIP+.
  • Supplementary evaluation: Supplementary experiments report subset-level MVTec and VisA results together with image-level AUPR and pixel-level PRO.

G. Visualization Results of Attention Map

The attention visualizations distinguish QK attention, which shifts from local to global information across layers, from VV attention, which remains local and supports localization. PromptAD therefore selects intermediate layers to preserve complementary local and global information.

  • QK attention focuses on local information in shallow layers but becomes more global from layer 5 onward.
  • PromptAD stores features from the 3th and 8th layers instead of PatchCore’s 6th and 9th layers to preserve local and global information.The 3th+8th features improve results by 1.2% over the 6th+9th features.
  • VV attention remains focused on local information throughout the vision encoder, making it more conducive to localization.
Loading 2404.05231v2…