PaperlayerView exact revision on arXiv

Source-linked AI summary

AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection

Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, Jiming Chen

arXiv:2310.18961v12cs.CV

TL;DR

Zero-shot anomaly detection lacks target-domain training data, while CLIP-based methods can focus on object semantics instead of abnormalities. AnomalyCLIP learns object-agnostic normality and abnormality prompts with global and local context optimization, achieving superior ZSAD performance across 17 public datasets.

  • Problem

    ZSAD must detect anomalies without target-domain training examples, but CLIP-based methods primarily align object semantics rather than abnormality or normality.

  • Method

    AnomalyCLIP learns object-agnostic normality and abnormality prompts using joint global and local context optimization for detection and segmentation.

  • Results

    AnomalyCLIP achieves superior ZSAD performance across 17 public datasets spanning diverse industrial defect inspection and medical imaging domains.

  • Takeaways & Limitations

    Object-agnostic prompt learning supports generalized anomaly detection and segmentation across datasets with diverse foreground objects and anomaly types.

Abstract

from arXiv · show

Zero-shot anomaly detection (ZSAD) requires detection models trained using auxiliary data to detect anomalies without any training sample in a target dataset. It is a crucial task when training data is not accessible due to various concerns, eg, data privacy, yet it is challenging since the models need to generalize to anomalies across different domains where the appearance of foreground objects, abnormal regions, and background features, such as defects/tumors on different products/organs, can vary significantly. Recently large pre-trained vision-language models (VLMs), such as CLIP, have demonstrated strong zero-shot recognition ability in various vision tasks, including anomaly detection. However, their ZSAD performance is weak since the VLMs focus more on modeling the class semantics of the foreground objects rather than the abnormality/normality in the images. In this paper we introduce a novel approach, namely AnomalyCLIP, to adapt CLIP for accurate ZSAD across different domains. The key insight of AnomalyCLIP is to learn object-agnostic text prompts that capture generic normality and abnormality in an image regardless of its foreground objects. This allows our model to focus on the abnormal image regions rather than the object semantics, enabling generalized normality and abnormality recognition on diverse types of objects. Large-scale experiments on 17 real-world anomaly detection datasets show that AnomalyCLIP achieves superior zero-shot performance of detecting and segmenting anomalies in datasets of highly diverse class semantics from various defect inspection and medical imaging domains. Code will be made available at https://github.com/zqhang/AnomalyCLIP.

1 INTRODUCTION

AnomalyCLIP addresses weak zero-shot anomaly detection by learning object-agnostic prompts for generic normality and abnormality rather than foreground-object semantics. It uses auxiliary data, an object-agnostic prompt template, and a glocal abnormality loss combining global and local objectives.

  • Anomaly detection supports applications including industrial defect inspection and medical image analysis.
  • Large pre-trained vision-language models demonstrate strong zero-shot recognition across vision tasks, including anomaly detection.
  • AnomalyCLIP learns object-agnostic text prompts whose embeddings model generic abnormality and normality instead of object semantics.
  • AnomalyCLIP uses auxiliary data, an object-agnostic prompt template, and a glocal abnormality loss combining global and local loss functions.

2 PRELIMINARY

CLIP uses text and visual encoders to compare textual and visual embeddings for zero-shot recognition. For zero-shot anomaly detection, this comparison extends from global image representations to local patch representations to produce anomaly segmentation maps.

  • CLIP preliminaries: CLIP comprises text and visual encoders, denoted T(·) and F(·), that produce textual embeddings and global or local visual representations.The class token f_i ∈ R^D is the global visual embedding, while patch tokens f_i^m ∈ R^(H×W×D) are local visual embeddings.
  • CLIP preliminaries: CLIP performs zero-shot recognition by measuring cosine similarity between textual and visual embeddings.The prediction uses a temperature hyperparameter τ in the similarity-based probability computation.
  • ZSAD formulation: For ZSAD, the computation is extended from global visual embeddings to local visual embeddings to derive normality and abnormality segmentation maps.The resulting maps are S_n ∈ R^(H×W) and S_a ∈ R^(H×W), with entries computed from local patch embeddings.

3 ANOMALYCLIP: OBJECT-AGNOSTIC PROMPT LEARNING

AnomalyCLIP adapts CLIP for zero-shot anomaly detection by learning object-agnostic prompts for generic normality and abnormality. It combines global and local context optimization with textual and local visual-space refinement to detect and segment anomalies across diverse domains.

  • 3.1 APPROACH OVERVIEW: AnomalyCLIP introduces object-agnostic text prompts to learn generalized embeddings for normality and abnormality regardless of foreground object semantics.The approach combines global and fine-grained anomaly semantics during prompt learning.
  • 3.2 OBJECT-AGNOSTIC TEXT PROMPT DESIGN: Common CLIP templates emphasize object semantics, so they fail to produce textual embeddings that capture anomaly and normality semantics for visual matching.The method therefore incorporates prior anomaly semantics into its prompt templates.
  • 3.2 OBJECT-AGNOSTIC TEXT PROMPT DESIGN: AnomalyCLIP targets generic anomaly patterns shared across industrial defects and medical tumors despite substantial differences in object semantics.Examples include scratches, misplaced components, and tumors on different organs.
  • 3.2 OBJECT-AGNOSTIC TEXT PROMPT DESIGN: The abnormality template uses [damaged][object] rather than a dataset-specific object name, enabling anomaly recognition across diverse objects and domains without target-domain modification.The design requires no knowledge of target object names or anomaly types.
  • 3.3 LEARNING GENERIC ABNORMALITY AND NORMALITY PROMPTS: Glocal context optimization jointly learns normality and abnormality prompts from global and local perspectives.Global optimization matches object-agnostic textual embeddings with global visual embeddings from normal and abnormal auxiliary images.
  • 3.3 LEARNING GENERIC ABNORMALITY AND NORMALITY PROMPTS: Focal loss addresses the smaller size of anomalous regions, while Dice loss helps establish an accurate segmentation decision boundary.The segmentation objective uses both focal and Dice losses.
  • 3.3 LEARNING GENERIC ABNORMALITY AND NORMALITY PROMPTS: Learnable token embeddings refine CLIP’s textual space for adaptation to anomaly detection, with prefix replacement controlling the refinement degree.Additional multi-layer trainable tokens provide layer-wise textual adaptation.
  • 3.3 LEARNING GENERIC ABNORMALITY AND NORMALITY PROMPTS: During inference, P(ga, fi) serves as the image-level anomaly score, while pixel predictions merge selected-layer segmentations before interpolation and smoothing.The score increases toward one when the abnormality embedding aligns with the global visual embedding.

4 EXPERIMENTS

Experiments across 17 diverse industrial and medical anomaly-detection datasets show that AnomalyCLIP generalizes effectively for zero-shot detection and segmentation. Ablations further attribute its performance to object-agnostic prompts, global/local context optimization, DPAM, and learnable text tokens.

  • Datasets and Evaluation Metrics: AnomalyCLIP is evaluated on 17 publicly available datasets spanning industrial inspection and medical imaging domains.The industrial settings include varied defect scenarios, while the medical settings cover photography, endoscopy, and radiology.
  • ZSAD performance on diverse industrial inspection domains: AnomalyCLIP substantially outperforms five competing methods across most of seven industrial defect datasets with diverse objects, backgrounds, and anomaly types.The weak performance of CLIP and CLIP-AC is attributed to limited fine-grained local anomaly semantics and emphasis on object alignment.
  • ZSAD performance on diverse industrial inspection domains: Object-agnostic prompts model generic abnormality and normality through global and local context optimization, enabling generalization across datasets and domains.This design shifts attention from foreground-object semantics toward abnormal image regions and supports both global and local anomaly modeling.
  • Generalization from defect datasets to diverse medical domain datasets: AnomalyCLIP and VAND achieve promising zero-shot anomaly-detection performance across 10 medical image datasets covering different organs and imaging devices.Learning-based methods, including AnomalyCLIP, are tuned using MVTec AD data before evaluation on the medical datasets.
  • Module ablation: DPAM, object-agnostic prompts, and learnable text-encoder tokens each contribute to AnomalyCLIP’s performance in module ablation experiments.DPAM improves segmentation by enhancing local visual semantics, while object-agnostic prompts emphasize image abnormality and normality over object semantics.
  • Context optimization: Both global and local context optimization contribute to AnomalyCLIP’s superiority, with global optimization supporting image-level anomaly detection and local optimization supporting segmentation.The ablation studies evaluate the two optimization terms used to drive object-agnostic prompt learning.

5 RELATED WORK

Prior CLIP-based zero-shot anomaly detection methods primarily address anomaly classification or require target-domain auxiliary-data tuning, whereas AnomalyCLIP is designed for cross-dataset use after training on one dataset. In prompt learning, AnomalyCLIP extends parameter-efficient learnable prompting toward object-agnostic anomaly detection.

  • Zero-shot anomaly detection: CLIP-AD and ZOC were early CLIP-based ZSAD studies, but mainly focused on anomaly classification.These methods are contrasted with AnomalyCLIP’s broader zero-shot anomaly detection setting.
  • Zero-shot anomaly detection: ACR requires target-domain-relevant auxiliary-data tuning for ZSAD on different target datasets.The passage contrasts this requirement with AnomalyCLIP’s cross-dataset applicability after training on one dataset.
  • Prompt learning: Prompt learning offers a parameter-efficient alternative to full network fine-tuning for achieving satisfactory results.The passage situates AnomalyCLIP within prior prompt-learning research.
  • Prompt learning: CoOp introduced learnable text prompts for few-shot classification, while DenseCLIP extended prompt learning to dense prediction with an extra image decoder.AnomalyCLIP is presented as a different prompt-learning approach that proposes object-agnostic prompts.

6 CONCLUSION

The paper addresses zero-shot anomaly detection without target-dataset training data by proposing AnomalyCLIP, which uses object-agnostic prompt learning to improve CLIP’s generalization across diverse foreground objects.

  • 6 CONCLUSION: AnomalyCLIP targets ZSAD, where no training data are available from the target dataset.The method is designed to address CLIP’s weak generalization performance in this setting.
  • 6 CONCLUSION: Object-agnostic prompt learning learns generic abnormality and normality text prompts for images containing diverse foreground objects.This is intended to support generalized ZSAD across varied image datasets.
  • 6 CONCLUSION: AnomalyCLIP further seeks to incorporate global and local anomaly semantics.The supplied passage introduces this objective but ends before describing the specific mechanism.

A IMPLEMENTATION DETAILS AND BASELINES · A.1 IMPLEMENTATION DETAILS · A.2 BASELINES

AnomalyCLIP uses a frozen CLIP VIT-L/14@336px backbone with learnable object-agnostic prompt tokens, fine-tuned on MVTec AD and VisA test data. It is compared with CLIP, CLIP-AC, WinCLIP, VAND, and CoOp using baseline-specific prompt and segmentation procedures.

  • A.1 IMPLEMENTATION DETAILS: AnomalyCLIP uses publicly available CLIP VIT-L/14@336px as its frozen backbone.The learnable text prompt length M is set to 12; tokens attach to the first 9 text-encoder layers, each with length 4.
  • A.1 IMPLEMENTATION DETAILS: AnomalyCLIP is fine-tuned on MVTec AD test data and evaluated on other datasets, with VisA used for MVTec AD fine-tuning.
  • A.2 BASELINES: CLIP uses prompts “A photo of a normal [cls]” and “A photo of an anomalous [cls]” for anomaly detection.Anomaly scores follow Eq. 1, while segmentation extends the computation to local visual embeddings.
  • A.2 BASELINES: CLIP-AC averages textual embeddings from ImageNet-recommended prompt ensembles for normal and anomaly classes.It computes probabilities and segmentation in the same way as CLIP.
  • A.2 BASELINES: WinCLIP uses hand-crafted anomaly-detection prompt templates and window scaling for anomaly segmentation.All parameters are kept the same as in the WinCLIP paper.
  • A.2 BASELINES: VAND improves WinCLIP by adjusting text prompts and adding learnable linear projections for local visual semantics.The projections are used to derive more accurate segmentation, with parameters kept as in the original paper.
  • A.2 BASELINES: CoOp adapts prompt learning to ZSAD by inserting learnable embeddings before normality or abnormality tokens and the target class.Anomaly probabilities and segmentation are obtained in the same way as for AnomalyCLIP.

B DATASET

The study evaluates its methods on 17 public datasets spanning two domains and three modalities, using only test data with standardized preprocessing. It also constructs a medical dataset by combining existing datasets to provide pixel-level annotations and normal samples.

  • Dataset scope: 17 public datasets spanning two domains and three modalities are used for extensive evaluation.The reported dataset information concerns the test sets because only test data are used.
  • Preprocessing: All datasets undergo OpenCLIP’s default normalization and are resized to 518 × 518.The resizing is intended to obtain an appropriate visual feature map resolution.
  • Medical dataset construction: A medical dataset is created by combining ColonDB for pixel-level annotations with another 2D medical dataset supplying normal samples.The construction addresses the lack of publicly available 2D medical anomaly-detection datasets containing both category labels and segmentation ground truths.

C DETAILED ANALYSIS OF DPAM

The analysis identifies CLIP’s global semantic attention as a source of local-feature disturbance and introduces modified self-attention to preserve local visual semantics. The resulting more diagonal attention maps improve anomaly segmentation.

  • Problem in CLIP Attention: CLIP’s visual encoder prioritizes global object semantics, causing attention to focus on specific tokens that disrupt local visual embeddings.The contrastive pretraining objective produces representative global embeddings for semantic-class recognition.
  • Problem in CLIP Attention: When the original attention map concentrates on token m, q_i assigns the largest score to k_m, so Attention(Q, K, V)_i becomes dominated by v_m.This dominance causes the local visual embedding at index i to be disturbed by the focused token.
  • Modified Attention: The modification gives q_i, k_i, and v_i significant weight in self-attention, preserving local visual semantics and producing more diagonal attention maps.The modified forms are Attention(Q, Q, V)_i, Attention(K, K, V)_i, and Attention(V, V, V)_i.
  • Modified Attention: More diagonal attention maps improve performance in anomaly segmentation.The improvement is shown in Fig. 3c, Fig. 3d, and Fig. 3e.

D ADDITIONAL RESULTS AND ABLATIONS

AnomalyCLIP performs comparably to strong full-shot methods and surpasses them on some datasets, while ablations support the effectiveness and robustness of its object-agnostic prompts. Hyperparameter analysis shows prompt length requires tuning because excessively long prompts can reduce performance.

  • Comparison with SOTA full-shot methods: AnomalyCLIP achieves comparable detection and segmentation performance to PatchCore and RD4AD, outperforming them on some datasets.Experiments were conducted on six public datasets lacking normal training data.
  • Comparison with SOTA full-shot methods: Generic prompt embeddings enable AnomalyCLIP to capture normality and abnormality beyond the performance boundary imposed by training data.
  • Hyparameter analysis: Detection and segmentation performance initially improves as learnable text-prompt length E increases, then declines for lengths from 12 to 16.The decline suggests that excessively long learnable text prompts may introduce redundant information.
  • Prompt template ablation: Replacing “damaged” with semantically similar anomaly terms produces steady results, indicating robustness to the abnormality-related token.Tested alternatives include anomalous, flawed, defective, and blemished.
  • Object ablation: AnomalyCLIPre remains effective in ZSAD after replacing the generic object token with a specific target class, supporting that prompts learn underlying anomaly patterns.The result holds even when object semantics are blocked during fine-tuning.

E VISUALIZATION · F FINE-GRAINED ZSAD PERFORMANCE

The visualization results show that CLIP’s object-semantic alignment produces overlapping normal and anomaly scores, whereas AnomalyCLIP improves anomaly-focused visualization and segmentation across diverse datasets. Fine-grained results are reported through subset-level comparisons and anomaly maps for industrial, texture, and medical subsets.

  • E VISUALIZATION: AnomalyCLIP similarity-score visualizations provide a comparison with CLIP on MVTec AD and VisA after adaptation.The passage reports a significant overlap between normal and anomaly scores for AnomalyCLIP but is truncated before describing the full comparison.
  • E VISUALIZATION: CLIP’s normal and anomaly similarity scores severely overlap on MVTec AD and VisA, with scores centered around 0.5.These visualizations indicate that CLIP’s textual and visual spaces, originally aligned for object semantics, are not well suited to ZSAD.
  • E VISUALIZATION: AnomalyCLIP anomaly-score maps visualize segmentation across industrial objects, VisA objects, MPDD objects, and industrial textures.Examples include hazelnut, pill, screw, candle, chewinggum, capsule, cashew, pcb, pip fryum, bracket, metal plate, tube, grid, leather, carpet, tile, and wood.
  • F FINE-GRAINED ZSAD PERFORMANCE: The fine-grained ZSAD section reports detailed data-subset-level performance comparisons.The supplied passage identifies this section’s purpose but does not provide the numerical table values.
  • F FINE-GRAINED ZSAD PERFORMANCE: MVTec AD fine-grained comparisons cover anomaly-segmentation AUROC and PRO, plus anomaly-classification AUROC and AP.These metrics are presented in Tables 10–13.
  • F FINE-GRAINED ZSAD PERFORMANCE: VisA fine-grained comparisons cover anomaly-segmentation AUROC and PRO, plus anomaly-classification AUROC and AP.These metrics are presented in Tables 14–17.
  • F FINE-GRAINED ZSAD PERFORMANCE: MPDD fine-grained comparisons cover anomaly-segmentation AUROC and PRO, plus anomaly-classification AUROC and AP.These metrics are presented in Tables 18–21.
Loading 2310.18961v12…