PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection

Yunkang Cao, Jiangning Zhang, Luca Frittoli, Yuqi Cheng, Weiming Shen, Giacomo Boracchi

arXiv:2407.15795v1cs.CV

TL;DR

Zero-shot anomaly detection seeks to identify anomalies in images from unseen categories without category-specific training images. AdaCLIP adapts CLIP with hybrid learnable prompts and achieves state-of-the-art performance across 14 industrial and medical datasets, including cross-domain categories.

  • Problem

    ZSAD seeks anomaly detection across unseen categories without category-specific training images, addressing settings where few normal samples prevent conventional training.

  • Method

    AdaCLIP adapts pre-trained CLIP using static and dynamic learnable prompts, while its HSF module fuses region-level anomaly information for image-level detection.

  • Results

    AdaCLIP achieves state-of-the-art ZSAD performance across 14 industrial and medical datasets and outperforms alternative methods across domains.

  • Takeaways & Limitations

    Diverse annotated auxiliary data can support ZSAD generalization across categories and between medical and industrial domains.

  • Takeaways & Limitations

    Without reference images, AdaCLIP may fail to detect anomalies lacking structural deviations, particularly those defined by contextual rather than structural abnormalities.

Abstract

from arXiv · show

Zero-shot anomaly detection (ZSAD) targets the identification of anomalies within images from arbitrary novel categories. This study introduces AdaCLIP for the ZSAD task, leveraging a pre-trained vision-language model (VLM), CLIP. AdaCLIP incorporates learnable prompts into CLIP and optimizes them through training on auxiliary annotated anomaly detection data. Two types of learnable prompts are proposed: static and dynamic. Static prompts are shared across all images, serving to preliminarily adapt CLIP for ZSAD. In contrast, dynamic prompts are generated for each test image, providing CLIP with dynamic adaptation capabilities. The combination of static and dynamic prompts is referred to as hybrid prompts, and yields enhanced ZSAD performance. Extensive experiments conducted across 14 real-world anomaly detection datasets from industrial and medical domains indicate that AdaCLIP outperforms other ZSAD methods and can generalize better to different categories and even domains. Finally, our analysis highlights the importance of diverse auxiliary data and optimized prompts for enhanced generalization capacity. Code is available at https://github.com/caoyunkang/AdaCLIP.

1 Introduction

Zero-shot anomaly detection addresses cold-start settings by detecting anomalies in unseen categories without category-specific training images. AdaCLIP adapts CLIP with hybrid learnable prompts and achieves state-of-the-art ZSAD performance across industrial and medical datasets while leveraging auxiliary data across domains.

  • 1 Introduction: Zero-shot anomaly detection (ZSAD) targets anomalies in unseen categories without requiring category-specific training images, addressing cold-start settings where normal data are unavailable.Traditional unsupervised and semi-supervised methods require normal or annotated abnormal images from the target category.
  • 1 Introduction: The introduction argues that universal normal or anomalous patterns and auxiliary data can transfer across categories, such as scratches in pills aiding detection on chewing gum.It further reports that optimized prompts matter for detecting anomalies within individual images and that other VLMs besides CLIP can be adapted for ZSAD.
  • 1 Introduction: AdaCLIP adapts CLIP for ZSAD using hybrid static and dynamic learnable prompts that exploit auxiliary data and improve generalization.Static prompts are shared across images, while dynamic prompts provide image-specific adaptation; the method also introduces projection and prompting layers.
  • 1 Introduction: AdaCLIP adds a hybrid-semantic fusion module to extract region-level anomaly context and enhance image-level anomaly detection.Its design computes similarities between patch embeddings and text embeddings describing normal and abnormal states.
  • 1 Introduction: AdaCLIP achieves state-of-the-art ZSAD performance across 14 industrial and medical datasets, outperforming alternative ZSAD methods.The method also leverages auxiliary datasets across different domains and shows that diverse auxiliary data benefits ZSAD.

2 Related Work · 3 Problem Formulation

Traditional anomaly detection relies on category-specific normal or annotated abnormal data, whereas ZSAD targets anomaly detection in unseen categories without target-category training samples. The formulation maps each image to image- and pixel-level abnormality predictions using auxiliary annotated data from distinct categories.

  • 2.1 Traditional Anomaly Detection: Traditional unsupervised anomaly detection learns target-category normal distributions and detects anomalies by comparing test samples with those distributions.Common implementations extract features using pre-trained neural networks and model feature distributions through distillation [53], reconstruction, or memory banks.
  • 2.1 Traditional Anomaly Detection: Semi-supervised anomaly detection uses annotated normal and abnormal images from target categories to learn a compact boundary around normal samples.Traditional methods become less effective when fewer normal samples are available, motivating generic ZSAD across unseen categories without training samples.
  • 2.2 Zero-shot Anomaly Detection: Zero-shot anomaly detection leverages pre-trained vision-language models to identify anomalies across unbounded categories.WinCLIP uses CLIP to compare image-patch embeddings with captions describing normal and abnormal states, with text augmentation further enhancing the approach.
  • 2.3 Prompt Learning: Prompt learning adapts vision-language models by adding learnable tokens to image or text inputs, with early methods primarily using static text prompts.CoOp adds learnable tokens to the text branch, while later work notes that static prompts can be vulnerable to distribution diversity.
  • 2.3 Prompt Learning: Recent prompt-learning research emphasizes visual prompting, and AdaCLIP combines image-plus-text static-plus-dynamic hybrid prompts for anomaly detection.The proposed multimodal hybrid design adapts vision-language models to better exploit their multimodal capabilities.
  • 3 Problem Formulation: The problem is to associate an input image I ∈ R^H×W×3 with an image-level anomaly score S and pixel-level anomaly map M ∈ R^H×W.Scores and map values lie in [0, 1], with larger values indicating higher abnormality probabilities.
  • 3 Problem Formulation: In ZSAD, training uses an auxiliary dataset containing annotated normal and abnormal images from categories distinct from the testing categories.Masks G ∈ R^H×W label normal pixels as 0 and abnormal pixels as 1; learning from the auxiliary data should capture patterns shared across classes for novel-category detection.

4 AdaCLIP

AdaCLIP adapts CLIP for zero-shot anomaly detection with lightweight prompting layers that combine auxiliary-data-trained static prompts and image-conditioned dynamic prompts. It aligns visual and textual embeddings, aggregates multi-layer anomaly maps, and uses HSF to derive robust image-level anomaly scores.

  • 4 AdaCLIP: AdaCLIP computes patch-level anomaly maps from cosine similarities with normal and abnormal text embeddings, aggregates maps across multiple layers, and trains them with dice and focal losses.Patch scores are reshaped and interpolated into anomaly maps, which are combined into the final prediction M on auxiliary data.
  • 4 AdaCLIP: AdaCLIP replaces CLIP transformer layers with prompting layers that concatenate M learnable tokens to N vanilla tokens, using M ≪ N for lightweight adaptation.Learnable prompting is applied only through depth J, while later-layer prompts are propagated; a small J helps reduce overfitting on auxiliary data.
  • 4 AdaCLIP: AdaCLIP combines static prompts shared across images with dynamic prompts generated from each testing image, forming hybrid prompts for both CLIP encoders.Static prompts are learned from auxiliary data, while a frozen CLIP backbone and learnable linear layer generate the dynamic prompts.
  • 4 AdaCLIP: A projection layer aligns patch and text embedding dimensions while adding learnable parameters for CLIP adaptation.The projection layer is appended to the image encoder before anomaly-score computation.
  • 4 AdaCLIP: The HSF module clusters patch embeddings, selects the cluster with the highest aggregated anomaly score, and combines its centroids into a semantic-rich image embedding for robust image-level detection.Image-level anomaly scores compare this embedding with normal and abnormal text embeddings and are optimized using focal loss.

5 Experiments

AdaCLIP is evaluated across industrial and medical anomaly-detection datasets against training-free and auxiliary-data-trained baselines. It achieves the strongest overall ZSAD performance, while analyses attribute gains to hybrid prompts, HSF, diverse auxiliary data, and prompt optimization.

  • Experimental Setup: AdaCLIP is compared with training-free SAA and WinCLIP, auxiliary-data-trained APRIL-GAN and AnomalyCLIP, and adapted DINOV2 [39] and SAM.The experiments span industrial datasets MVTec AD, VisA, MPDD, BTAD, KSDD, DAGM, and DTD-Synthetic, plus medical datasets.
  • Overall Comparison: 3.7% image-level and 3.3% pixel-level max-F1 improvements over the second-place method establish AdaCLIP as the top-ranked approach across industrial and medical datasets.AdaCLIP also achieves the highest average rankings and more precise anomaly localization across both domains.
  • Influence of Prompts: Hybrid prompts outperform static-only and dynamic-only variants because static prompts lack anomaly diversity and dynamic prompts alone provide insufficient adaptation.The combined static and dynamic prompts offer robust, flexible image adaptation and produce the strongest ZSAD performance.
  • Influence of Training Data: Training on diverse industrial anomalies improves cross-domain generalization, whereas medical-only training can underperform on ISIC because selected medical datasets lack anomaly diversity.ColonDB training yields promising results on ClinicDB, demonstrating that auxiliary-data composition materially affects transfer.
  • Rationale and Prompt Optimization: Auxiliary-data training improves ZSAD because VLMs perceive high-level similarities between normal and abnormal patch embeddings across MVTec and VisA.Optimizing prompts with corresponding anomaly masks makes abnormal embeddings more discernible and anomaly maps finer, especially for individual images.

6 Conclusion

AdaCLIP is a generic zero-shot anomaly detection model that adapts pre-trained CLIP with learnable hybrid prompts using annotated auxiliary anomaly-detection data. It also introduces an HSF module for region-level anomaly information to improve image-level detection, with experiments spanning 14 industrial and medical datasets.

  • 6 Conclusion: AdaCLIP detects anomalies across arbitrary novel categories without requiring reference images.It is designed as a generic ZSAD model for unseen categories.
  • 6 Conclusion: AdaCLIP adapts pre-trained CLIP for ZSAD by training learnable hybrid prompts on annotated auxiliary anomaly-detection data.The hybrid prompts provide the model’s learnable adaptation mechanism.
  • 6 Conclusion: The proposed HSF module extracts region-level anomaly information to enhance image-level anomaly-detection performance across 14 industrial and medical datasets.The conclusion frames the evaluation as extensive experimentation across these datasets.

Supplementary Materials for AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for … 2 Module Details

The supplementary materials detail AdaCLIP’s datasets, hybrid prompting design, Hybrid Semantic Fusion module, baselines, and comparative experiments. They report evaluations across industrial and medical anomaly-detection settings, including comparisons with AnomalyCLIP and full-shot unsupervised methods.

  • Zero-Shot Anomaly Detection: The appendix organizes further details on datasets, AdaCLIP, baselines, comparisons with AnomalyCLIP, and comparisons with full-shot unsupervised anomaly-detection methods.The latter comparison is presented as evidence of AdaCLIP’s potential practical applicability.
  • 1 Dataset Details: AdaCLIP is evaluated using test data from 14 public industrial and medical datasets spanning photography, radiology, and endoscopy.MVTec AD and ClinicDB are used by default as auxiliary training data, with VisA additionally used for evaluations on MVTec AD and ClinicDB.
  • 2.1 Hybrid Learnable Prompts: AdaCLIP adapts frozen pre-trained CLIP with hybrid learnable prompts that combine static prompts with image-specific dynamic prompts for both image and text encoders.Image embeddings from the frozen CLIP image encoder are projected through linear layers to generate the dynamic prompts, which are combined with static prompts from the initial J layers.
  • 2.1 Hybrid Learnable Prompts: Prompting both CLIP encoders outperforms prompting only the text encoder, with text-only prompting decreasing image- (pixel-) level AUROCs by 7.2% (0.1%) in medical and 2.0% (0.7%) in industrial domains.The text-only design corresponds to CoCoOp [15], whereas AdaCLIP uses multimodal prompting.
  • 2.2 Hybrid Semantic Fusion: Hybrid Semantic Fusion fuses region-level anomalies into a semantic-rich image embedding, improving image-level AUROCs over maximum value-based methods.Reported improvements for medical (industrial) settings are 2.4% (2.9%), 2.8% (4.2%), 3.8% (2.4%), and 0.3% (4.1%), respectively.
  • 2.2 Hybrid Semantic Fusion: HSF performance depends on the cluster count K: larger K creates smaller clusters, and sufficiently small clusters reduce HSF to the maximum value-based method.The ideal K matches testing-anomaly size, but anomaly-size variability makes one optimal K difficult across medical and industrial domains; K = 20 is used by default.

3 Comparison Method Details

The section positions AdaCLIP against SOTA ZSAD methods, emphasizing its use of static and dynamic prompts for both CLIP encoders and detailing the comparison implementations, backbones, prompts, and training settings.

  • 3 Comparison Method Details: AdaCLIP is compared with SOTA methods, especially AnomalyGPT and AnomalyCLIP, while uniquely developing static and dynamic prompts for both text and image encoders.AnomalyGPT uses static prompts but lacks ZSAD capability, whereas AnomalyCLIP adds static prompts only to CLIP’s text encoder.
  • 3 Comparison Method Details: SAA uses Grounding DINO and SAM without training, while WinCLIP uses manual anomaly prompts and window scaling for segmentation.Official prompts are used for MVTec AD and VisA with SAA, default prompts elsewhere, and WinCLIP follows its original text prompts.
  • 3 Comparison Method Details: APRIL-GAN [6] follows its official implementation and jointly trains on industrial and medical auxiliary datasets, while DINOV2 is adapted with ViT-S/14 and multi-hierarchy patch embeddings.DINOV2 uses learnable projection layers, the same training set as AdaCLIP, and patch embeddings from the 3rd, 6th, 9th, and 12th layers.
  • 3 Comparison Method Details: SAM [9] is repurposed for ZSAD using its ViT-L backbone, auxiliary-data training, and trainable projections on patch embeddings from the 6th, 12th, 18th, and 24th layers.SAM’s prompting encoder and mask decoder are discarded for patch-embedding extraction.

4 Comparison with AnomalyCLIP

The section evaluates AdaCLIP within AnomalyCLIP’s framework for fair comparison and examines how learnable versus frozen projection layers affect the method.

  • Comparison setup: AdaCLIP is evaluated within AnomalyCLIP’s framework because their experimental settings differ and AnomalyCLIP’s code was unavailable before submission.AnomalyCLIP introduces learnable object-agnostic prompts based on generic normality and abnormality across categories.
  • Projection layers: Replacing AdaCLIP’s learnable projection layers with CLIP’s frozen pre-trained projection layers produces significantly different comparison results in Table 5.AnomalyCLIP uses frozen CLIP projection layers, whereas AdaCLIP introduces learnable projection layers; the supplied passage does not specify the direction of the performance difference.

5 Comparison with SOTA Full-shot Methods

AdaCLIP achieves comparable anomaly detection and localization performance to SOTA full-shot methods PatchCore [13] and CDO across seven public industrial datasets, outperforming them on some datasets despite training only on auxiliary data.

  • Comparison with SOTA Full-shot Methods: AdaCLIP achieves comparable anomaly detection and localization performance to PatchCore [13] and CDO, outperforming them on some datasets.Experiments use seven public industrial datasets because some datasets lack normal training data.
  • Comparison with SOTA Full-shot Methods: Training on auxiliary data enables AdaCLIP to detect anomalies effectively in unseen categories.The comparison demonstrates AdaCLIP’s effectiveness despite the absence of category-specific normal training data.

6 Category-Level Quantitative Results

This section presents detailed category-level quantitative results for datasets containing multiple categories, reported in Tables 9–14.

  • 6 Category-Level Quantitative Results: Datasets with multiple categories are evaluated at the category level, with detailed quantitative results reported in Tables 9–14.

7 Additional Qualitative Results

AdaCLIP is qualitatively evaluated across industrial and medical anomaly-detection datasets using anomaly-map visualizations and additional method comparisons. The failure cases show that anomalies without structural deviations remain challenging, especially when abnormality depends on contextual norms.

  • Additional Comparisons: The section also reports comparisons with AnomalyCLIP [16] under AnomalyCLIP’s experimental setting and with full-shot unsupervised AD methods PatchCore and CDO.Tables 7 and 8 identify best and second-best performance, but the supplied passages do not provide their numerical outcomes.
  • Failure Cases: AdaCLIP may fail to detect anomalies lacking structural deviations when abnormality arises from departures from contextual norms, such as transistor positioning.Figure 3 illustrates three categories of such failures using input images, ground truth, and AdaCLIP anomaly maps.
  • Industrial Domain: Additional qualitative results cover industrial datasets from MVTec AD, VisA, MPDD, BTAD, DAGM, and DTD-Synthetic, with anomaly-map visualizations spanning Figures 4–23.The section explicitly provides further industrial-domain qualitative results and visualizes representative categories across these datasets.
  • Medical Domain: Additional medical-domain qualitative results span the Clinicdb, Colondb, and ISIC datasets in Figures 24–26.Each visualization presents input images, ground truth, and AdaCLIP-generated anomaly maps.
Loading 2407.15795v1…