Source-linked AI summary
AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIP
Wenxin Ma, Xu Zhang, Qingsong Yao, Fenghe Tang, Chenxu Wu, Yingtai Li, Rui Yan, Zihang Jiang, S. Kevin Zhou
TL;DR
CLIP generalizes well for zero-shot anomaly detection but has difficulty separating normal and abnormal semantics. AA-CLIP uses two-stage adaptation with anomaly-aware text anchors and patch-level visual alignment, and reports state-of-the-art cross-dataset zero-shot results with limited training data. The method preserves CLIP’s generalization through residual adapters, although full-shot training shows signs of overfitting.
Problem
CLIP’s normal and abnormal text embeddings can remain highly similar because general non-anomalous training lacks defect-detection guidance, making subtle or context-specific anomalies difficult to detect.
Method
AA-CLIP creates anomaly-aware text anchors and aligns patch-level visual features with them in two stages, using residual adapters for controlled adaptation.
Results
AA-CLIP achieves state-of-the-art results in cross-dataset zero-shot testing with 64-shot training per seen class across industrial and medical domains.
Takeaways & Limitations
AA-CLIP provides a resource-efficient approach for zero-shot anomaly detection while preserving CLIP’s generalization ability across seen and novel classes.
Takeaways & Limitations
Full-shot training shows signs of overfitting, suggesting possible saturation during CLIP adaptation and requiring further investigation.
Abstract
from arXiv · showhide
Anomaly detection (AD) identifies outliers for applications like defect and lesion detection. While CLIP shows promise for zero-shot AD tasks due to its strong generalization capabilities, its inherent Anomaly-Unawareness leads to limited discrimination between normal and abnormal features. To address this problem, we propose Anomaly-Aware CLIP (AA-CLIP), which enhances CLIP's anomaly discrimination ability in both text and visual spaces while preserving its generalization capability. AA-CLIP is achieved through a straightforward yet effective two-stage approach: it first creates anomaly-aware text anchors to differentiate normal and abnormal semantics clearly, then aligns patch-level visual features with these anchors for precise anomaly localization. This two-stage strategy, with the help of residual adapters, gradually adapts CLIP in a controlled manner, achieving effective AD while maintaining CLIP's class knowledge. Extensive experiments validate AA-CLIP as a resource-efficient solution for zero-shot AD tasks, achieving state-of-the-art results in industrial and medical applications. The code is available at https://github.com/Mwxinnn/AA-CLIP.
1. Introduction
Existing anomaly detection methods often require substantial labeled data and generalize poorly to novel or rare classes. AA-CLIP addresses CLIP’s difficulty distinguishing normal from abnormal semantics through two-stage adaptation, achieving efficient zero-shot anomaly detection.
- Motivation: Traditional anomaly detection frameworks often require sufficient class-specific labeled data, limiting generalization to novel and rare classes.Real-world anomaly detection frequently cannot obtain comprehensive labeled datasets.
- Motivation: CLIP offers strong zero-shot generalization, but its text and visual representations can overlap between normal and abnormal textures.This Anomaly-Unawareness problem reduces the precision of text-guided anomaly detection.
- Method: AA-CLIP first creates anomaly-aware text anchors, then aligns patch-level visual features with those anchors for focused anomaly localization.The two stages separately refine text semantics and visual anomaly-relevant regions.
- Method: Residual adapters provide controlled adaptation that enhances fine-grained anomaly detection while preserving CLIP’s pretrained class knowledge and generalization ability.Adapters are used in both stages to avoid sacrificing the original model’s capabilities.
- Results: 64-shot training per seen class yields state-of-the-art results in cross-dataset zero-shot testing across industrial and medical domains.The method is also reported to improve zero-shot anomaly detection in data-limited settings, including 2-shot training conditions.
2. Related Work
Prior work models normal data distributions or adapts CLIP for anomaly detection, but CLIP remains limited in fine-grained and spatial representations. AA-CLIP instead explicitly refines anomaly-discriminative text embeddings.
- Traditional Anomaly Detection: Traditional anomaly detection methods include reconstruction-based, augmentation-based, and discriminative approaches that depend heavily on modeling the normal data distribution.
- CLIP: CLIP uses large-scale image-text pretraining and contrastive learning to generalize effectively to new categories without task-specific training.
- CLIP: Despite strong generalization, CLIP often misses nuanced details and spatial relationships needed for precise boundaries and fine-grained feature extraction.
- CLIP-based Anomaly Detection: CLIP-based anomaly detection methods align visual features with anomaly-related text or adapt representations to bridge natural and anomaly-detection domains.
- CLIP-based Anomaly Detection: AA-CLIP differs by explicitly refining text embeddings to incorporate anomaly-discriminative representations rather than relying on anomaly-unaware prompts or unchanged feature spaces.
3. Method
AA-CLIP addresses CLIP’s anomaly-unawareness through controlled adaptation of text and visual spaces. Its two-stage pipeline separates normal and anomalous semantics, then aligns patch features with anomaly-aware text anchors for localization.
- Problem Formulation: Zero-shot AD targets normal and anomalous patterns across unseen classes and datasets, including image-level classification and pixel-level segmentation.Image-level AD predicts normal or anomalous labels, while segmentation identifies anomalous regions with masks.
- Current Challenges: CLIP’s normal and anomaly text embeddings can remain entangled, causing defective images to match normal prompts more strongly than anomaly prompts.This anomaly-unawareness is linked to CLIP’s limited guidance for defect detection and weak separation of subtle or context-specific anomalies.
- Two-Stage Training Strategy: AA-CLIP sequentially adapts semantic-rich text features and detail-focused visual features while freezing original CLIP parameters.The first stage uses shallow text residual adapters and a disentangle loss; the second aligns visual patch features with the resulting text anchors.
- Residual Adapter: Lightweight residual adapters in shallow text and vision layers control adaptation while preserving CLIP’s pretrained knowledge and generalization ability.A residual ratio hyper-parameter controls how much anomaly-specific knowledge is fused with the original features.
- Visual Alignment: Normal and anomaly prompts are averaged into text anchors, which are refined through image- and patch-level alignment with enhanced visual features.Multi-granularity visual features are projected to the text-anchor channel, aggregated, and compared by cosine similarity to produce patch-level prediction maps.
- Training Objective: The alignment objective combines binary-cross-entropy classification and segmentation losses, while a weighted disentangle regularizer encourages orthogonality between normal and anomaly anchors.The total objective is Ltotal = Lalign + γLdis, with Lalign = Lcls + Lseg.
- Results: AA-CLIP’s adapted text features disentangle normal and anomaly semantics while preserving original object-recognition capability and generalizing to novel classes.The reported t-SNE visualization shows intertwined original features becoming separated after adaptation.
4. Experiments
AA-CLIP is evaluated across industrial and medical benchmarks, data regimes, localization outputs, and training-strategy ablations. It maintains strong performance with limited data, improves anomaly localization, and benefits from residual adapters, disentangled text semantics, and two-stage training.
- Experimental Setup: The evaluation spans 11 industrial and medical benchmarks, including photography, endoscopy, CT, MRI, and OCT modalities.Industrial datasets include MVTec AD, VisA, BTAD, and MPDD; medical datasets include BMAD and colon polyp datasets.
- Experimental Setup: Training uses 2-shot, 16-shot, 64-shot, and full-shot settings with a consistent 1:1 ratio of normal and anomaly samples.Samples are randomly selected from each class, and VisA-trained results are obtained using MVTec-AD as the training dataset.
- Comparison with SOTA Methods: With the full dataset, AA-CLIP achieves a pixel-level AUROC of 93.4%, while 64 shots per class yields an image-level SOTA AUROC of 83.1%.The method is also competitive at the image level with only 2 shots per class and improves average pixel-level zero-shot performance in the 2-shot setting.
- Comparison with SOTA Methods: Across data levels, AA-CLIP generally outperforms other methods, remains competitive with limited data, and maintains its lead as training data increases.Other methods show signs of underfitting, whereas AA-CLIP establishes new pixel- and image-level SOTA results.
- Anomaly Localization: AA-CLIP produces fewer false negatives and more accurately highlights anomalous regions in both industrial and medical examples.The visualization compares original CLIP, AnomalyCLIP, VAND, and AA-CLIP.
- Ablation Analysis: Residual adapters mitigate the generalization decline caused by vanilla linear adapters, while disentangle loss further improves anomaly discrimination and image-level performance.The ablations also show that refining text space provides a more precise semantic foundation and that independence between normal and anomaly anchors is useful.
- Ablation Analysis: One-stage joint adaptation damages zero-shot generalization by exaggerating anomaly semantics and forgetting CLIP’s class information.On MVTec-AD, one-stage training obtains P-/I-AUROC 90.1(-1.8)/88.6(-1.9), while two-stage training enables controlled adaptation.
5. Conclusion and Discussion
AA-CLIP embeds anomaly-aware information into CLIP through two-stage training, disentangling anomaly representations across seen and novel classes while preserving generalization. The authors also report potential broader feature-space applications, but observe overfitting signs with full-shot training that warrant further investigation.
- AA-CLIP uses a two-stage training strategy to embed anomaly-aware information into CLIP and disentangle anomaly representations across seen and novel classes.
- Residual adapters help preserve CLIP’s strong generalization ability while supporting zero-shot performance across multiple datasets.
- AA-CLIP suggests that refining CLIP’s feature space may support improved performance in downstream applications and other unawareness issues.
- Full-shot training shows signs of overfitting and potential saturation during CLIP adaptation, warranting further investigation.