PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-AD: A Language-Guided Staged Dual-Path Model for Zero-shot Anomaly Detection

Xuhai Chen, Jiangning Zhang, Guanzhong Tian, Haoyang He, Wuhao Zhang, Yabiao Wang, Chengjie Wang, Yong Liu

arXiv:2311.00453v2cs.CV

TL;DR

Zero-shot anomaly detection avoids relying on reference images, while prior approaches often train separate models for object categories. CLIP-AD combines representative text-vector selection with staged segmentation paths, and its SDP and SDP+ variants outperform WinCLIP on MVTec-AD segmentation metrics.

  • Problem

    Collecting training samples for the varied anomalies is difficult, and training a separate model for each object category is not cost-effective.

    Source: §1 · p. 1

  • Method

    CLIP-AD uses distributional Representative Vector Selection for text features, a staged dual-path model for segmentation, and fine-tuned linear layers in SDP+ to align image features.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    On MVTec-AD, SDP improves F1-max/PRO over WinCLIP by +4.2/+10.7, while SDP+ improves them by +8.3/+20.5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CLIP-AD is presented as a zero-shot framework requiring no pre- or post-processing, with fine-tuning available to enhance performance.

    Source: §1 · p. 2

Abstract

from arXiv · show

This paper considers zero-shot Anomaly Detection (AD), performing AD without reference images of the test objects. We propose a framework called CLIP-AD to leverage the zero-shot capabilities of the large vision-language model CLIP. Firstly, we reinterpret the text prompts design from a distributional perspective and propose a Representative Vector Selection (RVS) paradigm to obtain improved text features. Secondly, we note opposite predictions and irrelevant highlights in the direct computation of the anomaly maps. To address these issues, we introduce a Staged Dual-Path model (SDP) that leverages features from various levels and applies architecture and feature surgery. Lastly, delving deeply into the two phenomena, we point out that the image and text features are not aligned in the joint embedding space. Thus, we introduce a fine-tuning strategy by adding linear layers and construct an extended model SDP+, further enhancing the performance. Abundant experiments demonstrate the effectiveness of our approach, e.g., on MVTec-AD, SDP outperforms the SOTA WinCLIP by +4.2/+10.7 in segmentation metrics F1-max/PRO, while SDP+ achieves +8.3/+20.5 improvements.

1. Introduction

The introduction motivates zero-shot anomaly detection as an alternative to costly per-object training and presents CLIP-AD’s prompt-selection and segmentation methods. It reports benchmark gains for SDP and SDP+ over WinCLIP.

1. Introduction

  • Anomaly variety makes comprehensive training data difficult to collect, while training and deploying a separate model for each object category is not cost-effective.Source: §1 · p. 1
  • CLIP-AD is presented as a zero-shot framework that requires no pre- or post-processing and can be enhanced through fine-tuning.Source: §1 · p. 2
  • RVS treats prompt design from a distributional perspective, selecting representative text vectors rather than assuming more descriptions are better.The paper presents diverse representative-vector selection methods as a direction beyond adjective crafting.Source: §1 · p. 2
  • Directly comparing text features with image feature maps can produce opposite predictions and irrelevant highlights; SDP adapts architecture and feature surgery to address them without fine-tuning.Source: §1 · p. 2
  • The paper attributes these segmentation issues to image-feature and text-feature misalignment, and adds fine-tuned linear layers in SDP+ to map image features into the joint embedding space.The linear-layer variant is the fine-tuned model.Source: §1 · p. 2
  • On MVTec-AD, SDP improves pixel-level AUROC/F1-max/PRO over WinCLIP by +2.4/+4.2/+10.7, and SDP+ by +6.1/+8.3/+20.5.Source: §1 · p. 2

2. Related Works

Related work covers anomaly detection settings from per-object unsupervised training to zero-shot methods, alongside CLIP-based approaches to segmentation. It notes efficiency and prompting or post-processing demands in prior methods.

2.1. Anomaly Detection

  • Prior anomaly-detection work includes per-object unsupervised models, multi-class detection, and few-shot methods that use a few normal images for new categories.Source: §2.1 · p. 2
  • WinCLIP brings language-guided CLIP to zero-shot anomaly detection, but its window-based segmentation requires multiple encodings of each image.Source: §2.1 · p. 2

2.2. Vision-Language Models

  • CLIP aligns image and language representations with separate encoders and can be transferred to classification through prompting.Source: §2.2 · pp. 2–3
  • CLIP Surgery uses architecture and feature surgeries to address opposite visualization and noisy activation from direct image-text comparison, without fine-tuning.Source: §2.2 · p. 3

3. Methodology of CLIP-AD

CLIP-AD combines distribution-based text-prompt design for classification with staged dual-path segmentation, then adds fine-tuned linear layers to address image–text feature misalignment.

3. Methodology of CLIP-AD

  • CLIP-AD uses RVS for text-prompt design, SDP for zero-shot segmentation, and SDP+ with fine-tuned linear layers for further performance gains.The framework follows CLIP's pipeline for anomaly classification.Source: §3 · p. 3

3.1. Text prompts design based on RVS

  • RVS treats normal and abnormal text descriptions as samples from separate distributions and selects representative vectors for cosine-similarity classification.Direct averaging recovers CPE, while the paradigm allows other representative-vector methods; DBSCAN is one explored option that uses the mean of the largest cluster to reduce sampling outliers.Source: §3.1 · p. 3

3.2. Zero-Shot AD without Fine-tuning

  • Directly comparing text features with image feature maps can produce predictions opposite to the ground truth and noisy high-scoring points.The authors report observing these phenomena across backbones.Source: §3.2 · pp. 3–4
  • SDP uses value-based attention and removes the feedforward network in surgery layers to address opposite predictions.The attention uses V-V similarity to retain token features with less influence from other tokens.Source: §3.2 · p. 4
  • SDP applies feature surgery to address irrelevant highlights, then sums anomaly maps from staged dual-path blocks using features from multiple ViT levels.Each stage pairs original ViT layers with a symmetric surgery pathway.Source: §3.2 · pp. 4–5, Figure 4 · p. 4

3.3. Zero-Shot AD with Fine-tuning

  • CLIP's class token receives language supervision, but its image feature maps do not, leaving those maps misaligned with text features.The authors identify this misalignment as the reason direct comparison for anomaly maps is unfeasible.Source: §3.3 · p. 5
  • SDP+ maps each block's ViT features with a fine-tuned linear layer, compares them with text features stage by stage, and sums the resulting anomaly maps.The method also combines SDP's output; CLIP parameters are frozen while the added linear layers are trained with focal and dice losses.Source: §3.3 · p. 5

4. Experiments

The experiments compare SDP and SDP+ with zero-shot AD methods on industrial and medical datasets, then ablate representative-vector selection, block outputs, and SDP+ mapping complexity. Results favor the proposed models across datasets and show that all-block outputs and a simple linear mapping are useful configurations.

4.1. Experimental Setups

  • The evaluation covers two industrial and four medical datasets, with cross-training between industrial datasets for SDP+ to preserve the zero-shot principle.Metrics include AUROC, AP, F1-max, and PRO for segmentation; the default CLIP backbone is ViT-B/16+ at resolution 240.Source: §4.1 · p. 5

4.2. Comparison with State-of-the-Arts

  • On four medical datasets, SDP and SDP+ outperform other methods; untrained SDP also beats trained AnomalyCLIP on HeadCT, BrainMRI, and ISIC.The medical evaluation applies the Tab. 1 model without reselection or retraining; HeadCT and BrainMRI are classification-only, while ISIC and CVC-ClinicDB are segmentation-only.Source: §4.1 · p. 5, Table 2 · p. 6, Table 3 · p. 6
  • Across MVTec-AD and VisA, SDP performs best overall in segmentation on MVTec-AD, while SDP+ surpasses comparison methods on all metrics for both tasks there and leads on VisA.On MVTec-AD segmentation, SDP's F1-max is slightly below SAA+'s; both SDP and SDP+ outperform WinCLIP by a large margin in segmentation on VisA.Source: Table 1 · p. 6, §4.2 · p. 6
  • Figure 5 compares qualitative examples across six datasets; SDP+ predictions are described as more precise and clean than SDP's.The authors report that SAA+ struggles when normal and abnormal regions lack clear boundaries, including for bottle, cable, and transistor.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3. Ablation Study

  • 0.7 AUROC and 0.9 AP and PRO improvements in anomaly segmentation over the traditional mean are reported for DBSCAN representative vectors.The RVS contribution is a distributional framework for text-prompt design, not a claim that one representative-vector calculation method is universally best.Source: Table 4 · p. 7, §4.3 · p. 7
  • For SDP, deeper blocks and combining block outputs improve performance overall; the best results generally use all blocks.Different levels work better for different categories: the first three blocks fail to provide meaningful hazelnut results, while the second and third blocks work well for carpet.Source: §4.3 · p. 7, Table 5 · p. 8
  • For SDP+, using outputs from all blocks gives the best results, with the added linear layers reducing disparities among blocks.The authors report that satisfactory anomaly position and coverage are obtained only when all blocks are used together.Source: Table 5 · p. 8, §4.3 · p. 8
  • More complex SDP+ mappings perform worse at the standard learning rate, while lowering it tenfold improves results; a single linear layer performs better than the more complex alternatives.The authors attribute the complex mappings' initial poor test performance to overfitting and caution that training-loss convergence need not improve test results.Source: Table 6 · p. 8, §4.3 · p. 8

5. Conclusion

The conclusion presents CLIP-AD as a zero-shot AD framework combining RVS, SDP, and SDP+. It reports state-of-the-art performance on six real-world datasets and notes limits in representative-vector selection and cross-dataset training.

5. Conclusion

  • CLIP-AD combines distribution-based Representative Vector Selection, staged dual-path surgery, and fine-tuned linear layers to address text-feature quality, anomaly-map issues, and feature misalignment.The authors describe SDP as operating without fine-tuning and SDP+ as an extension that fine-tunes a few linear layers.Source: §5 · p. 8
  • The authors report state-of-the-art performance across six real-world datasets.Source: §5 · p. 8
  • The authors say RVS may be underexplored and representative-vector selection remains simple; cross-dataset training may overfit, which an extra validation set could mitigate.Source: §5 · p. 8

Supplementary Material

The supplement adds detail on RVS, analyzes CLIP feature misalignment, reports results across backbones, discusses WinCLIP multi-crop prediction, and provides per-category results for two datasets.

Supplementary Material

A. More Details about RVS

RVS treats normal and abnormal text prompts as distributions and uses representative vectors to summarize them. The section details prompt construction and several ways to calculate those vectors, while clarifying that the paradigm—not a particular calculation method—is the contribution.

A. More Details about RVS

  • Prompts combine WinCLIP templates such as “a photo of a [c]” with state prompts containing an object name; medical datasets use adapted state lists.For HeadCT, BrainMRI, and CVC-ClinicDB, the state lists remove “scratch” and “crack” and add “hemorrhage,” “tumor,” “polypus,” and “polyp,” respectively.Source: A. More Details about RVS · p. 12
  • Mean Vector averages the normal and abnormal prompt features into representative vectors, following the method used by WinCLIP.The prompt index i identifies different text prompts.Source: A. More Details about RVS · p. 12
  • PCA uses the first principal component as a representative direction, with its sign corrected using the mean vector because PCA vectors can be reversed.The first component captures the distribution’s most significant direction.Source: A. More Details about RVS · p. 12
  • KDE weights prompt vectors by estimated density so high-density samples contribute more and inappropriate outliers have less impact.The method uses a Gaussian kernel with bandwidth 0.3.Source: A. More Details about RVS · pp. 12–13
  • Mean Shift and DBSCAN represent a distribution with the mean of its largest cluster to mitigate outlier effects.Mean Shift uses bandwidth 2; DBSCAN uses an epsilon of 0.5 and 15 minimum samples, adjusted for VisA to 1.5 and 25.Source: A. More Details about RVS · p. 13
  • The contribution is the distributional interpretation and RVS paradigm, not a specific representative-vector calculation method.Average or maximum cosine similarity are mentioned as possible distributional considerations but are outside the RVS discussion.Source: A. More Details about RVS · p. 13

B. Reasons for the Misalignment

CLIP’s training directly supervises class-token image embeddings, but not the patch tokens used as image feature maps. The paper links this misalignment with text features to unreasonable anomaly maps and lower segmentation and classification metrics, and reports that SDP+ addresses it.

B. Reasons for the Misalignment

  • Only CLIP’s class-token image embeddings are supervised by its similarity-based training objective; patch tokens lack direct supervision and therefore are not aligned with text features in the joint embedding space.The objective maximizes similarity for real image-text pairs and minimizes it for incorrect pairs using symmetric cross-entropy.Source: B. Reasons for the Misalignment · pp. 13–14
  • Misaligned feature maps produce opposite predictions and irrelevant highlights, yielding unreasonable anomaly maps and low segmentation metrics.Because the classification anomaly score is related to the maps’ maximum value, the paper also reports a noticeable decrease in classification metrics.Source: B. Reasons for the Misalignment · p. 14
  • The analysis and experiments are reported to demonstrate misalignment’s presence and detrimental effects, as well as SDP+’s rationale and effectiveness.Source: B. Reasons for the Misalignment · p. 14

C. Using Different CLIP Backbones

Across the tested CLIP backbones, ViTs outperform ResNets on anomaly classification and segmentation. The reported SDP gap is attributed to limited surgery with ResNets, while fine-tuning significantly reduces the gap for SDP+; larger models tend to do better, but not consistently across metrics.

C. Using Different CLIP Backbones

  • ViTs outperform ResNets in both anomaly classification and segmentation across the tested CLIP backbones.The comparison is on MVTec-AD; ViT-B/16+ is the default, while ViT-L/14+ is the large-scale 336-resolution ViT also used by AnomalyCLIP.Source: C. Using Different CLIP Backbones · p. 14, Table 8 · p. 14
  • The SDP performance gap is attributed to limited surgery with ResNets, whereas fine-tuning significantly reduces the gap for SDP+.With ResNets, SDP surgery applies only to the final attention pooling layer; SDP+ adds linear layers to outputs of different blocks.Source: C. Using Different CLIP Backbones · p. 14
  • Larger models tend to perform better, but this trend is not consistent across metrics.The paper notes that a different ResNet might surpass ViTs with SDP+, but says this is beyond its scope.Source: C. Using Different CLIP Backbones · p. 14

D. Multi-crop Prediction

Multi-crop prediction improves classification metrics over WinCLIP in the reported experiment, but it requires repeated image encoding. The authors therefore do not use it in the final SDP/SDP+ models.

D. Multi-crop Prediction

  • Image-level AUROC 92.8, F1-max 93.1, and AP 96.6 are reported after multi-crop prediction, respectively 1.0↑, 0.2↑, and 0.1↑ higher than WinCLIP.For a fair comparison, the maximum anomaly-map value is excluded; the passage attributes the improvement to RVS.Source: D. Multi-crop Prediction · p. 14
  • The classification procedure averages predictions from five 240 x 240 patches and their horizontal reflections, then averages that result with the original anomaly score.The image is first resized to 270 x 270; the patches are the four corners and center.Source: D. Multi-crop Prediction · p. 14
  • Multi-crop prediction is omitted from final SDP/SDP+ because it requires encoding the image multiple times, increasing runtime tenfold.Source: D. Multi-crop Prediction · p. 15

E. Detailed Quantitative Results

The section reports detailed category-level evaluation results for MVTec-AD and VisA, with SDP+ further enhancing performance over SDP across various categories.

E. Detailed Quantitative Results

Loading 2311.00453v2…