PaperlayerView exact revision on arXiv

Source-linked AI summary

WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation

Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, Onkar Dabeer

arXiv:2303.14814v1cs.CVcs.AIcs.CL

TL;DR

Industrial anomaly inspection spans diverse objects and defects, while few-normal-shot settings remain challenging for existing approaches. WinCLIP combines compositional prompts with multi-scale language-aligned features, and WinCLIP+ incorporates normal references; across MVTec-AD and VisA, the methods outperform state of the art in zero-/few-shot classification and segmentation.

  • Problem

    Anomaly inspection spans diverse objects and defects, while existing normal-image-based methods still have room to improve in few-normal-shot settings.

    Source: §1 · p. 1

  • Method

    WinCLIP combines compositional prompts for zero-shot classification with multi-scale language-aligned features for segmentation, while WinCLIP+ adds information from a few normal reference images.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    Across MVTec-AD and VisA, WinCLIP/WinCLIP+ outperform state-of-the-art methods in zero-/few-shot anomaly classification and segmentation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pretrained CLIP can support anomaly recognition, with normal-reference cues complementing language definitions in few-shot settings.

    Source: §6 · p. 8

    The framework assumes an available list of task-specific text tags, such as object and defect names.

    Source: §3 · p. 3

Abstract

from arXiv · show

Visual anomaly classification and segmentation are vital for automating industrial quality inspection. The focus of prior research in the field has been on training custom models for each quality inspection task, which requires task-specific images and annotation. In this paper we move away from this regime, addressing zero-shot and few-normal-shot anomaly classification and segmentation. Recently CLIP, a vision-language model, has shown revolutionary generality with competitive zero-/few-shot performance in comparison to full-supervision. But CLIP falls short on anomaly classification and segmentation tasks. Hence, we propose window-based CLIP (WinCLIP) with (1) a compositional ensemble on state words and prompt templates and (2) efficient extraction and aggregation of window/patch/image-level features aligned with text. We also propose its few-normal-shot extension WinCLIP+, which uses complementary information from normal images. In MVTec-AD (and VisA), without further tuning, WinCLIP achieves 91.8%/85.1% (78.1%/79.6%) AUROC in zero-shot anomaly classification and segmentation while WinCLIP+ does 93.1%/95.2% (83.8%/96.4%) in 1-normal-shot, surpassing state-of-the-art by large margins.

1 AWS AI Labs 2 KAIST · 1. Introduction

The paper addresses zero-shot and few-normal-shot anomaly classification and segmentation with CLIP-based methods designed for long-tail industrial inspection. It introduces WinCLIP and WinCLIP+ and reports that they outperform state-of-the-art methods on MVTec-AD and VisA.

1. Introduction

  • Anomaly inspection is a long-tail problem because defects are rare and vary widely across objects, domains, colors, textures, and sizes.Existing methods often fit models to normal images, but few-normal-shot performance remains a challenge relative to fully supervised methods.Source: §1 · p. 1
  • Bespoke models for each inspection task do not scale, motivating zero-shot methods and few-normal-shot methods when defects are defined relative to normal examples.With only a few normal images, segmentation supervision is unavailable, making localization especially challenging across inspection tasks.Source: §1 · p. 1
  • WinCLIP extracts and aggregates multi-scale spatial features aligned with language for zero-shot anomaly segmentation, addressing CLIP’s lack of dense visual-language alignment.The method uses window-based features; the zero-shot models require no tuning for individual cases.Source: §1 · p. 2
  • A compositional prompt ensemble improves zero-shot anomaly classification over naïve CLIP prompts by better describing normal and anomalous states.The paper motivates state-level language because what counts as normal or anomalous depends on context and defects can be distinguished from acceptable deviations.Source: §1 · p. 2
  • WinCLIP+ combines language-guided WinCLIP with visual cues from normal reference images for few-normal-shot anomaly recognition and segmentation.Its reference association method operates on multi-scale feature maps, and the few-normal-only setup uses no segmentation annotation.Source: §1 · p. 2
  • WinCLIP and WinCLIP+ outperform state-of-the-art methods in zero- and few-shot anomaly classification and segmentation on MVTec-AD and VisA.The introduction reports large margins but provides no benchmark values in these passages.Source: §1 · p. 2

2. Related work

Related work links anomaly inspection to CLIP’s general vision-language transfer [27], prior anomaly methods that often require many normal images, and fine-grained state classification.

Vision-language modeling

  • CLIP [27] pioneered web-scale image-text pretraining and demonstrated broad zero-shot generality, while later VLMs explored data scaling, efficient designs, and multitask pretraining [1] [18] [21] [22].Open resources LAION-5B [36] and OpenCLIP broaden access; other work explores CLIP transfer beyond classification [11], prompt engineering [27], and language-guided detection and segmentation [28].Source: §2 · p. 3

Anomaly classification and segmentation

  • Most anomaly work uses many normal images [7] [8] [20] [51]; although MVTec-AD is saturated [31] [48] [51], full-normal-shot requirements limit scalability.Few-shot work expands small normal support sets through augmentation [39], while RegAD [14] reuses object-agnostic registration pretraining; VisA adds a challenging benchmark and ViT has shown inspection potential [9] [25].Source: §2 · p. 3

State classification

  • State classification predicts whether an object is normal or anomalous, focusing on fine-grained physical properties or attributes across objects and scenes [34].Related work introduces generic state and attribute datasets [15] [23] and graphs linking attributes with objects, whose relationships are learned using graph neural networks [24] [26].Source: §2 · p. 3

3. Background

Anomaly classification detects whether an image is abnormal, while segmentation localizes abnormal pixels; the paper considers both in one-class zero- and few-shot settings and reviews CLIP’s text-guided classification.

Anomaly classification and segmentation

  • Anomaly classification predicts image-level abnormality, whereas anomaly segmentation extends the prediction to pixel-level anomaly locations.In practice, both tasks can be cast as predicting anomaly scores; classification can use a threshold to produce a binary result.Source: §3 · p. 3
  • The paper follows a one-class protocol with only normal training samples, focusing on K = 1 to 4 few-shot and zero-shot setups for both tasks.It assumes an available list of task-specific text tags, such as object and defect names.Source: §3 · p. 3

Zero-shot classification with CLIP

  • CLIP learns aligned image and text representations from web image-text pairs using contrastive learning, then classifies images by similarity to a closed set of texts [27].Its zero-shot predictions use a temperature-scaled categorical distribution over the text options.Source: §3 · p. 3
  • Prompt templates accompanying class words improve CLIP accuracy, and ensembles of multiple prompt embeddings can further boost performance [27].The paper explores extracting CLIP’s visual knowledge in ways more suitable for anomaly recognition.Source: §3 · p. 3

4. WinCLIP and WinCLIP+

The section builds anomaly scoring from compositional language prompts, extends CLIP with window-based features for zero-shot segmentation, and adds normal-image references in WinCLIP+.

4.1. Language-driven zero-shot AC

  • CLIP-AC compares “normal [o]” and “anomalous [o]” prompts, while its compositional prompt ensemble averages text embeddings across state-word and template combinations.The ensemble can include common or task-specific state words and anomaly-oriented templates; the paper reports that the two-class design significantly outperforms the one-class design.Source: §4.1 · p. 4

4.2. WinCLIP for zero-shot AS

  • WinCLIP extracts dense features from sliding windows and applies language-guided anomaly scoring spatially, using multi-scale features and harmonic aggregation.For ViT, its window features come from class tokens aligned with text and can be computed efficiently by dropping masked patches.Source: §4.2 · pp. 4–5
  • WinCLIP’s window features provide language-aligned local detail, whereas penultimate patch features yield poor dense predictions because they lack direct language supervision and retain global context.The paper also notes that this penultimate representation is useful in WinCLIP+ for defining normality from visual references.Source: §4.2 · pp. 4–5, §4.3 · p. 6

4.3. WinCLIP+ with few-normal-shots

  • WinCLIP+ combines language-guided scores with visual predictions from a few normal reference images for anomaly classification and segmentation.It builds reference memories from small- and mid-scale WinCLIP features and penultimate features, averages their segmentation predictions, and combines the maximum visual anomaly score with the zero-shot classification score.Source: §4.3 · pp. 5–6

5. Experiments

WinCLIP and WinCLIP+ are evaluated for anomaly classification and segmentation in low-shot settings on MVTec-AD and VisA. The experiments compare them with prior methods and ablate their components.

5. Experiments

  • Experiments on MVTec-AD and VisA evaluate industrial anomaly classification and segmentation; classification reports AUROC, AUPR, and F1-max, while segmentation reports pAUROC and PRO.The benchmarks include diverse object subsets, high-resolution images, and full pixel-level annotations. The implementation uses LAION-400M-based CLIP with ViT-B/16+ unless otherwise noted.Source: §5 · p. 6

5.1. Zero-/few-shot anomaly classification

  • WinCLIP+ improves over PatchCore by 9.7% on 1-shot MVTec-AD and 5.3% on 1-shot VisA, while zero-shot WinCLIP significantly improves over naïve CLIP adaptations.On MVTec-AD, zero-shot WinCLIP also outperforms few-shot versions of prior works; WinCLIP+ at 1/2/4 shots exceeds zero-shot WinCLIP.Source: §5.1 · p. 6, Table 1 · p. 7

5.3. Comparison with many-shot methods

  • On MVTec-AD, 4-shot WinCLIP+ is competitive with full-shot CutPaste, while zero-shot WinCLIP beats few-shot DifferNet and TDG and aggregated 4-shot RegAD.WinCLIP’s zero-shot result exceeds DifferNet and TDG even when they use more than 10 shots.Source: §5.3 · p. 7, Table 2 · p. 7

5.4. Ablation study

  • For classification ablations, anomalous-word supervision is crucial, state- and prompt-level diversity help, state ensembling contributes more, and multi-crop yields a minor improvement.The component-wise analysis is on MVTec-AD.Source: §5.4 · p. 7, Table 3 · p. 7
  • WinCLIP+ ablations report that ascore0 complements max MW for AC, even in higher-shot regimes where max MW can be better, while MW adds local-awareness to MP predictions for AS.Task-specific state words improve zero-shot VisA classification by 0.8% average AUROC, with a 5.3% gain on challenging PCB2.Source: §5.4 · pp. 7–8

5.2. Zero-/few-shot anomaly segmentation

  • WinCLIP outperforms Trans-MM and MaskCLIP in zero-shot segmentation, and WinCLIP+ outperforms prior localization methods across metrics on both benchmarks.Qualitative results show more concentrated 1-shot WinCLIP+ masks, and 1/2/4-shot WinCLIP+ performs better than zero-shot WinCLIP.Source: §5.2 · p. 6, Table 4 · p. 8, Figure 5 · p. 8

5.4. Ablation study

  • For segmentation, WinCLIP’s window-based local features enable accelerated inference with better performance than patch tokens or image tiling, and multi-scale features improve results.Segmentation benefits from image-level and middle/local context; image tiling incurs significant computational overhead, while patch tokens are not language-aligned.Source: §5.4 · p. 7, Table 8 · p. 8

6. Conclusion

WinCLIP combines fine-grained text definitions with normal reference images to define normality and anomalies, using CLIP for zero-shot recognition and segmentation and reference association for few-shot performance.

Conclusion

  • WinCLIP defines normality and anomalies through fine-grained textual definitions and normal reference images.The compositional prompt ensemble specifies normality and anomaly in text, distilling knowledge from pretrained CLIP, whose image-text alignment supports anomaly recognition.Source: §6 · p. 8
  • On recent benchmarks, WinCLIP and WinCLIP+ outperform state-of-the-art methods in zero- and few-shot setups by considerable margins.The conclusion gives no metric values for this comparison.Source: §6 · p. 8
  • For zero-shot segmentation, WinCLIP aggregates multi-scale, image-text-aligned features from windows and images.The conclusion attributes anomaly recognition to CLIP’s pretrained representation and alignment between texts and images.Source: §6 · p. 8
  • With a few normal samples, vision-based reference association complements language definitions and enables few-shot WinCLIP+.Source: §6 · p. 8
  • Industrial-domain vision-language pretraining is identified as a promising direction for future improvement.Source: §6 · p. 8

Supplementary Material · A. Experimental details

The supplementary material specifies WinCLIP’s prompt construction, preprocessing, evaluation metrics, and implementation choices. It also cautions that high pixel-AUROC does not mean low-shot anomaly segmentation is solved.

A. Experimental details

  • WinCLIP composes prompts by replacing [c] in a template-level prompt with a normal- or anomaly-state prompt, each using dataset-specific object-name words for [o].The prompt ensemble uses state-level and template-level prompts.Source: A. Experimental details · p. 12
  • CLIP-based models use OpenCLIP preprocessing: channel-wise standardization after RGB normalization, bicubic resizing to a 240-pixel shorter edge, and the same resize policy for baselines.The stated mean is [0.48145466, 0.4578275, 0.40821073] and standard deviation is [0.26862954, 0.26130258, 0.27577711].Source: A. Experimental details · p. 12
  • Despite >95% pixel-AUROC, WinCLIP+’s low-shot anomaly segmentation remains below 60% F1-max on both MVTec-AD and VisA; both methods outperform compared methods across reported metrics.The authors note that AUROC can overstate performance on imbalanced segmentation data, where normal pixels dominate.Source: A. Experimental details · p. 12
  • F1-max measures anomalous-sample precision and recall at the optimal threshold, while AUROC can inflate performance estimates on imbalanced anomaly-segmentation data.The authors cite Zou et al. in discussing this limitation.Source: A. Experimental details · p. 12
  • The primary ViT-B/16+ [16] modifies ViT-B/16 with larger image and text embeddings and 240^2 input resolution, increasing tokens from 196 to 225.Embedding dimensions increase from 768 to 896 for images and from 512 to 640 for text.Source: A. Experimental details · pp. 12–13
  • For non-square VisA images, WinCLIP uses overlapping square tiles of the shorter-edge size and averages tile predictions; baseline implementations include SPADE [7], PaDiM [8], and official PatchCore.Tile stride is at most 0.8 times the shorter edge, ensuring at least 0.2-edge overlap.Source: A. Experimental details · p. 13

B. Additional results on ablation study

The ablation results show that WinCLIP+ outperforms a CLIP-backed PatchCore variant, while ViT-based CLIP models generally outperform ResNets in zero-shot anomaly classification and segmentation.

Comparison with CLIP-based PatchCore

  • With CLIP ViT-B/16+ patch-token features, CLIP-based PatchCore variants achieve no better performance than original PatchCore [31], while WinCLIP+ significantly outperforms the last-layer variant, supporting its association-based module and WinCLIP features.The variants use CLIP ViT-B/16+ patch-token features from layers 6 and 9 (“hidden”) or the last layer (“last”); WinCLIP+ also uses last patch-token features.Source: B. Additional results on ablation study · p. 13

Effect of different CLIP backbones

  • Across zero-shot AC and AS, ViTs generally outperform ResNets; ResNet patch size 32 versus ViT 16 may blur AS outputs, while larger models or resolutions benefit both.For ResNets, WinCLIP applies window-based inference to the convolutional feature map before attention pooling.Source: B. Additional results on ablation study · pp. 13–14

C. Additional qualitative results

The section presents additional qualitative segmentation results for WinCLIP and WinCLIP+ on MVTec-AD and VisA, then examines four kinds of zero-shot WinCLIP failure. It notes that few-normal-shot references alleviate logical-anomaly failures, while other failure modes suggest areas for future improvement.

C. Additional qualitative results

D. Detailed quantitative results

This section reports subset-level performance for MVTec-AD and VisA across anomaly classification and segmentation metrics, with means and standard deviations over five random seeds.

MVTec-AD anomaly classification

MVTec-AD anomaly segmentation

VisA anomaly classification

VisA anomaly segmentation

Related papers

Loading 2303.14814v1…