PaperlayerView exact revision on arXiv

Source-linked AI summary

WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation

Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, Onkar Dabeer

arXiv:2303.14814v1cs.CVcs.AIcs.CL

TL;DR

Industrial anomaly inspection needs classification and segmentation methods that do not rely on bespoke task-specific training, especially when anomalies or normal references are scarce. The paper proposes WinCLIP, which combines compositional language prompts with aligned multi-scale features, and WinCLIP+, which adds normal-image references. On MVTec-AD and VisA, both methods outperform state-of-the-art methods in zero- and few-normal-shot settings.

  • Problem

    Industrial inspection commonly relies on task-specific images and annotation, while rare defects and limited normal data make scalable zero- and few-normal-shot anomaly recognition important.

  • Method

    WinCLIP combines compositional state-word and prompt-template ensembles with aligned multi-scale window, patch, and image features; WinCLIP+ adds reference association from normal images.

  • Results

    WinCLIP and WinCLIP+ outperform state-of-the-art methods by large margins on zero- and few-shot anomaly classification and segmentation benchmarks.

  • Takeaways & Limitations

    Language and complementary normal-image references support anomaly recognition across zero-shot and few-normal-shot classification and segmentation settings.

  • Takeaways & Limitations

    The one-class setup assumes an available list of task-specific text tags for objects and relevant defects.

Abstract

from arXiv · show

Visual anomaly classification and segmentation are vital for automating industrial quality inspection. The focus of prior research in the field has been on training custom models for each quality inspection task, which requires task-specific images and annotation. In this paper we move away from this regime, addressing zero-shot and few-normal-shot anomaly classification and segmentation. Recently CLIP, a vision-language model, has shown revolutionary generality with competitive zero-/few-shot performance in comparison to full-supervision. But CLIP falls short on anomaly classification and segmentation tasks. Hence, we propose window-based CLIP (WinCLIP) with (1) a compositional ensemble on state words and prompt templates and (2) efficient extraction and aggregation of window/patch/image-level features aligned with text. We also propose its few-normal-shot extension WinCLIP+, which uses complementary information from normal images. In MVTec-AD (and VisA), without further tuning, WinCLIP achieves 91.8%/85.1% (78.1%/79.6%) AUROC in zero-shot anomaly classification and segmentation while WinCLIP+ does 93.1%/95.2% (83.8%/96.4%) in 1-normal-shot, surpassing state-of-the-art by large margins.

1. Introduction

The paper targets scalable industrial anomaly classification and segmentation when anomalous examples, normal images, or task-specific supervision are limited. It proposes WinCLIP and WinCLIP+ to combine language, multi-scale visual features, and normal reference images for zero- and few-normal-shot inspection.

  • Motivation: Industrial anomaly classification identifies defective images, while segmentation localizes defects at the pixel level across diverse manufacturing domains.Defects vary widely in color, texture, and size, creating a long-tail inspection problem.
  • Motivation: Rare and varied defects limit representative anomaly samples, leaving few-normal-shot performance below the fully supervised upper bound.Existing one-class methods generally fit models to normal images and treat deviations as anomalous.
  • Motivation: Task-specific bespoke models are not scalable, motivating zero-shot and few-normal-shot anomaly recognition with only limited normal images.Some defects, such as missing components, are most naturally defined relative to normal reference images.
  • Approach: WinCLIP combines compositional state-word prompts with efficient multi-scale spatial features aligned to language for zero-shot classification and segmentation.It addresses weak dense language alignment and aggregates window-, patch-, and image-level information.
  • Approach: WinCLIP+ combines language-guided WinCLIP with visual reference association from a few normal images for few-normal-shot anomaly recognition.The few-normal-only setup uses no segmentation annotations.
  • Results: WinCLIP and WinCLIP+ outperform state-of-the-art methods by large margins on zero- and few-shot anomaly classification and segmentation benchmarks.The experiments cover MVTec-AD and VisA without tuning individual cases.

2. Related work

Related work centers on large pretrained vision-language models and one-class or few-shot anomaly methods. The paper positions its approach at their intersection, addressing scalable anomaly recognition with limited normal data.

  • Vision-language modeling: CLIP established web-scale image-text pretraining and broad zero-shot transfer, while later vision-language models extended scaling, efficiency, multitask learning, and dense prediction.Prompt engineering and CLIP-based language-guided detection and segmentation can improve downstream generalization.
  • Anomaly classification and segmentation: Anomaly research has mainly used one-class methods trained with many normal images because anomalous samples are scarce.Recent few-shot approaches augment small support sets, while full-normal-shot setups remain difficult to scale across tasks.
  • State classification: State classification predicts fine-grained physical properties or attributes, providing a conceptual relation to distinguishing normal and anomalous object states.Prior computer-vision work commonly studies object, scene, or material recognition and generic attributes.

3. Background

The paper formalizes anomaly classification as image-level abnormality prediction and anomaly segmentation as its pixel-level extension. It adopts a one-class protocol and uses CLIP’s text-image alignment for zero-shot inference.

  • Anomaly classification and segmentation: Anomaly classification predicts whether an image is normal or anomalous, whereas anomaly segmentation predicts anomaly locations over the image grid.Classification can be expressed through an anomaly score thresholded into binary predictions.
  • One-class setting: The one-class protocol uses only normal training samples and focuses on zero-shot and few-shot settings with K = 1 to 4.The setup assumes task-specific text tags for objects and relevant defects.
  • CLIP background: CLIP learns joint vision-language representations by contrastive training on large-scale image-text pairs and classifies inputs by text-image similarity.Its zero-shot classifier produces a temperature-scaled categorical distribution over candidate texts.
  • Prompting: Prompt templates and ensembles of prompt embeddings improve label-conditioned classification by retrieving visual knowledge through CLIP’s language interface.The paper explores prompt constructions better suited to anomaly recognition.

4. WinCLIP and WinCLIP+

WinCLIP builds zero-shot anomaly classification and segmentation from compositional language prompts and multi-scale CLIP visual features. WinCLIP+ adds normal-reference associations to complement language-guided predictions for few-normal-shot recognition.

  • WinCLIP: WinCLIP establishes a binary zero-shot framework using normal and anomalous class prompts, optionally with object-level labels.The object label can be a known object such as “bottle” or the generic “object”.
  • WinCLIP: Compositional prompt ensembles combine state words and templates to define normal and anomalous states more precisely.State words may be common across objects or task-specific, such as “bad soldering” for PCBs.
  • WinCLIP: WinCLIP extracts multi-scale window embeddings from masked image regions and applies language-aligned anomaly scoring spatially for segmentation.Sliding windows produce embeddings from active image areas, with harmonic aggregation combining overlapping-window scores.
  • WinCLIP: Window features improve dense language alignment and local-detail capture over penultimate patch features, while remaining efficiently computable with ViT.Penultimate patch features are not directly language-supervised and may retain excessive global context through self-attention.
  • WinCLIP: Multi-scale aggregation balances local details and global information using small-, mid-, and image-scale features for defects of varying sizes.The cited scales include 2 × 2 patches, 3 × 3 patches, and the ViT class-token image-scale feature.
  • WinCLIP+: WinCLIP+ incorporates few normal reference images through memory-based association at patch, small-window, and mid-window scales, then combines visual and language-guided predictions.Separate reference memories are built from WinCLIP and penultimate features, and multi-scale visual predictions are fused with language-guided scores.

5. Experiments

Experiments evaluate WinCLIP and WinCLIP+ for low-shot anomaly classification and segmentation on MVTec-AD and VisA, alongside ablations and comparisons with prior methods. Results show strong performance from language-guided, multi-scale local features and complementary normal-image references.

  • Evaluation setup: Experiments use MVTec-AD and VisA, reporting classification AUROC, AUPR, F1-max, and segmentation pAUROC, PRO, and F1-max.The study uses public pretrained OpenCLIP models and evaluates zero- and few-normal-shot settings.
  • Anomaly classification: WinCLIP significantly improves zero-shot anomaly classification over naïve CLIP adaptations on both MVTec-AD and VisA.The comparisons include original CLIP-AC prompts and an ImageNet-engineered prompt ensemble.
  • Anomaly classification: WinCLIP+ outperforms prior few-normal-shot classification methods across all metrics on both benchmarks, improving over PatchCore by 9.7% on 1-shot MVTec-AD and 5.3% on 1-shot VisA.WinCLIP+ with 1/2/4 normal shots also exceeds zero-shot WinCLIP, showing the value of reference normal images.
  • Anomaly segmentation: WinCLIP outperforms adapted Trans-MM and MaskCLIP by a wide margin for zero- and few-normal-shot anomaly segmentation on both benchmarks.The comparison indicates that generic CLIP adaptations are weaker than WinCLIP for this task.
  • Anomaly segmentation: WinCLIP+ produces masks more concentrated on ground truth than prior methods across qualitative 1-shot examples, while 1/2/4-shot results exceed zero-shot WinCLIP.These results support complementary benefits from language prompts and visual reference images.
  • Ablation study: Ablations identify anomalous-state supervision, diverse state and prompt texts, and multi-scale harmonic aggregation as useful components, with state ensembles especially influential.Window-based local features improve performance while avoiding the computational overhead of dense image tiling; local window information also complements image-level predictions.
  • Ablation study: Task-specific state words improve zero-shot VisA classification by 0.8% average AUROC, including a 5.3% gain on challenging PCB2.Examples include “missing part” for PCB and “burnt” for pipe fryum.

6. Conclusion

The paper concludes that WinCLIP defines normality and anomaly through complementary language and visual reference information. It reports strong zero- and few-shot performance while identifying industrial-domain vision-language pretraining as future work.

  • Contributions: WinCLIP combines fine-grained textual definitions with normal reference images for anomaly classification and segmentation.WinCLIP uses compositional prompt ensembles and efficiently aggregates aligned multi-scale window and image features, while WinCLIP+ adds normal-image association.
  • Results: WinCLIP and WinCLIP+ outperform state-of-the-art methods by considerable margins in zero- and few-shot benchmarks.The conclusion frames the methods as complementary to standard one-class approaches.
  • Future work: Vision-language pretraining with industrial-domain data is identified as a promising direction for further improvement.This direction is left as future work.

Supplementary Material

The supplementary material contains the paper title only.

  • The supplementary passage provides the title “WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation.”

A. Experimental details

The experiments specify prompt construction, CLIP preprocessing and backbone comparisons, while evaluating both performance and segmentation limitations.

  • Prompt construction: WinCLIP composes template-level prompts with normal or anomaly state-level prompts, each incorporating an object-level label.The prompt ensemble uses two levels: state and template.
  • Data preprocessing: OpenCLIP preprocessing standardizes RGB images after normalization and applies bicubic resizing.The pipeline is applied to both MVTec-AD and VisA.
  • Evaluation metrics: AUROC can overstate performance on imbalanced segmentation data, so F1-max is also used to assess anomaly prediction across thresholds.The authors note that low-shot segmentation remains unresolved despite high pixel-AUROC.
  • Backbone evaluation: The ViT-B/16+ architecture increases image and text embedding dimensions relative to ViT-B/16.The reported dimensions increase from 768 to 896 for image embeddings and from 512 to 640 for text embeddings.
  • Backbone evaluation: WinCLIP experiments compare zero-shot AUROC and pAUROC across different CLIP backbone architectures on MVTec-AD.The comparison covers anomaly classification and segmentation.

B. Additional results on ablation study

Additional ablations examine CLIP-backed PatchCore and backbone architecture, finding advantages for ViT-based, larger, or higher-resolution models.

  • CLIP-backed PatchCore: PatchCore variants are tested with patch-token features from the CLIP-based ViT-B/16+ backbone.The comparison evaluates whether a CLIP backbone benefits PatchCore.
  • CLIP backbone ablation: ViT-based models generally outperform ResNets in both anomaly classification and segmentation.The segmentation gap may relate to larger ResNet patch sizes producing blurrier outputs.
  • CLIP backbone ablation: Larger models or higher resolutions improve performance for both tested architecture families.The reported benefit applies to both anomaly classification and segmentation.

C. Additional qualitative results

The paper provides qualitative segmentation examples across MVTec-AD and VisA for zero-shot WinCLIP and few-shot WinCLIP+, alongside curated zero-shot failure cases.

  • Qualitative results: Figures 7–10 present additional anomaly-segmentation results for WinCLIP and WinCLIP+ on MVTec-AD and VisA.Figures 7 and 9 cover zero-shot WinCLIP, while Figures 8 and 10 cover four-shot WinCLIP+.
  • Failure cases: Figure 11 curates failure cases from zero-shot WinCLIP.The examples are drawn from both MVTec-AD and VisA.
  • Failure cases: Logical anomalies often require normal reference images, and the paper reports that WinCLIP+ alleviates these issues.Examples include misplaced, missing, or bent components and missing text.
  • Failure cases: Tiny defects remain difficult, motivating future exploration of spatial features with more local detail.The cited examples span carpet, wood, capsule, and macaroni images.
  • Failure cases: Irrelevant deviations from normality can be mistaken for defects of interest.Examples include benign dots, extra ingredients, designed holes, and acceptable scratches.

D. Detailed quantitative results

The supplementary material reports subset-level quantitative results for anomaly classification and segmentation across MVTec-AD and VisA using multiple evaluation metrics.

  • Dataset-wise results: Subset-level results are reported for MVTec-AD in Tables 10–15 and for VisA in Tables 16–21.The tables provide detailed performance values corresponding to the main-text evaluations.
  • MVTec-AD: MVTec-AD anomaly-classification results include class-wise AUROC, AUPR, and F1-max.These metrics are reported in Tables 10–12.
  • MVTec-AD: MVTec-AD anomaly-segmentation results include class-wise pixel-AUROC, PRO, and F1-max.These metrics are reported in Tables 13–15.
  • VisA: VisA anomaly-classification results include class-wise AUROC, AUPR, and F1-max.These metrics are reported in Tables 16–18.
  • VisA: VisA anomaly-segmentation results include class-wise pixel-AUROC, PRO, and F1-max.These metrics are reported in Tables 19–21.
Loading 2303.14814v1…