PaperlayerView exact revision on arXiv

Source-linked AI summary

ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation

Shengze Li, Jianjian Cao, Peng Ye, Yuhan Ding, Chongjun Tu, Tao Chen

arXiv:2401.12665v2cs.CVcs.AI

TL;DR

CLIP-based ZSAS can struggle to segment local anomalies precisely, while SAM-based methods can produce redundant masks when prompts are ambiguous. ClipSAM uses CLIP for localization and rough segmentation, then guides SAM with spatial prompts and merges masks; experiments report state-of-the-art results across datasets, including gains over a SAM-based method on MVTec-AD.

  • Problem

    CLIP-based methods can be imprecise on local anomalies, while ambiguous prompts for SAM can produce redundant masks requiring selection.

    Source: §1 · pp. 1–2

  • Method

    ClipSAM uses CLIP for anomaly localization and rough segmentation, then prompts SAM with localization-derived points and boxes to refine and fuse masks.

    Source: §1 · p. 2

  • Results

    On MVTec-AD, ClipSAM outperforms the SAM-based method by +19.1 pixel-level AUROC, +10.0 F1-max, and +45.5 Pro, with new state-of-the-art results reported across datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ClipSAM offers a direction for improving ZSAS by leveraging the characteristics of different foundation models.

    Source: §5 · p. 7

Abstract

from arXiv · show

Recently, foundational models such as CLIP and SAM have shown promising performance for the task of Zero-Shot Anomaly Segmentation (ZSAS). However, either CLIP-based or SAM-based ZSAS methods still suffer from non-negligible key drawbacks: 1) CLIP primarily focuses on global feature alignment across different inputs, leading to imprecise segmentation of local anomalous parts; 2) SAM tends to generate numerous redundant masks without proper prompt constraints, resulting in complex post-processing requirements. In this work, we innovatively propose a CLIP and SAM collaboration framework called ClipSAM for ZSAS. The insight behind ClipSAM is to employ CLIP's semantic understanding capability for anomaly localization and rough segmentation, which is further used as the prompt constraints for SAM to refine the anomaly segmentation results. In details, we introduce a crucial Unified Multi-scale Cross-modal Interaction (UMCI) module for interacting language with visual features at multiple scales of CLIP to reason anomaly positions. Then, we design a novel Multi-level Mask Refinement (MMR) module, which utilizes the positional information as multi-level prompts for SAM to acquire hierarchical levels of masks and merges them. Extensive experiments validate the effectiveness of our approach, achieving the optimal segmentation performance on the MVTec-AD and VisA datasets.

1 Introduction

ClipSAM combines CLIP localization and rough segmentation with SAM refinement to address the limits of each model in zero-shot anomaly segmentation. The framework adds multi-scale cross-modal interaction and multi-level mask refinement, and its reported results include state-of-the-art performance across datasets.

1 Introduction

  • ZSAS seeks to localize and segment anomalous image regions without prior class-specific training samples, despite diverse products and uncertain anomaly types.CLIP-based approaches have difficulty precisely identifying local anomaly edges, while ambiguous SAM prompts can produce redundant masks that require selection.Source: §1 · pp. 1–2
  • ClipSAM first uses CLIP for anomaly localization and rough segmentation, then uses that positioning information to guide SAM in refining segmentation.This collaboration is intended to leverage the complementary characteristics of the two foundation models.Source: §1 · p. 2
  • UMCI interacts language features with visual features at row-column and multi-scale levels to learn local and global semantics about anomalous parts.It supports CLIP's localization and rough segmentation by aggregating image patch tokens from horizontal and vertical directions and considering multi-scale visual features.Source: §1 · p. 2
  • MMR extracts point and bounding-box prompts from CLIP localization results to guide SAM masks, then fuses those masks with CLIP results for fine-grained segmentation.The strategy integrates masks at different levels rather than relying on further mask filtering.Source: §1 · p. 2
  • On MVTec-AD, ClipSAM outperforms the SAM-based method by +19.1 pixel-level AUROC, +10.0 F1-max, and +45.5 Pro.The paper also reports new state-of-the-art zero-shot anomaly segmentation results across various datasets.Source: §1 · p. 2

2 Related Work

Prior ZSAS approaches use CLIP for patch-text similarity or SAM with text prompts and mask filtering, while related work also explores foundation-model collaboration and cross-modal fusion. ClipSAM builds on these threads with row-column and multi-scale interaction between text and visual features.

2.1 Zero-shot Anomaly Segmentation

  • CLIP-based ZSAS methods compare image-patch tokens with text features or improve modality alignment and text generalization, while SAM-based SAA uses text prompts to generate candidate masks and filter irrelevant ones.The paper identifies local-anomaly precision for CLIP and prompt dependence for SAM as limitations motivating collaboration.Source: §2.1 · p. 2

2.2 Foundation Models

  • CLIP provides multimodal semantic understanding and SAM fine-grained prompt-based segmentation; prior work has connected or merged them for other downstream tasks.The cited examples include surgical-instrument segmentation and broader downstream-task facilitation.Source: §2.2 · pp. 2–3

2.3 Cross-modal Interaction

  • Cross-modal interaction methods fuse modalities or correlate image regions with text, motivating ClipSAM's interaction of text with row-column and multi-scale visual features for anomaly localization.The cited related work includes feature concatenation and convolution, keyword-to-image-area correlations, and cross-modal encoder-block information exchange.Source: §2.3 · p. 3

3 Methodology

ClipSAM uses CLIP for anomaly localization and rough segmentation, then guides SAM to refine the result. Its UMCI and MMR modules implement this collaboration through multi-scale cross-modal features and spatial prompts.

3.1 CLIP and SAM Collaboration

  • ClipSAM first uses CLIP for anomaly localization and rough segmentation, then uses that information to guide SAM in refining the segmentation.The framework assigns complementary roles to the two models: CLIP supplies the initial localization and SAM refines it.Source: §3.1 · pp. 3–4

3.2 Unified Multi-scale Cross-modal Interaction

  • UMCI combines a Strip Path that captures row- and column-level features with a Scale Path that captures visual features at multiple scales.The Strip Path interacts with language features to predict pixel-level normal or abnormal regions; the paths' predictions are fused with a residual connection from the input patch tokens.Source: §3.2 · p. 4

3.3 Multi-level Mask Refinement

  • MMR binarizes CLIP's rough segmentation, extracts point and box prompts for SAM, and fuses SAM's refined masks with the rough segmentation.The framework uses points and boxes together as prompts, and SAM produces three masks for each box.Source: §3.3 · pp. 4–5

3.4 Objective Function

  • Focal and Dice losses train the UMCI module; the four CLIP encoder stages receive loss weights of 0.1, 0.1, 0.1, and 0.7.Focal loss addresses class imbalance, which is relevant because anomalies usually occupy a small fraction of an object.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate ClipSAM on MVTec-AD and VisA, report performance gains over existing methods, and use MVTec-AD ablations to examine its components and settings.

4.1 Experimental Setup

  • Experiments use MVTec-AD and VisA, training on one dataset and testing on the other in both directions, and assess AUROC, AP, F1-max, and PRO.The study uses CLIP ViT-L-14-336 and SAM ViT-H; UMCI training runs for six epochs.Source: §4.1 · p. 5

4.2 Experiments on MVTec-AD and VisA

  • 19.1% AUROC, 10.0% F1-max, 17.1% AP, and 45.5% PRO improvements over the SAM-based approach on MVTec-AD; ClipSAM also leads existing methods across all four metrics.The comparison reports overall enhancement on VisA as well.Source: Table 1 · p. 6, §4.2 · p. 6
  • Visual comparisons show ClipSAM's accurate localization and segmentation, while Figure 5 traces CLIP rough masks into location-derived prompts for SAM refinement.Figure 4 compares ClipSAM with CLIP-based and SAM-based methods on MVTec-AD.Source: Figure 4 · p. 6, Figure 5 · p. 6, §4.2 · p. 7

4.3 Ablation Studies

  • Removing either UMCI path reduces performance, with a greater impact from removing the scale path; removing MMR slightly lowers performance.The component ablation is on MVTec-AD.Source: §4.3 · p. 7, Table 2 · p. 7
  • Removing UMCI causes a performance collapse because similarity-based segmentation lacks text-aligned patch tokens and accurate local prompts for MMR.Figure 6 compares anomaly localization and rough segmentation using UMCI with similarity-based segmentation.Source: §4.3 · p. 7, Figure 6 · p. 7
  • Larger ch values help interpret visual features, si should be moderate for visual context, and thr should be moderate to avoid missed anomalies or false regions.The hyperparameter analysis covers hidden dimension, kernel size, and binarization threshold.Source: §4.3 · p. 7, Table 3 · p. 7

5 Conclusion

ClipSAM combines CLIP localization and rough segmentation with SAM refinement guided by precise spatial prompts. Its experiments support this approach as a direction for improving zero-shot anomaly segmentation.

5 Conclusion

  • ClipSAM is a two-module framework for zero-shot anomaly segmentation, using UMCI for anomaly localization and rough segmentation and MMR to refine results with SAM and precise spatial prompts.The conclusion identifies integration of knowledge from different models as future work to enhance zero-shot anomaly segmentation.Source: §5 · p. 7
  • UMCI explicitly reasons about anomaly locations and produces rough segmentation.Source: §5 · p. 7
  • MMR refines rough segmentation by using SAM with precise spatial prompts.Source: §5 · p. 7

A Text design

ClipSAM builds category-specific normal and anomaly text representations by averaging CLIP text features from descriptive prompts.

Text prompt construction

  • For each object category, normal-state and anomaly-state phrases are combined with prompt templates, and CLIP text features are averaged separately for each state.With m templates and n phrases, each state yields m × n descriptive statements.Source: A Text design · p. 8
  • The resulting normal and abnormal features are concatenated into a two-category text representation; object categories are assumed known, but anomaly categories are not.When object categories are unknown, the placeholder “object” can replace a specific category.Source: A Text design · p. 8

B Strip path and Scale path

UMCI uses parallel strip and scale paths to interact language features with visual features, combining directional local cues with multi-scale context.

Strip Path

  • The Strip Path pools visual features into row- and column-level representations and interacts them with language features to identify salient pixels.Its row and column outputs are combined to form the strip-path output.Source: B Strip path and Scale path · p. 8
  • Strip-path attention uses visual and language features, with a residual connection adding the original row-level visual feature to the result.The interaction uses scaled dot-product attention [Vaswani et al., 2017].Source: B Strip path and Scale path · p. 8

Scale Path

  • The Scale Path pools visual features at different scales, matches language-feature dimensions, and performs cross-modal attention at each scale.A residual connection adds the original visual features to the interaction result.Source: B Strip path and Scale path · pp. 8–9

UMCI design

C Additional experiments on more datasets

Additional experiments examine ClipSAM across MVTec-AD, VisA, MTD, and KSDD2, with dataset descriptions and a comparison table covering MTD and KSDD2.

C Additional experiments on more datasets

  • The additional experiments evaluate ClipSAM on MTD and KSDD2, and Table 4 covers AUROC, F1-max, AP, and PRO for both datasets.The study also reports validation on MVTec-AD and VisA, with MTD and KSDD2 used to further assess generalizability.Source: C Additional experiments on more… · p. 9, Table 4 · p. 10

C.1 MVTec-AD

  • MVTec-AD includes 3,466 unlabeled images and 1,888 pixel-annotated images across 15 categories and 73 anomaly types.Its training set has 3,629 defect-free images; its test set has 1,725 defective and defect-free samples.Source: §C.1 · p. 9

C.2 VisA

  • VisA contains 10,821 images across 12 object-category subsets, including complex printed circuit boards and views with multiple instances.The dataset has 9,621 normal samples and 1,200 anomaly samples.Source: §C.2 · p. 9

C.3 MTD

  • MTD is a single-object-category dataset with six magnetic-tile defect types and 925 defect-free and 392 anomalous images.The images have annotations, and evaluation uses AUROC, AP, F1-max, and PRO.Source: §C.3 · p. 9

C.4 KSDD2

  • KSDD2 contains 356 images with defects and 2,979 without defects, with image dimensions of approximately 230 × 630 pixels.Its defect types include scratches, small spots, and surface defects.Source: §C.4 · p. 10

C.5 Experimental results

On MTD, ClipSAM reports substantial metric improvements over APRIL-GAN and SAA+, alongside qualitative comparisons on MTD and KSDD2. The experiments train on MVTec-AD and test on MTD and KSDD2.

C.5 Experimental results

  • The experiments train on MVTec-AD and test on MTD and KSDD2, using all MTD samples and the KSDD2 test set for testing.The CLIP encoder uses ViT-L-14-336 and the SAM model uses ViT-H; no comparison is provided with CLIP methods not evaluated on these datasets.Source: §C.5 · p. 10
  • On MTD, ClipSAM improves over APRIL-GAN by 36.9% in AUROC, 38.6% in F1-max, 42.1% in AP, and 54.1% in PRO.Against SAA+, the reported improvements are 18.6% in AUROC, 17.9% in F1-max, and 23.1% in AP.Source: §C.5 · p. 11
  • Qualitative comparisons report that ClipSAM localizes anomalies and segments them precisely on MTD and KSDD2, while APRIL-GAN and SAA show segmentation errors.APRIL-GAN has inaccurate regions and false predictions outside real masks; SAA produces erroneous large masks or misses mask regions.Source: §C.5 · p. 11

D Additional visualization

The visual comparisons show ClipSAM locating and segmenting anomalous regions more effectively than the compared CLIP-based and SAM-based methods. Its CLIP-to-SAM strategy mitigates misdetections and omissions.

D Additional visualization

  • On MVTec-AD, ClipSAM shows better anomaly location and segmentation boundaries than the compared CLIP-based and SAM-based methods.The comparisons include APRIL-GAN and SAA.Source: D Additional visualization · p. 11, Figure 11 · p. 12, Figure 12 · p. 12
  • CLIP rough segmentation followed by SAM refinement mitigates some anomaly misdetections and omissions.The authors attribute ClipSAM’s stronger understanding of anomalous regions to UMCI and MMR.Source: D Additional visualization · p. 11
  • APRIL-GAN tends to locate anomalies less accurately, while SAM-based results can deviate when masks lack precise constraints.CLIP-based segmentation can also misclassify non-anomalous regions or diverge from ground-truth labels.Source: D Additional visualization · p. 11

E Two-stage visualization

The visualizations trace ClipSAM’s two-stage process: CLIP produces rough anomaly regions and spatial prompts, then SAM refines the segmentation. The examples also show limits of CLIP’s fine-grained segmentation and threshold-based binarization.

E Two-stage visualization

  • Binarized CLIP rough segmentation supplies connected-region spatial prompts, namely points and boxes, for SAM in the second stage.The visualizations separately show the outputs of the stages.Source: E Two-stage visualization · p. 12
  • With UMCI, CLIP produces rough abnormal regions, but its classification-oriented multimodal semantics can limit fine-grained segmentation to parts of a true anomaly.The visualizations cover MVTec-AD and VisA.Source: E Two-stage visualization · pp. 12–13, Figure 15 · p. 14, Figure 16 · p. 14
  • SAM can refine CLIP-predicted abnormal regions into masks closer to ground truth, including by focusing on small anomalous areas.The rough segmentation and binarized mask need not have a one-to-one correspondence between red regions and connected regions.Source: E Two-stage visualization · pp. 13–14
  • On VisA, where anomaly categories are typically small, UMCI-assisted CLIP often segments accurately but can exceed ground-truth regions.The text notes that value suppression is applied to regions beyond ground truth when computing metrics.Source: E Two-stage visualization · p. 14, Figure 17 · p. 15

F Comparing with more methods.

ClipSAM is compared with AnoVL and AnomalyCLIP and is reported to achieve optimal zero-shot anomaly segmentation performance. Its text strategy remains unchanged from WinCLIP, leaving text generality as a direction for future exploration.

F Comparing with more methods.

  • ClipSAM achieves optimal reported zero-shot anomaly segmentation performance in comparison with AnoVL and AnomalyCLIP.AnoVL uses domain-specific prompt-template designs, while AnomalyCLIP applies prompt learning to CLIP’s text encoder.Source: F Comparing with more methods · p. 15
  • ClipSAM adopts WinCLIP’s text strategy without modification, and the authors identify text generality as a potential direction for future exploration.Source: F Comparing with more methods · p. 15
Loading 2401.12665v2…