PaperlayerView exact revision on arXiv

Source-linked AI summary

Segment Any Anomaly without Training via Hybrid Prompt Regularization

Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, Weiming Shen

arXiv:2305.10724v1cs.CVcs.AI

TL;DR

Anomaly segmentation often relies on category-specific training, while zero-shot deployment faces the challenge of adapting foundation models to target anomalies. SAA+ combines foundation models with hybrid prompts from expert knowledge and image context, and the paper reports state-of-the-art zero-shot results across several benchmarks.

  • Problem

    Many anomaly-segmentation methods require domain-specific training, while zero-shot adaptation must handle ambiguity in naive prompts.

    Source: §1 · p. 1, §3.3 · p. 5

  • Method

    SAA+ assembles foundation models and regularizes them with hybrid prompts derived from domain expertise and target-image context.

    Source: §1 · pp. 2–3

  • Results

    SAA+ achieves state-of-the-art zero-shot anomaly-segmentation performance on VisA, MVTec-AD, KSDD2, and MTD.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The work presents hybrid prompting as a training-free way to adapt foundation models for zero-shot anomaly segmentation.

    Source: §1 · p. 3, §6 · p. 10

    Because of computation restrictions, the authors do not test the method on larger-scale foundation models.

    Source: §6 · p. 10

Abstract

from arXiv · show

We present a novel framework, i.e., Segment Any Anomaly + (SAA+), for zero-shot anomaly segmentation with hybrid prompt regularization to improve the adaptability of modern foundation models. Existing anomaly segmentation models typically rely on domain-specific fine-tuning, limiting their generalization across countless anomaly patterns. In this work, inspired by the great zero-shot generalization ability of foundation models like Segment Anything, we first explore their assembly to leverage diverse multi-modal prior knowledge for anomaly localization. For non-parameter foundation model adaptation to anomaly segmentation, we further introduce hybrid prompts derived from domain expert knowledge and target image context as regularization. Our proposed SAA+ model achieves state-of-the-art performance on several anomaly segmentation benchmarks, including VisA, MVTec-AD, MTD, and KSDD2, in the zero-shot setting. We will release the code at \href{https://github.com/caoyunkang/Segment-Any-Anomaly}{https://github.com/caoyunkang/Segment-Any-Anomaly}.

1 Introduction

Many anomaly segmentation methods train separate models for limited categories, while zero-shot anomaly segmentation provides a setting without target-category training images. SAA assembles foundation models, and SAA+ adds expert- and image-context-derived hybrid prompts to address false alarms from naive prompts.

1 Introduction

  • Separate category-specific training limits the cost-effective deployment of anomaly segmentation across the many products found in real-world settings.The paper frames zero-shot anomaly segmentation as a setting with neither normal nor abnormal target-category images available during training.Source: §1 · p. 1
  • SAA cascades prompt-guided detection and segmentation foundation models, using a naive language prompt to generate and refine anomaly regions into masks.The detector generates box-level regions and the refiner produces final masks.Source: §1 · p. 2
  • Naive prompts can cause false alarms because of domain shift and because anomaly descriptions need to reflect the target object’s context.For example, SAA marks all wicks as anomalous even though only the overlong wick is the ground-truth anomaly.Source: §1 · p. 2, Figure 1 · p. 2
  • SAA+ uses expert-derived class-specific language and object-property prompts alongside image-context saliency and confidence-ranking prompts to regularize foundation models.The prompts are intended to help identify and calibrate anomaly-region predictions without training.Source: §1 · pp. 2–3
  • SAA+ achieves state-of-the-art zero-shot anomaly-segmentation performance on VisA, MVTec-AD, KSDD2, and MTD.The paper also reports texture-anomaly detection without annotation.Source: §1 · p. 3

2 Related work

Related work covers trained anomaly-segmentation approaches, emerging zero-shot methods, and foundation-model prompting. The paper positions its hybrid prompts as a training-free alternative for zero-shot anomaly segmentation.

2 Related work

  • Unsupervised anomaly segmentation commonly learns normality from normal images using reconstruction, teacher-student, one-class, or memory-based approaches.These methods identify anomalies by comparing inputs with reconstructions or by finding feature-distribution differences.Source: §2 · p. 3
  • Emerging zero-shot anomaly-segmentation work avoids training images, while WinClip uses text-visual similarity and this paper instead generates and scores anomaly proposals.The paper describes its proposal-based approach as achieving better segmentation performance than WinClip.Source: §2 · p. 3
  • Foundation models support zero-shot vision tasks, including open-set detection with Grounding DINO and open-world mask generation with SAM.The paper explores adapting these off-the-shelf models to anomaly detection without downstream training.Source: §2 · p. 3
  • Learnable-token prompting requires training data unavailable in ZSAS, whereas the paper proposes training-free hybrid prompts from domain expertise and target-image context.Prior prompting work has used text, vision, or both as inputs.Source: §2 · p. 3

3 SAA: Vanilla Foundation Model Assembly for ZSAS

The section defines zero-shot anomaly segmentation and assembles a vanilla model from foundation-model detection and segmentation stages. It then identifies language ambiguity as a source of false alarms in that baseline.

3.1 Problem Definition: Zero-shot Anomaly Segmentation (ZSAS)

  • ZSAS aims to produce a pixel-level anomaly map for novel objects without corresponding object training data.The task is framed with an empty training set and an image of a novel object.Source: §3.1 · p. 3

3.2 Baseline Model Assembly: Segment Any Anomaly (SAA)

  • SAA first uses a language-driven GroundingDINO generator to propose coarse anomaly regions, then refines them with SAM into pixel-wise masks.The proposals are boxes, and the refiner treats them as prompts for segmentation.Source: §3.2 · p. 4, §3.2.2 · pp. 4–5

3.2.1 Anomaly Region Generator

  • The generator uses language prompts to retrieve image regions and outputs bounding boxes with confidence scores.Its detector architecture is based on text-guided open-set object detection.Source: §3.2.1 · p. 4

3.2.2 Anomaly Region Refiner

  • The refiner uses SAM’s prompt-conditioned mask decoder to convert box candidates into pixel-wise segmentation masks.SAM is presented as an open-world visual segmentation foundation model.Source: §3.2.2 · pp. 4–5

3.3 Analysis on the ZSAS Performance of Vanilla Foundation Model Assembly

  • With a naive prompt such as “anomaly,” SAA can label all wicks as anomalous even when only one wick is defective.The example illustrates the language ambiguity observed in preliminary experiments.Source: §3.3 · p. 5, §3.2.2 · p. 5
  • The paper attributes this ambiguity to domain gaps and object-specific meanings of anomalies, and proposes expert- and image-context-derived hybrid prompts to reduce it.For example, an anomaly may mean scratches on leather or a crack on a hazelnut.Source: §3.3 · p. 5

4 SAA+: Foundation Model Adaption via Hybrid Prompt Regularization

SAA+ addresses language ambiguity in zero-shot anomaly segmentation by combining foundation-model knowledge with domain-expert knowledge and target-image context to generate more accurate masks.

4 SAA+: Foundation Model Adaption via Hybrid Prompt Regularization

  • SAA+ combines foundation-model knowledge with domain-expert knowledge and target-image context to address language ambiguity in zero-shot anomaly segmentation.It uses hybrid prompts to regularize SAA and make anomaly predictions more reliable.Source: §4 · p. 5, §4.2.2 · p. 7

4.1.1 Anomaly Language Expression as Prompt

  • Expert-informed class-specific language prompts describe anomaly patterns, while class-agnostic prompts such as “anomaly” provide generic initial prompts.Examples of class-specific expressions include “black hole” and “white bubble.”Source: §4.1.1 · pp. 5–6

4.1.2 Anomaly Object Property as Prompt

  • Property prompts use expert-set location and area rules to filter candidate regions that do not meet the desired criteria.Location uses an IoU threshold against the inspected object; area uses a threshold relative to object area.Source: §4.1.2 · p. 6

4.2 Prompts Derived from Target Image Context

  • The framework also derives prompts from the target image context to improve anomaly-region detection.It introduces anomaly saliency and anomaly confidence prompts.Source: §4.2 · p. 6, §4.2.1 · p. 6, §4.2.2 · p. 7

4.2.1 Anomaly Saliency as Prompt

  • Anomaly saliency is estimated from pixel-feature distances to nearest neighbors, then averaged within candidate masks to provide region saliency prompts.These prompts recalibrate foundation-model confidence scores using both model confidence and region saliency.Source: §4.2.1 · pp. 6–7

4.2.2 Anomaly Confidence as Prompt

  • Anomaly confidence prompts select the K candidate regions with the highest image-content-based confidence scores for estimating the final anomaly map.The selected candidates’ average values are used for final anomaly-region detection.Source: §4.2.2 · p. 7
  • The hybrid language, property, saliency, and confidence prompts regularize SAA to produce more reliable anomaly predictions.Language prompts and property filters address expert-specified anomaly descriptions and region characteristics, while image-derived prompts use target-image context.Source: §4.1 · p. 5, §4.2 · p. 6, §4.2.2 · p. 7

5 Experiments

Experiments evaluate SAA+ against zero-shot methods on four benchmarks and examine its hybrid prompts; SAA+ leads on both reported metrics, with ablations showing distinct contributions from the prompts.

5.1 Experimental Setup

  • Evaluation uses VisA, MVTec-AD, KSDD2, and MTD, with max-F1-pixel (Fp) and max-F1-region (Fr) metrics.Images are evaluated at 400 × 400 resolution; the default confidence-prompt hyperparameter is K = 5.Source: §5.1 · p. 7

5.2 Main Results

  • SAA+ outperforms WinClip, UTAD, ClipSeg, and SAA by a significant margin on both Fp and Fr.The comparison covers zero-shot anomaly segmentation.Source: Table 1 · p. 8, §5.2 · p. 8
  • Figure 3 shows better qualitative performance for SAA+ and illustrates detection of small texture anomalies such as leather scratches.The figure compares ClipSeg, UTAD, SAA, and SAA+ across VisA, MVTec-AD, KSDD2, and MTD.Source: §5.2 · p. 8, Figure 3 · p. 8

5.3 Ablation study

  • The confidence-prompt sensitivity analysis shows performance initially rising with K and then declining slightly beyond around K = 5.The figure focuses on the effect of K for confidence prompts.Source: Figure 5 · p. 9, §5.3 · p. 9
  • +3.90% in Fp and +4.90% in Fr accompany domain-expert language prompts; general and specific descriptions each perform reasonably, and their combination improves segmentation.The ablation examines language prompt designs within the hybrid prompt study.Source: §5.3 · p. 9, Table 2 · p. 9
  • On texture categories, property prompts raise Fp from 21.83% to 53.79% by filtering region candidates using properties such as location and area.The ablation attributes the improvement to removing falsely detected anomaly candidates.Source: §5.3 · p. 9, Table 2 · p. 9
  • Saliency prompts use region deviation from surroundings to calibrate confidence scores, helping highlight abnormal regions and improve segmentation precision.Examples include localizing a cracked hazelnut region and an overlong candle wick.Source: §5.3 · p. 9, Figure 4 · p. 9, Table 2 · p. 9
  • Confidence prompts limit anomaly regions and reduce false positives, yielding 0.72% average Fp improvement across categories.Performance peaks around K = 5, with average Fp of 34.85%, then drops slightly when K exceeds that threshold.Source: §5.3 · p. 9, Table 2 · p. 9

6 Conclusion

SAA+ adapts foundation models for zero-shot anomaly segmentation without further training, using hybrid prompts from expert knowledge and target-image context. The authors report new state-of-the-art results on several benchmarks, while noting that larger foundation models remain untested.

6 Conclusion

  • Hybrid prompts from expert knowledge and target-image context regularize foundation models for zero-shot anomaly segmentation without further training.The authors frame prompt design as key to adapting off-the-shelf foundation models.Source: §6 · p. 10
  • SAA+ achieves new state-of-the-art results on several zero-shot anomaly segmentation benchmarks.Source: §6 · p. 10
  • Larger-scale foundation models remain untested because of computation restrictions; scaling effects are left for future exploration.Source: §6 · p. 10

Related papers

Loading 2305.10724v1…