PaperlayerView exact revision on arXiv

Source-linked AI summary

Proximity-CLIP: Text-Guided Semantic Proximity Learning for Zero-Shot Anomaly Detection

Manwen Yang, Leqian Ding, Yu Guo, Fei Wang

arXiv:2609.07229v1cs.CV

TL;DR

CLIP’s object-centric bias leaves normal and anomalous text prototypes highly overlapping, while strict orthogonality can disrupt pre-trained structural continuity. Proximity-CLIP learns a visually calibrated semantic margin and uses the anomalous prototype to retrieve localized defect cues; it outperforms current state-of-the-art methods across multiple ZSAD benchmarks with minimal architectural modifications.

  • Problem

    CLIP’s object-centric bias limits fine-grained sensitivity for localized anomalies, and strict orthogonality between text prototypes can disrupt pre-trained structural continuity.

    Source: §1 · p. 1

  • Method

    Proximity-CLIP learns a visually calibrated semantic margin with bounded dynamic regularization, then uses the calibrated anomalous prototype as a query to retrieve localized defect cues.

    Source: §1 · pp. 2–3

  • Results

    Proximity-CLIP outperforms current state-of-the-art methods across multiple zero-shot anomaly detection benchmarks with minimal architectural modifications.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The framework combines discriminative separation with preservation of pre-trained metric integrity and retrieval of localized defect cues.

    Source: §1 · p. 3, §5 · p. 14

Abstract

from arXiv · show

Vision-language models offer a promising approach for zero-shot anomaly detection (ZSAD). However, due to object-centric bias, normal and anomalous text prototypes exhibit a high semantic overlap. While enforcing strict orthogonality between them improves discriminability, mapping highly contiguous visual inputs onto drastically orthogonal prototypes introduces a geometric dilemma, disrupting the pre-trained structural continuity. To address this problem, we propose Proximity-CLIP, a framework that visually calibrates the semantic margin to guide visual adaptation. First, we introduce a visually-calibrated semantic proximity learning mechanism that uses a bounded dynamic regularization to learn an appropriate semantic margin, ensuring discriminative separation while preserving structural alignment. Second, we design an Anomaly Query Module (AQM) driven by these text priors. Using the calibrated anomalous prototype as a semantic query, the AQM actively retrieves localized defect cues from contextual visual patches, mitigating the dilution of subtle anomalies during global pooling. Extensive experiments demonstrate that Proximity-CLIP outperforms current state-of-the-art methods across multiple ZSAD benchmarks with minimal architectural modifications.

1 Introduction

CLIP’s object-centric bias makes it difficult to detect fine-grained, localized anomalies, while strict orthogonality between normal and anomalous text prototypes can disrupt pre-trained structural continuity. Proximity-CLIP calibrates semantic proximity to guide visual adaptation and uses the calibrated anomalous prototype to retrieve localized defect cues.

1 Introduction

  • CLIP’s object-centric bias limits its fine-grained sensitivity to localized structural variations needed for zero-shot anomaly detection.Strict orthogonality between text prototypes may disrupt pre-trained feature continuity and compromise generalization to unseen categories.Source: §1 · pp. 1–2
  • Visually-calibrated semantic proximity learning uses bounded dynamic regularization to learn a semantic margin that preserves structural integrity while separating normal and anomalous states.The margin is driven by actual visual variance rather than rigid angular constraints.Source: §1 · pp. 2–3
  • The Anomaly Query Module uses the calibrated anomalous prototype as a semantic query to retrieve localized defect cues from contextual visual patches.This target-aware retrieval addresses the dilution of subtle anomalies during global pooling.Source: §1 · p. 3
  • Proximity-CLIP outperforms current state-of-the-art methods across multiple zero-shot anomaly detection benchmarks.The framework requires minimal architectural modifications.Source: §1 · p. 3

2 Related Work

Traditional anomaly detectors rely on category-specific data, while CLIP offers zero-shot capabilities but has limited sensitivity to fine-grained local variations. Recent ZSAD methods improve text-guided alignment and localization, yet aggressive separation of normal and anomalous features can disrupt feature continuity and unseen-category generalization.

2.1 Traditional Anomaly Detection

  • Traditional reconstruction- and embedding-based anomaly detectors can work in constrained environments but rely on category-specific data, limiting scalability to open-world scenarios.Reconstruction methods detect discrepancies from normal-pattern reconstructions; embedding methods compare test features with distributions of normal features.Source: §2.1 · p. 3

2.2 CLIP and Parameter-Efficient Fine-Tuning

  • CLIP aligns visual and textual representations for zero-shot use, but its object-centric representations are insensitive to fine-grained localized structural variations in dense prediction tasks.Prompt tuning modifies textual inputs, whereas adapter tuning inserts lightweight trainable components into transformer blocks to adapt features.Source: §2.2 · p. 4

2.3 CLIP-Based Anomaly Detection

  • ZSAD methods have explored compositional prompts, multi-level feature aggregation, object-agnostic or dynamic prompts, and anomaly-aware text anchors.Aggressively maximizing normal–abnormal feature distance can disrupt pretrained metric continuity and compromise generalization to unseen open-world categories.Source: §2.3 · pp. 4–5

3 Method

Proximity-CLIP is a two-stage parameter-efficient framework for zero-shot anomaly detection, calibrating text-prototype separation to preserve visual structure and using the calibrated anomalous prototype to query localized visual evidence.

3.1 Problem Setting and Geometric Dilemma

  • CLIP’s object-centric bias leaves normal and anomalous prototypes highly similar, while forcing orthogonality conflicts with the visual proximity of localized defects and risks unseen-category generalization.The method frames this as a geometric dilemma: anomalous instances are normal objects with localized defects, so their visual representations remain proximate.Source: The Geometric Dilemma · pp. 6–7

3.2 Visually-Calibrated Semantic Proximity

3.3 Text-Guided Visual Adaptation

  • Visual adaptation freezes the calibrated text branch and jointly optimizes spatial segmentation alignment and image-level classification against the frozen prototypes.The visual-stage objective combines image alignment and classification losses, with the pooled anomaly token supervised by image-level labels.Source: §3.3 · p. 8, Global-Local Joint Optimization · p. 9
  • The Anomaly Query Module uses the calibrated anomalous prototype to query multi-scale visual patches and retrieve localized defect cues that uniform processing may dilute during global pooling.The anomalous prototype acts as the semantic query, while adapted patch features provide contextual keys and values; a residual connection stabilizes gradient flow.Source: Semantic-Guided Anomaly Querying · pp. 8–9

4 Experiments

Proximity-CLIP is evaluated on industrial and medical zero-shot anomaly detection benchmarks, with strong localization and classification results. The analyses and ablations examine semantic calibration, feature-space structure, qualitative localization, and component contributions.

4.1 Experimental Setup

  • Evaluation uses cross-dataset training across seven industrial and medical datasets, with image- and pixel-level AUROC as primary metrics.VisA trains the model for the other six datasets, while MVTec-AD is used to evaluate VisA.Source: §4.1 · pp. 9–10

4.2 Main Results

  • 95.7% average Pixel AUROC across seven industrial and medical datasets; Proximity-CLIP also achieves robust localization on subtle-pattern Brain and Retina datasets.The comparison covers four industrial and three medical datasets.Source: Table 1 · p. 10, Anomaly Localization · p. 11
  • 81.0% overall average classification score, the highest among evaluated baselines, with results across diverse semantic domains.Table 2 reports image-level anomaly classification performance.Source: Anomaly Classification · p. 11, Table 2 · p. 11

4.3 In-Depth Analysis

  • Visually calibrated text-to-text similarities closely mirror unseen-class image similarities, avoiding forced orthogonality while resolving semantic entanglement and preserving metric integrity.Figure 3 compares original CLIP text embeddings, calibrated text embeddings, and actual image-feature similarities.Source: Semantic Proximity Calibration · p. 11, Figure 3 · p. 12
  • Proximity-CLIP separates normal and anomalous features within seen and unseen classes while preserving inter-class topology and zero-shot generalization.The analysis attributes this to a controlled angular shift rather than orthogonal or uncorrelated spaces.Source: Preserving Metric Space Integrity · pp. 12–13, Figure 4 · p. 12
  • Compared with AA-CLIP, Proximity-CLIP more precisely localizes subtle, fine-grained defects and suppresses background noise in qualitative heatmaps.The paper attributes this to using calibrated anomalous semantics as an active query in cross-attention.Source: Qualitative Anomaly Localization · p. 13, Figure 5 · p. 13

4.4 Ablation Study

  • Semantic calibration alone raises Medical pixel AUROC from 55.5% to 84.4%, while omitting calibration drops Industrial image AUROC from 86.2% to 70.4%.The first comparison is for the w/o AQM variant; the second is for w/o SC.Source: Table 3 · p. 12, §4.4 · pp. 13–14
  • Removing both Norm-Preserving mechanisms drops Image AUROC by 4.4% in Medical and 4.2% in Industrial; active querying beats passive multi-scale pooling by 1.3% Industrial Pixel and 2.2% Medical Image AUROC.The full framework achieves the best overall performance among evaluated variants.Source: Table 3 · p. 12, §4.4 · p. 14

5 Conclusion

Proximity-CLIP uses calibrated semantic proximity and active visual retrieval to address the mismatch between local anomalies and strict prototype orthogonality. Experiments on industrial and medical benchmarks report state-of-the-art zero-shot performance with minimal architectural changes.

5 Conclusion

  • Bounded dynamic regularization learns a semantic margin that separates normal and anomalous prototypes while preserving the original metric integrity.The paper frames strict orthogonality as a geometric conflict with pre-trained structural continuity.Source: §5 · p. 14
  • The Anomaly Query Module uses the calibrated anomalous prototype to retrieve localized defect cues from visual patches, preventing subtle anomalies from being diluted by global pooling.Source: §5 · p. 14
  • Experiments on industrial and medical benchmarks demonstrate state-of-the-art zero-shot performance with minimal architectural changes.The paper presents the continuity-preserving and active-retrieval approach as a practical foundation for future open-world dense prediction tasks.Source: §5 · p. 14

A More Quantitative Results

Additional evaluations cover four medical datasets composed exclusively of anomalous images, so the reported measure is pixel-level localization rather than image-level classification.

A More Quantitative Results

  • Proximity-CLIP consistently outperforms previous methods across all four evaluated medical datasets in pixel-level anomaly localization.The datasets contain only anomalous images, so image-level classification metrics cannot be formulated.Source: A More Quantitative Results · p. 18, Table A.1 · p. 18
  • The appendix reports Pixel AUROC only because these datasets contain no normal samples for image-level classification.Source: A More Quantitative Results · p. 18
  • Table A.1 compares Pixel AUROC across four medical datasets, with best and second-best results highlighted.Source: Table A.1 · p. 18

B Extended Analysis

The extended analysis motivates bounding the learned semantic margin λ: unconstrained optimization can leave the valid cosine-similarity range and exploit a trivial solution. The bounded formulation is reported to maintain non-zero convergence and distinct textual representations.

B.1 Design Rationale of Bounded Dynamic Proximity Regularization

B.2 Sensitivity Analysis of Margin Bounds

The sensitivity analysis tests how the margin bounds α and β affect performance across six datasets. Performance is stable within broad bounds but degrades when the bounds constrain the margin too tightly.

B.2 Sensitivity Analysis of Margin Bounds

  • The analysis varies α and β and averages performance across all six evaluated datasets.Source: §B.2 · p. 20, Figure B.2 · p. 20
  • With β fixed at 0.3, performance remains stable for α ≥ 0.5 but degrades sharply for α ≤ 0.35.The authors link the degradation to disruption of pre-trained structural continuity under strict orthogonality.Source: §B.2 · p. 20, Figure B.2 · p. 20
  • With α fixed at 0.65, performance is stable until β approaches 1.0, which restricts dynamic expansion.The authors state that a sufficiently wide upper bound lets the network adapt to the optimal margin.Source: Figure B.2 · p. 20, §B.2 · p. 21
  • The tested bounds α = 0.65 and β = 0.3 are described as a safe, physically grounded constraint requiring no domain-specific tuning.Source: §B.2 · p. 21

B.3 Computational Efficiency and Overhead Analysis

Proximity-CLIP adds a small trainable module budget and modest Stage 2 training overhead. Its inference latency matches AA-CLIP at 11.7 FPS.

B.3 Computational Efficiency and Overhead Analysis

  • ∼22.0M trainable parameters account for 5% of the backbone model's total parameters.Source: §B.3 · p. 21
  • Stage 2 training adds 4% computational overhead relative to AA-CLIP, while Stage 1 is slightly faster.Source: §B.3 · p. 21, Table B.1 · p. 21
  • 11.7 FPS: the AQM has inference latency identical to AA-CLIP.The efficiency comparison caption characterizes this as identical inference latency despite performance gains.Source: §B.3 · p. 21, Table B.1 · p. 21

C More Qualitative Results

The additional qualitative comparisons cover medical and industrial datasets. They show localized anomaly maps in both settings, including subtle anatomical anomalies amid complex textures.

C More Qualitative Results

Related papers

Loading 2609.07229v1…