PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images

Juzheng Miao, Yuchen Yuan, Cheng Chen, Pheng-Ann Heng

arXiv:2609.12454v1cs.CVcs.AIcs.LG

TL;DR

CLIP enables few-shot medical anomaly detection, but its global contrastive pretraining does not ensure precise lesion localization. Spatial-FAD integrates DINO spatial priors with CLIP semantics and adds spatially dense aggregation and support prototypes, reporting improved segmentation across three datasets.

  • Problem

    CLIP-based methods can detect anomalies effectively, but global image-text alignment lacks explicit fine-grained correspondence between local patches and lesion descriptions, limiting lesion-map quality.

    Source: §1 · p. 2

  • Method

    Spatial-FAD injects DINO-derived structural affinity into CLIP features, aggregates overlapping high-resolution windows, and uses compact normal and abnormal support prototypes alongside text similarities.

    Source: §1 · pp. 2–3, §2.2 · p. 5, §2.3 · pp. 5–6

  • Results

    In the 4-shot setting, Spatial-FAD improves average Dice by over 11.4% and average AUC by more than 1.8% across three datasets.

    Source: §1 · p. 3, §4 · p. 9

  • Takeaways & Limitations

    Few-shot support examples contribute instance-level knowledge alongside CLIP text alignment in the reported medical anomaly-detection setting.

    Source: §2.3 · pp. 5–6

Abstract

from arXiv · show

Vision-Language Models such as CLIP enable effective few-shot medical anomaly detection (AD) via strong image-text semantic alignment. However, their globally contrastive pretraining lacks explicit spatial supervision, limiting precise lesion localization. In contrast, Vision Foundation Models (VFMs) such as DINO learn spatially coherent patch representations via self-distillation and local-to-global consistency, better capturing fine-grained anatomical structures. Leveraging this complementarity, we propose Spatial-FAD, a spatial-aware few-shot medical AD framework that improves lesion localization by combining VFM spatial priors with CLIP semantics. Specifically, we introduce a VFM-enhanced adapter that injects a structural affinity prior derived from DINO into CLIP features. This structure-guided refinement encourages visual embeddings to better adhere to lesion boundaries while maintaining semantic alignment. To address the loss of spatial detail from patchification and the limited input resolution of CLIP, we adopt a sliding-window aggregation strategy. This generates high-resolution, spatially dense embeddings to further enhance localization granularity. Moreover, we introduce a prototype-enhanced support memory scheme to efficiently exploit the few-shot support set. This module stores compact prototypes for normal and abnormal patterns, reducing memory costs while boosting performance by fusing patch-to-prototype and image-text similarities. Extensive experiments on three benchmark datasets, including Liver CT, Retinal OCT, and Brain MRI, demonstrate that Spatial-FAD significantly outperforms state-of-the-art methods, especially in lesion segmentation. Notably, in the 4-shot scenario, our method achieves an average improvement of over 11.4% in Dice score and 1.8% in AUC. Code is available at: https://github.com/JuzhengMiao/Spatial-FAD.

1 Introduction

Few-shot CLIP methods use limited support data but often localize lesions poorly, reflecting weak fine-grained spatial grounding. Spatial-FAD combines DINO spatial priors with CLIP semantics and adds dense aggregation and compact support memory to improve localization.

1 Introduction

  • Earlier anomaly-detection methods model normality from large collections of normal samples, creating expensive data-collection requirements.Few-shot VLM approaches instead use a handful of normal and abnormal images.Source: §1 · p. 2
  • CLIP-based anomaly detection compares image-patch embeddings with prompt embeddings to create localization maps, while medical adaptations use residual adapters or learnable prompts.MediCLIP also synthesizes diverse anomaly patterns to mimic common medical diseases.Source: §1 · p. 2
  • CLIP-based methods can achieve better AUC scores yet produce lesion maps that miss locations or align poorly with lesion boundaries.The stated limitation is tied to image-level contrastive pretraining without explicit fine-grained correspondence between local patches and lesion descriptions.Source: §1 · p. 2
  • Spatial-FAD combines DINO’s spatially coherent patch representations with CLIP’s semantic alignment to improve lesion segmentation accuracy.DINO’s self-distillation and local-to-global correspondence encourage spatially coherent patch embeddings that align with object boundaries.Source: §1 · pp. 2–3
  • A DINO-derived structural affinity prior refines CLIP patch features to retain text alignment while better following lesion boundaries.Lightweight adapters further adapt the refined features to the medical domain.Source: §1 · pp. 2–3
  • Sliding-window aggregation produces spatially dense features, while compact normal and abnormal prototypes fuse support-set similarities with text scores at inference.Across three datasets in the 4-shot setting, the method reports over 11.4% average Dice and more than 1.8% average AUC improvement.Source: §1 · p. 3

2 Method

Spatial-FAD combines CLIP-based few-shot anomaly detection with DINO-derived structural guidance, high-resolution sliding-window features, and compact support prototypes. Its method refines localization while retaining text-based and support-based cues.

2.1 Problem Formulation and Baseline Architecture

  • The k-shot setup uses labeled support images and masks, CLIP vision-stage adapters, and similarities between visual patches and normal or abnormal text prompts.Training combines pixel-level Dice and focal losses with image-level binary cross-entropy.Source: §2.1 · pp. 3–4

2.2 VFM-enhanced Structural Affinity with Sliding-window Aggregation

  • DINO patch affinities guide CLIP features before adaptation, encoding shared anatomical structure and smoothing responses to better align predictions with boundaries.The affinity prior is computed from DINO patch embeddings and applied to CLIP features at each encoder stage.Source: §2.2 · p. 5
  • Sliding windows process overlapping image crops and stitch their features into a dense map, preserving fine details and improving localization of small anomalies.Overlapping regions are averaged when the crop features are placed back at their corresponding spatial locations.Source: §2.2 · p. 5

2.3 Prototype-enhanced Support Memory at Test Time

  • At inference, prototype-enhanced support memory represents normal and abnormal patterns compactly and fuses support-based predictions with text-based predictions across stages.The design combines CLIP text-alignment generalization with instance-level knowledge from the support set.Source: §2.3 · pp. 5–6

3 Experimental Results

Spatial-FAD outperforms competing few-shot methods on most benchmark settings, with especially strong gains in lesion segmentation. LiverCT ablations report added Dice gains from sliding-window aggregation, DINO structural priors, and prototype memory.

  • Evaluation covers LiverCT, RESC, and BrainMRI at 2, 4, 8, and 16 shots, using Dice for segmentation and AUC for classification.Source: §3 · p. 6
  • Dice improves by more than 8.9% on average across three datasets and shot numbers, while AUC exceeds other methods by over 1.8% with 4- and 16-shot examples.Source: Table 1 · p. 7, §3 · p. 7
  • On LiverCT, Spatial-FAD exceeds other state-of-the-art methods in Dice by at least 19.76%, 23.73%, 21.85%, and 21.86% for 2-, 4-, 8-, and 16-shot settings, respectively.With 4-shot examples, it also obtains higher Dice and AUC than other methods trained with 8- or 16-shot data.Source: §3 · p. 7
  • On LiverCT at 4 shots, sliding-window aggregation improves segmentation, DINO structural priors add 5.46% Dice, and prototype memory adds another 2.68%.The reported gains are relative to the preceding ablation configurations.Source: §3 · p. 8
  • The prototype memory is not sensitive to prototype count, although increasing abnormal-image prototypes to 50 decreases Dice by 0.67%.The authors suggest this may reflect shared lesion features, such as dark areas in CT liver tumors.Source: §3 · p. 8, Figure 3 · p. 8

4 Conclusion

Spatial-FAD targets weak spatial grounding in CLIP-based few-shot medical anomaly detection by integrating VFM priors with CLIP. Across three datasets, it improves lesion segmentation, including an 11.4% average Dice gain in the 4-shot setting.

  • CLIP-based few-shot medical anomaly detection combines strong image-text alignment with weak spatial grounding.Source: §4 · p. 9
  • Spatial-FAD integrates VFM spatial priors into CLIP and adds sliding-window aggregation and lightweight prototype-based support memory to improve localization.Source: §4 · p. 9
  • Across three datasets, Spatial-FAD achieves an 11.4% average Dice improvement in the 4-shot setting, alongside competitive average AUC performance.Source: §4 · p. 9
Loading 2609.12454v1…