PaperlayerView exact revision on arXiv

Source-linked AI summary

AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectors

Matic Fučka, Vitjan Zavrtanik, Danijel Skočaj

arXiv:2601.20524v2cs.CV

TL;DR

Zero-shot anomaly detection must localise defects without in-domain images, but VFM-based methods have lagged because of limited auxiliary-data diversity and shallow adaptation. AnomalyVFM combines synthetic anomaly generation with low-rank feature adaptation and confidence-weighted loss, achieving strong results across industrial benchmarks. The framework also generalises to medical data and supports few-shot fine-tuning, while synthetic labels and defect realism remain improvement areas.

  • Problem

    Zero-shot anomaly detection requires generalisation without in-domain images, while VFM methods face limited auxiliary-dataset diversity and shallow adaptation strategies.

  • Method

    AnomalyVFM uses a three-stage synthetic dataset generator and parameter-efficient low-rank feature adapters with a decoder and confidence-weighted pixel loss.

  • Results

    94.1% average image-level AUROC across 9 diverse industrial datasets, improving previous methods by 3.3 percentage points.

  • Takeaways & Limitations

    AnomalyVFM transforms pretrained VFMs into strong zero-shot detectors and achieves performance comparable to state-of-the-art in the few-shot regime.

  • Takeaways & Limitations

    The authors identify defect realism, anomaly-mask estimation, and image filtering as areas for further improvement.

Abstract

from arXiv · show

Zero-shot anomaly detection aims to detect and localise abnormal regions in the image without access to any in-domain training images. While recent approaches leverage vision-language models (VLMs), such as CLIP, to transfer high-level concept knowledge, methods based on purely vision foundation models (VFMs), like DINOv2, have lagged behind in performance. We argue that this gap stems from two practical issues: (i) limited diversity in existing auxiliary anomaly detection datasets and (ii) overly shallow VFM adaptation strategies. To address both challenges, we propose AnomalyVFM, a general and effective framework that turns any pretrained VFM into a strong zero-shot anomaly detector. Our approach combines a robust three-stage synthetic dataset generation scheme with a parameter-efficient adaptation mechanism, utilising low-rank feature adapters and a confidence-weighted pixel loss. Together, these components enable modern VFMs to substantially outperform current state-of-the-art methods. More specifically, with RADIO as a backbone, AnomalyVFM achieves an average image-level AUROC of 94.1% across 9 diverse datasets, surpassing previous methods by significant 3.3 percentage points. Project Page: https://maticfuc.github.io/anomaly_vfm/

1. Introduction

Zero-shot anomaly detection targets unseen object classes without in-domain images, yet VFM-based methods have lagged behind VLMs. AnomalyVFM addresses limited dataset diversity and shallow adaptation with synthetic data and parameter-efficient feature adaptation.

  • Zero-shot methods must generalise to unseen object classes without any in-domain images.
  • VLM-based approaches benefit from high-level concept knowledge, whereas pure VFMs have trailed behind despite their strong visual representations.
  • Limited diversity in auxiliary anomaly datasets restricts broadly applicable defect cues for arbitrary object classes.
  • Prior VFM adaptations usually fine-tune only a small output head with simple pixel-wise losses, leaving internal visual representations essentially unchanged.
  • AnomalyVFM combines a three-stage synthetic dataset generator with low-rank feature adapters, a lightweight decoder, and confidence-weighted pixel loss.
  • 3.3 percentage points higher image-level AUROC and 0.9 percentage points higher pixel-level AUROC were achieved than previous zero-shot methods across nine industrial benchmarks.

2. Related Work

Related work distinguishes reconstructive, discriminative, and embedding-based anomaly detection, while emphasizing the stricter data requirements of zero-shot detection. The paper positions diverse synthetic generation and VFM fine-tuning as a route to state-of-the-art performance.

  • Anomaly detection methods are commonly categorized as reconstructive, discriminative, or embedding-based.
  • Discriminative methods train on synthetic anomalies, but methods generating samples similar to training data cannot reliably cover unseen anomalies.
  • Zero-shot anomaly detection requires no normal or abnormal samples and evaluates anomalies on object instances never observed during training.
  • The paper demonstrates state-of-the-art performance by fine-tuning a pretrained VFM on a sufficiently diverse dataset.

3. Dataset Generation Scheme

The dataset generation scheme creates anomaly-free object images, inserts realistic local defects, and derives feature-based masks for filtering. It uses text-conditioned generation, foreground-constrained inpainting, and cosine-distance verification.

  • The three-stage scheme generates an initial object image, inpaints a realistic defect, and produces an anomaly map for filtering.
  • Anomaly-Free Image Generation: FLUX generates anomaly-free images from anomaly-free text prompts containing object and background categories.
  • Anomalous Image Generation: A salient-object segmentation network extracts the foreground mask, which constrains sampling of the rough anomaly region.
  • Anomalous Image Generation: Anomalous images are generated by prompting inpainting within the sampled region while preserving the remaining image.
  • Anomalous Image Generation: Anomaly descriptions are generated for each object so synthesized defects remain relevant to the object category.
  • Feature-Based Verification: Feature cosine distances between normal and anomalous images produce a distance map, thresholded into a mask and used to accept generated samples.

4. AnomalyVFM

AnomalyVFM adapts internal VFM representations and improves decoding and loss weighting for zero-shot anomaly detection. Its architecture combines feature adaptation, a convolutional decoder, confidence prediction, and image-level and segmentation outputs.

  • Feature Adaptation Module and Decoder: AnomalyVFM injects LoRA feature adaptation modules into each VFM transformer block's query, value, and output projections.The adaptation modules enable internal-layer adaptation while remaining parameter-efficient.
  • Feature Adaptation Module and Decoder: A convolutional decoder upsamples final-block features and outputs both an anomaly segmentation map and a confidence map.It uses two sequential convolution, GroupNorm, ReLU, and bilinear upsampling blocks followed by a final convolutional layer.
  • Confidence-Weighted Loss: The confidence-weighted segmentation loss is designed to handle noisy generated masks and ambiguities in ground-truth masks.The confidence map defines C = 1 + exp(c), with α = 0.1.
  • Evaluation: Table 1 reports average results across 9 industrial datasets and compares different VFMs, synthetic datasets, and feature adaptors.The table marks improvement over the baseline in green; SD denotes Synthetic dataset and FA denotes Feature Adaptors.
  • Training and Inference: The full training objective sums the segmentation loss and image-level loss, while inference returns both segmentation and image-level anomaly scores.The model directly produces the output anomaly segmentation mask Mo and image-level anomaly score Ao at inference.

5. Experiments

AnomalyVFM is evaluated across diverse industrial and medical datasets, foundation-model settings, and comparisons with zero-shot and few-shot methods. It improves detection and localisation performance while generalising beyond its training domain.

  • Evaluation setup: AnomalyVFM is evaluated on 9 industrial and 9 medical anomaly detection datasets, using automatically generated training data.The industrial benchmarks include MVTec AD, VisA, BTAD, MPDD, RealIAD, KSDD, KSDD2, DAGM, and DTD-Synthetic; the medical benchmarks include HeadCT, BrainMRI, BR35H, ISIC, ClinicDB, ColonDB, Kvasir, Endo, and TN3K.
  • VFM generalisation: Across nine industrial datasets, the proposed adaptation improves image-level AUROC by 6.1 percentage points and pixel-level AUROC by 10.7 percentage points.The comparison spans DINOv2, DINOv3, and RADIO under different training-dataset and adaptation-strategy settings.
  • Zero-shot comparison: 3.3 percentage points: AnomalyVFM exceeds Bayes-PFL in industrial image-level AUROC and also improves anomaly localisation.It improves pixel-level AUROC by 0.9 percentage points, while AdaCLIP remains ahead in pixel-level F1-Max.
  • Medical-domain evaluation: 1.2 percentage points: AnomalyVFM improves previous methods in medical pixel-level AUROC without fine-tuning on medical data.Its medical detection results are competitive, and the qualitative comparisons report sharper masks than Bayes-PFL and AdaCLIP.
  • Few-shot transfer: AnomalyVFM achieves the highest image-level AUROC in all MVTec AD few-shot settings and in the 1-shot VisA setting.Despite being designed for zero-shot detection, it matches or surpasses recent few-shot methods with minimal fine-tuning and no architecture changes.

6. Ablation study

Ablations show that dataset filtering, intelligent anomaly placement, and confidence-weighted supervision materially affect anomaly detection and localisation performance.

  • Image generation model: Replacing FLUX with QWEN-Image or WAN causes only slight image-level AUROC decreases of 0.1 and 0.4 percentage points, respectively.The corresponding pixel-level AUROC decreases are 0.5 and 2.1 percentage points, respectively.
  • Dataset generation: Omitting dataset filtering decreases image-level AUROC by 3.8 percentage points and pixel-level AUROC by 14.6 percentage points.The ablation attributes this result to generated images that may not actually contain relevant anomalies.
  • Anomaly location: Using the whole image instead of the foreground for anomaly placement decreases image-level AUROC by 1.4 percentage points and pixel-level AUROC by 5.8 percentage points.This setting can place defects in the background, encouraging the model to attend beyond the object.
  • Confidence loss: Removing the confidence loss decreases image-level AUROC by 0.6 percentage points and pixel-level AUROC by 2.0 percentage points.The result supports the role of confidence-weighted supervision in the adaptation strategy.
  • Efficiency: Model training requires about two hours on an A100 GPU.The supplied limitation passage reports this training cost alongside a separate inference-time comparison.

7. Conclusion

AnomalyVFM transforms pretrained VFMs into zero-shot anomaly detectors through diverse synthetic data and parameter-efficient backbone adaptation. It generalises across unseen classes and domains, while future work targets more realistic defects and labels.

  • Method: AnomalyVFM combines a three-stage synthetic dataset generator with low-rank adapters and a confidence-weighted loss.The adapters are inserted throughout the backbone to refine VFM representations with minimal parameters.
  • Results: 94.1% average image-level AUROC across 9 diverse industrial datasets, improving previous methods by 3.3 percentage points.The framework also achieves performance comparable to state of the art in the few-shot regime after fine-tuning on a few normal samples.
  • Implications: AnomalyVFM enables VFMs to generalise to unseen object classes and outperform existing VLM-based methods in the zero-shot regime.The results also indicate that it can serve as a backbone for future few-shot and full-shot models.
  • Future work: Further work is needed to improve defect realism and the resulting labels.The conclusion also identifies depth integration as a possible direction for zero-shot RGBD anomaly detection.

Anomaly Detectors

The appendix extends the main manuscript with limitations, dataset-generation analysis, competing-method results, ablations, qualitative results, and data-generation details.

  • Appendix contents: The appendix is organised into sections covering limitations, dataset generation, competing methods, synthetic-dataset details, ablations, qualitative results, and generation data.These sections provide supporting information beyond the scope of the main manuscript.

A. Limitations

The main practical limitation is the time required to generate AnomalyVFM’s synthetic dataset. Medical-image generation is another boundary, because pretrained generators did not produce sufficiently realistic medical images for training.

  • Generation cost: Synthetic dataset generation takes approximately one day on an A100 GPU, while model training requires about two hours.The generated dataset is reusable across VFMs, and fewer than 10,000 images can shorten generation time.
  • Medical-data scope: Pretrained image-generation models failed to produce realistic medical images suitable for zero-shot anomaly-detection training.Fine-tuning the generator on an auxiliary medical-imaging dataset may improve data quality for this setting.

B. Discussion about dataset generation phase

The synthetic dataset generation process is effective but remains improvable in data quality, tag coverage, and transfer to VLM-based methods. Its diversity substantially exceeds that of existing datasets, while filtering and generation choices define important quality boundaries.

  • Quality improvements: Synthetic dataset generation could be improved through better anomaly-mask estimation, image filtering, and [Object] tag content.Some images without anomalies still pass the filtering step, and the authors leave further tag improvements for future work.
  • Data controls: Manual removal of evaluation-set [Object] tags was used to prevent data leakage, while [Anomaly] and [Texture] tags were retained as general concepts.The dataset’s object, anomaly, and texture tag inventories are documented in the accompanying tables.
  • VLM comparison: Retraining VLM methods with the proposed synthetic dataset helps some methods but does not significantly alter their results.This indicates that VLM methods do not suffer from the same inadequate-data-diversity problem as VFMs.
  • Dataset diversity: The generated dataset contains 204 anomaly types across possible objects and backgrounds, compared with 73 anomaly types in MVTec AD.Generated anomalies occupy 2.52% of the image on average.

E. Additional Ablation Studies

Additional ablations examine filtering, dataset diversity and size, adapter configuration, model size, and qualitative segmentation behavior. Performance generally benefits from more diverse or numerous generated images, while some configuration changes reduce performance.

  • Dataset diversity: Performance consistently rises as the number of [Object] tags increases, with 20 tags performing similarly to training on MVTec AD’s 15 objects.The authors hypothesize that adding more tags could improve performance further.
  • Dataset size: Performance increases steadily as the generated training set grows from 100 to 10,000 images.The experiments used 10,000 generated images by default, and further scaling was not tested to match related methods’ training-set sizes.
  • Model size: Replacing RADIO’s ViT-L backbone with ViT-B decreases image-level AUROC by 1.8 percentage points and pixel-level AUROC by 1.2 percentage points.Using ViT-H decreases image-level AUROC by 0.6 percentage points and pixel-level AUROC by 1.2 percentage points.
  • Adapter configuration: Adding LoRA adapters to all linear layers produces the largest performance dip among the tested adapter layouts.The authors hypothesize that this prevents information from being passed globally rather than locally.
  • Qualitative results: Qualitative examples show AnomalyVFM detecting anomalies across a wide range of objects through predicted segmentation masks compared with ground truth masks.Figure 6 presents the image, predicted anomaly segmentation, and ground-truth mask in successive rows.
Loading 2601.20524v2…