Source-linked AI summary
VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer
Yanning Hou, Peiyuan Li, Zirui Liu, Yitong Wang, Yanran Ruan, Jianfeng Qiu, Ke Xu
TL;DR
Zero-shot anomaly detection must identify and localize anomalies in unseen categories without target-category anomaly samples, but existing VLM-based approaches depend on text encoders and cross-modal alignment. VisualAD removes the text branch by learning normal and abnormal tokens within a frozen Vision Transformer, adding spatial cross-attention and self-alignment. It achieves state-of-the-art image- and pixel-level performance across unseen industrial and medical categories under the stated zero-shot protocol.
Problem
Zero-shot anomaly detection evaluates unseen categories without category-specific training, addressing the need to generalize across industrial and medical domains under domain shift.
Method
VisualAD inserts learnable normal and abnormal tokens into a frozen Vision Transformer and uses SCA for spatial evidence and SAF for patch-feature recalibration.
Results
VisualAD delivers state-of-the-art image- and pixel-level performance on unseen industrial and medical categories without a text branch or cross-modal alignment.
Takeaways & Limitations
Discriminative anomaly features can be learned from visual cues, with token learning, SCA/SAF, and multi-layer fusion each contributing to detection and localization.
Takeaways & Limitations
The protocol trains on visible industrial categories and evaluates directly on disjoint unseen categories from other industrial or medical datasets, with substantial domain shifts.
Abstract
from arXiv · showhide
Zero-shot anomaly detection (ZSAD) requires detecting and localizing anomalies without access to target-class anomaly samples. Mainstream methods rely on vision-language models (VLMs) such as CLIP: they build hand-crafted or learned prompt sets for normal and abnormal semantics, then compute image-text similarities for open-set discrimination. While effective, this paradigm depends on a text encoder and cross-modal alignment, which can lead to training instability and parameter redundancy. This work revisits the necessity of the text branch in ZSAD and presents VisualAD, a purely visual framework built on Vision Transformers. We introduce two learnable tokens within a frozen backbone to directly encode normality and abnormality. Through multi-layer self-attention, these tokens interact with patch tokens, gradually acquiring high-level notions of normality and anomaly while guiding patches to highlight anomaly-related cues. Additionally, we incorporate a Spatial-Aware Cross-Attention (SCA) module and a lightweight Self-Alignment Function (SAF): SCA injects fine-grained spatial information into the tokens, and SAF recalibrates patch features before anomaly scoring. VisualAD achieves state-of-the-art performance on 13 zero-shot anomaly detection benchmarks spanning industrial and medical domains, and adapts seamlessly to pretrained vision backbones such as the CLIP image encoder and DINOv2. Code: https://github.com/7HHHHH/VisualAD
1. Introduction
VisualAD challenges the necessity of language in zero-shot anomaly detection, showing that visual prototypes can retain performance with far fewer parameters and smoother evaluation. It then replaces the text branch with learnable ViT tokens, spatial cross-attention, and self-alignment for anomaly detection and localization.
- A visual-only variant of AnomalyCLIP achieved comparable or slightly better results with 99% fewer parameters than its text-based counterpart.
- Zero-shot anomaly detection addresses costly collection of category-specific normal or anomalous data for new industrial and medical categories.
- The visual-only variant improved steadily with smoother evaluation curves, whereas AnomalyCLIP fluctuated noticeably across epochs.
- VisualAD inserts learnable normal and anomaly tokens into a frozen Vision Transformer, where self-attention lets them interact with patch tokens and guide anomaly-relevant cues.
- SCA injects spatial evidence into tokens, while SAF recalibrates patch features to support stable multi-layer alignment and improved localization.
- The paper summarizes contributions spanning the removal of text dependence, ViT-only token learning, spatial and feature-alignment modules, and zero-shot evaluation across industrial and medical benchmarks.
2. Related work
Related work in anomaly detection includes embedding-based, discriminative, and reconstruction-based methods, while zero-shot approaches divide into training-free methods and methods trained on auxiliary data. These approaches trade off generalization, localization, computational cost, and auxiliary alignment.
- Unsupervised anomaly detection methods use embedding-based, discriminative, or reconstruction-based strategies to distinguish anomalies from normal samples.
- Training-free zero-shot methods include WinCLIP, which repeatedly encodes multigranularity features, and SAA, which offers precise localization at higher computational cost.
- A second zero-shot family trains lightweight modules on auxiliary data while keeping the target domain unseen, including methods that align vision and text.
3. Approach
VisualAD performs zero-shot anomaly detection with two learnable visual tokens inserted into a frozen ViT, using multi-layer spatial alignment and patch recalibration to produce fused anomaly maps without a text branch.
- Problem Setting: VisualAD follows a zero-shot protocol that trains on visible categories and evaluates directly on disjoint unseen industrial or medical categories.The setting excludes category-specific fine-tuning and includes substantial domain shifts between auxiliary training and test datasets.
- Visual Tokenization: Two learnable global tokens encode anomaly and normality within the visual feature space of a frozen ViT.They are inserted alongside the class and image patch tokens, and intermediate layers are selected for multi-layer processing.
- Spatial-Aware Cross-Attention: SCA augments patch features with positional information, aggregates localized evidence through anchor queries, and injects token-guided spatial features into the global tokens.The gating mechanism selectively adds anchor-specific information while preserving each token’s semantic role.
- Self-Alignment and Scoring: SAF nonlinearly recalibrates patch features so they align with the evolving normal and anomaly tokens before cosine-contrast anomaly scoring.Layer-wise maps are formed from the enhanced tokens and recalibrated patches, then reshaped and upsampled.
- Anomaly Scoring and Training: The final image-level score averages the top-k most anomalous pixels, with k set to the floor of 1% of all pixels.Training updates the tokens, SCA modules, and per-layer transformations while keeping the ViT backbone frozen.
- Inference Pipeline: Training and inference share the same multi-layer pipeline, including SCA, per-layer transformations, fusion, and top-k aggregation, without text or cross-modal alignment.This keeps the deployed computation path consistent with the training path.
4. Experiments
VisualAD is evaluated on 13 industrial and medical benchmarks against established ZSAD methods, with experiments covering backbone choice, module contributions, anchor counts, layer ensembles, and representation evolution. The results show strong performance, complementary CLIP and DINOv2 strengths, and benefits from spatial alignment, self-alignment, multi-layer features, and appropriately sized anchor sets.
- Experimental Setup: Experiments span 13 real-world datasets across industrial and medical domains, using image-level and pixel-level anomaly metrics.Classification uses AUROC, F1-max, and AP; segmentation additionally uses pixel-level AUROC, F1-max, AP, and PRO.
- Performance Comparison: VisualAD achieves state-of-the-art performance on almost all datasets at both image and pixel levels, with CLIP favoring classification and DINOv2 delivering stronger pixel-wise segmentation.On industrial datasets, VisualAD with CLIP ViT-L/14@336px attains the best results across all classification metrics.
- Performance Comparison: VisualAD produces more accurate and complete anomaly localization than competing methods, especially on medical categories where boundaries are clearer and false positives are fewer.The qualitative comparison covers representative industrial and medical datasets.
- Ablation Studies: Removing SCA, SAF, or individual loss objectives degrades performance, while the full model achieves the best image-level and pixel-level results.SCA supports spatial alignment and fine-grained evidence aggregation; SAF recalibrates high-level semantics, and the hybrid losses balance classification, region consistency, and feature separability.
- Ablation Studies: Larger pretrained backbones and higher input resolutions generally improve pixel- and image-level performance, with CLIP ViT-L/14@336px strongest for sample-level results and DINO ViT-g/14 strongest for pixel-level precision.The gains taper at the largest scales, while CLIP and DINOv2 exhibit different strengths.
- Ablation Studies: Four SCA anchor queries provide the strongest overall pixel-level results, while larger anchor sets introduce redundancy and dilute spatial focus.Image-level gains from m = 16 over m = 4 remain within one to two percentage points, motivating m = 4 as the default.
- Ablation Studies: Combining patch features from layers {6,12,18,24} improves stability and accuracy over single-layer representations by capturing complementary information across scales.Among individual layers, layer {18} provides the best balance between local detail and global semantics.
- Visualization: SAF transforms weakly separated visual representations into a strongly discriminative space, raising PCA’s first-component variance from 9.0% with learnable tokens to 89.1%.After SAF, anomaly samples become more compact and farther from normal samples.
5. Conclusion
VisualAD performs zero-shot anomaly detection using visual tokens and spatial grounding within a frozen Vision Transformer, without a text branch or cross-modal alignment. It generalizes to unseen industrial and medical categories while achieving state-of-the-art image- and pixel-level performance.
- VisualAD inserts learnable normal and abnormal global tokens into a frozen Vision Transformer and grounds them with spatial evidence through SCA and SAF.The framework is trained only on auxiliary industrial data and omits text and cross-modal alignment.
- The method generalizes directly to unseen industrial and medical categories while delivering state-of-the-art image- and pixel-level performance.
- Ablations show that token learning, SCA/SAF, and multi-layer fusion each contribute indispensably to detection and localization.