Source-linked AI summary
Cross-Encoder Anchor Refinement for Zero-Shot Anomaly Detection
Kaisong Ji, Ziyang Xu, Chang Bian, Yu Zhou, Wenyu Liu, Xinggang Wang
TL;DR
Zero-shot anomaly detection seeks to transfer anomaly discrimination to unseen categories while localizing subtle defects, but constructing references from contextual and local evidence remains a challenge. CEAR refines ViT spatial anchors using independently encoded image evidence before prototype construction. Across 13 industrial and medical datasets, it achieves 57 best results in 68 evaluations, including 41 wins in 44 pixel-level comparisons.
Problem
Zero-shot anomaly detection must transfer discrimination to unseen categories while detecting subtle defects, motivating references that combine contextual representations with complementary local evidence.
Source: §1 · p. 1Method
CEAR uses ViT spatial anchors to retrieve evidence from a shared memory built by an independent encoder, then refines anchors before image-conditioned queries aggregate them into prototypes.
Source: §1 · p. 2Results
CEAR achieves 57 best results across 68 dataset–metric evaluations and exceeds all other state-of-the-art methods in 41 of 44 pixel-level comparisons.
Source: §1 · p. 2, §5 · p. 12Takeaways & Limitations
The results show cross-encoder interaction at the anchor stage can integrate contextual and local visual representations for fine-grained anomaly localization across domains.
Source: §1 · p. 2, §5 · p. 12Target categories are excluded from anomaly adaptation, and each target image is processed independently without target-domain parameter updates.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Zero-shot anomaly detection aims to transfer anomaly discrimination from auxiliary training domains to unseen categories while localizing subtle defects. A key challenge is to construct discriminative references that combine contextual representations with complementary local evidence. We present Cross-Encoder Anchor Refinement (CEAR), which connects two visual encoding paths for the same image at the anchor stage, before normal and anomalous prototypes are constructed. An independent convolutional encoder pretrained for industrial product recognition builds an image-specific memory shared across selected vision transformer (ViT) layers. Spatial anchors read from ViT patch features query this memory through cross-attention and undergo residual refinement. Image-conditioned normal and anomalous queries then gate anchor aggregation to construct prototypes for anomaly scoring. Across 13 industrial and medical datasets, CEAR achieves the best performance in 57 of 68 evaluations spanning image-level detection and pixel-level localization. Notably, CEAR surpasses all other state-of-the-art methods in 41 of 44 pixel-level comparisons. Across 11 localization datasets, mean gains in pixel-level maximum F1 score and average precision are 4.45 and 3.79 percentage points, respectively. This study shows how cross-encoder interaction at the anchor stage can integrate complementary visual representations for fine-grained anomaly localization across domains.
1. Introduction
Zero-shot anomaly detection must transfer anomaly discrimination to unseen categories while localizing subtle defects, motivating references that combine contextual representations with local evidence. CEAR refines spatial anchors through cross-encoder interaction before constructing prototypes, and its evaluations show broad gains, especially in pixel-level localization.
1. Introduction
- Zero-shot anomaly detection transfers anomaly discrimination from auxiliary supervision to unseen categories, where rare, diverse defects make precise localization challenging.The challenge includes detecting fine cracks, small notches, and low-contrast regions within otherwise normal structures.Source: §1 · p. 1
- Qualitative comparisons show CEAR concentrating responses more closely on annotated anomalies while reducing activations on some normal structures and background regions.The examples include industrial and medical images.Source: Figure 1 · p. 2, §1 · p. 2
- CEAR connects ViT-derived spatial anchors to local evidence from an independently encoded image before normal and anomalous prototypes are constructed.Image-conditioned queries gate anchor aggregation, and the evaluation spans 13 industrial and medical datasets.Source: §1 · p. 2
- CEAR achieves 57 best results across 68 dataset–metric comparisons and exceeds every other state-of-the-art method in 41 of 44 pixel-level comparisons.Across 11 localization datasets, mean gains are 4.45 percentage points in pixel-level F1-max and 3.79 points in AP.Source: §1 · p. 2
2. Related Work
Prior work develops transferable semantic references, varied reference settings, and methods for preserving local visual evidence. CEAR places independently encoded local evidence into anchor refinement before prototype construction.
2.1. Vision–Language References for Anomaly Detection
- Vision–language anomaly detectors vary prompt construction and local-context use, while recent methods also improve fine-grained discrimination through prompt flexibility and regional cues.Examples include distribution modeling, sample-dependent prompts, frequency-aware adaptation, regional visual prompting, and foreground–background separation [13].Source: §2.1 · pp. 2–3
2.2. Reference Requirements and Transfer Settings
- Anomaly detectors differ in target-domain access, including test-sample interactions, normal reference images, or target-category normal training data.CEAR instead uses auxiliary anomaly data and processes unseen target categories one image at a time, without target-domain normal references or test-sample interactions.Source: §2.2 · p. 3
2.3. Dense Representations and Local Evidence
- Dense-representation methods preserve or recover spatial detail through feature organization, upsampling, or CNN–ViT interactions; CEAR uses independently encoded convolutional features as local evidence before prototype construction.Its ConvNeXt V2 features are pretrained for product recognition on Real-IAD and condition the output of a frozen ViT.Source: §2.3 · p. 3
2.4. Visual Prototypes and Conditional Refinement
- CEAR differs from VisualAD and ReCAP in sourcing local evidence from an independent encoder and applying it to anchors before prototype construction.The frozen-backbone outputs feed layerwise branches that share local memory but retain independent refinement parameters.Source: §2.4 · p. 3
3. Method
CEAR processes the same image through ViT and CNN encoders, then uses CNN-derived local evidence to refine ViT spatial anchors before constructing normal and anomalous prototypes. Patch responses are combined into localization maps and image-level scores, while joint training supervises detection, segmentation, and region separation.
3.1. Problem Formulation and Framework Overview
- CEAR processes each image through ViT and CNN paths, using spatial anchors to connect their representations before constructing normal and anomalous prototypes.The model compares projected patch features with the prototypes to produce layerwise anomaly responses, which form the pixel map and image score.Source: §3.1 · p. 3
3.2. Visual Representations and Shared Local Memory
- Image-conditioned queries derive from deepest-layer spatial means, maxima, and standard deviations; multiscale CNN features are aligned and fused into a shared local memory.The memory is computed from the current image and shared across selected ViT layers, each with independent refinement parameters.Source: §3.2 · pp. 4–5
3.3. Cross-Encoder Anchor Refinement and Discriminative Prototypes
- SER forms anchors from patch features, CCAR refines them through self-attention, CNN-memory cross-attention, and a feed-forward update, then QGPC gates their aggregation into class-specific prototypes.The two branches share spatial readout and conditional-refinement parameters; queries and gating weights distinguish normal from anomalous prototypes.Source: §3.3 · p. 5
3.4. Patch Comparison and Anomaly Scoring
- Patch anomaly responses measure the cosine-similarity difference between projected patch features and anomalous versus normal prototypes; upsampled layer responses are summed into a map.The image score averages a fraction of the map’s highest-response pixels, retaining localized anomaly evidence while reducing sensitivity to a single extreme response.Source: §3.4 · pp. 5–6
3.5. Joint Training Objective
- Joint training combines image-level prediction, segmentation, query separation, and region-level margins, assigning deeper branches overall region separation and shallower branches emphasis on difficult local responses.The objective also uses a Tversky loss for region completeness in the fused anomaly map; anomaly adaptation updates modeling modules while keeping both encoder backbones frozen.Source: §3.5 · p. 6
4. Experiments
The experiments assess CEAR’s generalization across industrial categories and medical imaging domains, then use controlled comparisons and qualitative analyses to examine its components and applicability.
4. Experiments
- The evaluation tests cross-category and cross-domain generalization, then isolates input resolution, industrial pretraining, multiscale memory, conditional interactions, and layerwise supervision.Qualitative analyses complement the controlled comparisons and characterize the model’s range of applicability.Source: §4 · p. 6
4.1. Experimental Setup
The evaluation spans industrial and medical anomaly-detection and localization benchmarks, with controlled cross-category and cross-dataset protocols. Results are reported using image- and pixel-level metrics, with implementation details specified for adaptation and pretrained encoders.
4.1.1. Datasets and Training Protocol
- Five industrial benchmarks support image-level detection and pixel-level localization, while six medical datasets cover detection or localization.MVTec AD and VisA alternate as auxiliary and target domains; medical detection uses COVID-19 Radiography, Br35H, and Brain AD, while localization includes CVC-ClinicDB, Endo, Kvasir, ISIC, TN3K, and Brain AD.Source: §4.1.1 · pp. 6–7
4.1.2. Evaluation Metrics
- Image-level evaluation uses AUROC, F1-max, and AP; pixel-level evaluation additionally uses PRO, with category macroaveraging and three-seed mean and standard deviation.Source: §4.1.2 · p. 7
4.1.3. Implementation Details
- MoonViT-SO-400M supplies patch features at 672 × 672 resolution from layers {7, 14, 21, 27}, while ConvNeXt V2-Base provides multiscale local features and is separately pretrained on Real-IAD product categories.Both encoder bodies remain frozen during anomaly adaptation; the shared final MoonViT layer normalization and anomaly modules are updated. Adaptation uses AdamW for three epochs, batch size 4, and learning rate 10−4; image scores average the top 1% of anomaly-map responses.Source: §4.1.3 · pp. 7–8
4.2. Comparison with State-of-the-Art Methods
CEAR leads the broad benchmark comparison, with its strongest advantage in pixel-level localization across industrial and medical datasets. Medical results show transfer under a shared VisA-trained model, while anomaly maps illustrate spatial correspondence with defects and lesions.
4.2.1. Industrial Anomaly Detection and Localization
- 57 of 68 dataset–metric comparisons are best for CEAR, including 56 outright wins and one tie; it exceeds other methods in 41 of 44 pixel-level comparisons.The complete-system comparison covers 13 industrial and medical datasets; CEAR has the highest pixel AUROC on all 11 localization datasets, with mean gains of 2.20, 4.45, 3.79, and 2.39 percentage points in pixel AUROC, F1-max, AP, and PRO, respectively.Source: §4.2.1 · p. 8
4.2.2. Transfer to Medical Imaging
- CEAR achieves the highest scores in all 24 pixel-level comparisons across six medical localization datasets.All six datasets use the same VisA-trained model without target-domain parameter updates; examples include pixel AP rising from 50.7 to 60.8 on Endo and pixel F1-max from 50.0 to 60.1 on Kvasir.Source: §4.2.2 · p. 8
- On illustrated industrial and medical samples, CEAR’s high-response regions correspond closely to annotated defects and lesions, while other methods show scattered, incomplete, or overextended activations.Heatmaps are normalized individually, so the qualitative comparison concerns spatial distribution and ground-truth correspondence; Table 1 quantifies aggregate performance.Source: §4.2.2 · p. 8
4.3. Ablation Studies
The VisA ablations isolate resolution, local-memory pretraining and scale, refinement design, and layer-specific supervision. They find that local evidence is most useful when it conditions anchor refinement, while complementary scales and layers contribute differently to localization.
4.3. Ablation Studies
- VisA ablations follow the MVTec AD-to-VisA protocol, exclude target categories from anomaly adaptation, and hold training and evaluation constant except for the studied factor.The default input resolution is 672 × 672.Source: §4.3 · p. 8
4.3.1. Backbone Resolution and the Contribution of Local Evidence
- At 672 × 672, CEAR gains 0.4, 1.6, 1.3, and 1.0 percentage points over MoonViT-only in pixel AUROC, F1-max, AP, and PRO.It also improves image AUROC, F1-max, and AP by 2.5, 2.6, and 2.6 points, respectively; CEAR improves all four pixel metrics at each tested resolution and outperforms score fusion and feature concatenation.Source: Table 2 · p. 10, §4.3.1 · p. 10
- At matched 672 resolution, CEAR suppresses responses on a normal metal-nut ring and covers more of an annotated skin lesion than the single-backbone configurations.The examples distinguish changes from denser spatial sampling from those associated with conditional interaction.Source: §4.3.1 · p. 10, Figure 4 · p. 10
4.3.2. Industrial Pretraining and Multiscale Memory
- Industrial pretraining improves every localization metric over random and generic pretraining; the full multiscale pyramid gives the strongest localization performance.The finest scale preserves small-defect detail but can respond to normal texture, while the coarsest provides broader component context.Source: §4.3.2 · p. 10, Table 3 · p. 10, Table 4 · p. 10
4.3.3. Refinement Architecture and Interaction Stage
- Removing CNN cross-attention reduces localization performance, while anchor self-attention and the feed-forward update further improve evidence organization and transformation.Independent layer-specific refinement outperforms full parameter sharing; direct use of refined anchors has the highest scores on six metrics, although memory pre-normalization gives slightly higher PRO.Source: §4.3.3 · pp. 10–11, Table 6 · p. 12
4.3.4. Supervision across Layers
- Deeper branches preserve overall discrimination but have lower small-defect recall than the full model, while shallow-only responses are more susceptible to normal texture.The shallow hard-region constraint suppresses local false activations, the deep region margin promotes foreground–background separation, and fused-map supervision supports regional completeness; removing shallow region loss slightly raises PRO but lowers other localization metrics.Source: §4.3.4 · pp. 11–12, Table 7 · p. 12
4.4. Qualitative Analysis
The qualitative comparisons show CEAR’s anomaly maps following annotated regions more closely than the other methods shown. They examine both individual layer responses and their fused maps on industrial and medical examples.
4.4. Qualitative Analysis
- Across the examples, CEAR better distinguishes defects from normal appearance and more closely follows annotated regions than the other methods shown.The figure groups examples into normal textures with small anomalies, industrial structural defects, medical anomalies, and local boundary differences.Source: §4.4 · p. 12
- Figure 6 compares predictions from layers 7, 14, 21, and 27 with the pointwise sum of their score maps from the same trained checkpoint.The selected examples are two industrial and two medical cases from domains excluded from anomaly adaptation.Source: §4.4 · p. 12
- The layerwise visualizations show spatial differences among individual responses and the fused maps on the selected industrial and medical examples.The examples illustrate the layerwise behavior discussed in Section 4.3.4.Source: §4.4 · p. 12
5. Conclusion
CEAR refines ViT-derived spatial anchors with local evidence from a shared memory before prototype construction. Across 13 industrial and medical datasets, it achieves 57 best results among 68 evaluations, including 41 outright wins in 44 pixel-level comparisons.
5. Conclusion
- CEAR uses an independent encoder’s shared local memory to refine ViT-derived spatial anchors before queries aggregate them into normal and anomalous prototypes.The conclusion identifies anchor-stage cross-encoder interaction as a means of integrating complementary visual representations for fine-grained anomaly localization across domains.Source: §5 · p. 12
- Across 13 datasets, CEAR achieves 57 best results among 68 evaluations, including 41 outright wins in 44 pixel-level comparisons.Across 11 localization datasets, mean gains over each dataset’s strongest competing result are 4.45 percentage points in pixel F1-max and 3.79 points in AP.Source: §5 · p. 12
- The shared memory adds computation, motivating future work on memory efficiency and stability under domain shifts.Source: §5 · p. 12