Source-linked AI summary
APRIL-GAN: A Zero-/Few-Shot Anomaly Classification and Segmentation Method for CVPR 2023 VAND Workshop Challenge Tracks 1&2: 1st Place on Zero-shot AD and 4th Place on Few-shot AD
Xuhai Chen, Yue Han, Jiangning Zhang
TL;DR
Industrial inspection needs models that can adapt across product categories with no or few normal reference images because collecting extensive category-specific training data is difficult. APRIL-GAN extends CLIP with linear feature-mapping layers for zero-shot anomaly maps and memory banks for few-shot reference features, achieving first place in zero-shot evaluation and fourth overall in few-shot evaluation, with leading segmentation and classification results.
Problem
Collecting many training images for every industrial product category is difficult, motivating zero-/few-shot anomaly detection for practical inspection.
Method
APRIL-GAN adds linear layers to map CLIP image features into the text-feature space for anomaly maps and uses reference-image memory banks for few-shot comparison.
Results
APRIL-GAN ranked first in the zero-shot VAND track and fourth overall in few-shot, while achieving 0.0489 higher segmentation F1 than the second-ranked zero-shot team and first-place few-shot classification with F1 0.8687.
Takeaways & Limitations
The approach supports a joint zero-/few-shot model for anomaly classification and segmentation across diverse industrial product categories.
Takeaways & Limitations
On MVTec AD, the method’s precise anomaly regions can be smaller than ground truth, lowering its PRO metric despite correct localization.
Abstract
from arXiv · showhide
In this technical report, we briefly introduce our solution for the Zero/Few-shot Track of the Visual Anomaly and Novelty Detection (VAND) 2023 Challenge. For industrial visual inspection, building a single model that can be rapidly adapted to numerous categories without or with only a few normal reference images is a promising research direction. This is primarily because of the vast variety of the product types. For the zero-shot track, we propose a solution based on the CLIP model by adding extra linear layers. These layers are used to map the image features to the joint embedding space, so that they can compare with the text features to generate the anomaly maps. Besides, when the reference images are available, we utilize multiple memory banks to store their features and compare them with the features of the test images during the testing phase. In this challenge, our method achieved first place in the zero-shot track, especially excelling in segmentation with an impressive F1 score improvement of 0.0489 over the second-ranked participant. Furthermore, in the few-shot track, we secured the fourth position overall, with our classification F1 score of 0.8687 ranking first among all participating teams.
1. Introduction
Industrial anomaly detection usually requires category-specific models and many normal images, making rapid zero-/few-shot adaptation important for practical inspection. Building on CLIP-based language-guided AD, APRIL-GAN adds linear mappings and memory-based comparisons for zero- and few-shot classification and segmentation.
- Industrial anomaly detection identifies abnormal images and regions using models trained only on anomaly-free images.
- Category-specific methods typically rely on large collections of normal reference images, which are difficult to obtain across diverse industrial products.
- Zero-/few-shot settings are therefore important for bringing anomaly detection to practical applications.
- WinCLIP demonstrates rapid adaptation across categories with few or no normal images through language-guided anomaly detection and window-based segmentation.
- APRIL-GAN uses CLIP with extra linear layers for zero-shot segmentation and memory banks for comparing reference and test features in few-shot detection.
- The method ranked first in the zero-shot VAND track and fourth in the few-shot track, with strong classification and segmentation results.
2. Methodology
APRIL-GAN adapts CLIP for anomaly classification and segmentation by mapping multi-stage image features into CLIP’s joint embedding space and comparing them with text or reference features. In the few-shot setting, memory banks provide reference-based anomaly maps that are combined with the zero-shot outputs.
- Zero-shot AD Setting: CLIP supplies the vision-language foundation, while added linear layers extend its zero-shot classification capability to anomaly segmentation.
- Zero-shot AD Setting: State- and template-level prompt ensembles describe normal and abnormal objects for binary anomaly classification.
- Zero-shot AD Setting: Image features from encoder stages preserve spatial information and are mapped into the joint embedding space before comparison with text features.
- Zero-shot AD Setting: For ViT, the encoder is divided into four stages, with one linear layer per stage mapping output features into the joint embedding space.
- Zero-shot AD Setting: The mapped stage features are compared with text features to produce anomaly maps, which are combined across stages and supervised using focal and dice losses.
- Few-shot AD Setting: Few-shot anomaly segmentation stores multi-layer features from normal reference images in stage-specific memory banks.
- Few-shot AD Setting: Test-image features are compared with corresponding memory-bank features using cosine similarity, and the resulting anomaly maps are summed.
- Few-shot AD Setting: The few-shot feature-comparison map is added to the zero-shot map without fine-tuning the linear layers on reference images.
3. Experiments
Experiments evaluate the method on the VAND challenge, MVTec AD, and VisA under zero- and few-shot settings, using cross-dataset training and no few-shot linear-layer fine-tuning. Results show strong challenge performance, improved segmentation comparisons, and reduced false detections with reference images, while zero-shot localization can be overly narrow or misclassify complex normal regions.
- Experimental setup: The VAND challenge evaluates zero-shot and few-shot tracks, while standard experiments use MVTec AD and VisA with cross-dataset training for zero-shot evaluation.For zero-shot testing, linear layers are trained on the other dataset; few-shot experiments do not additionally fine-tune them with reference images.
- Experimental setup: The default backbone is CLIP ViT-L/14 at 336 resolution, with four additional linear layers mapping stage features into the joint embedding space.Reference-image features from the four stages are separately stored in the few-shot track.
- Qualitative results: Qualitatively, zero-shot predictions localize simple-category anomalies accurately but sometimes label normal regions as anomalous in complex categories such as macaroni and PCBs.Reference images reduce these false detections, narrowing anomalous regions for macaroni and lowering normal-component scores for PCBs.
- VAND challenge results: 0.3431 segmentation F1 and 0.7782 classification F1 place the method first overall in the zero-shot VAND track.It ranks first in anomaly segmentation and fifth in anomaly classification individually.
- VAND challenge results: 0.8687 classification F1 ranks first in the few-shot VAND track, although the method ranks fourth overall and exceeds the second-ranked team by 0.0207.The zero-shot segmentation margin over the second-ranked team is 0.0489.
- MVTec AD results: In zero-shot MVTec AD comparisons, the method exceeds WinCLIP by 2.5↑ AUROC-segm and 11.6↑ F1-max-segm but has lower PRO.Its precise predicted regions are often slightly smaller than ground truth, which may reduce PRO despite correct localization.
- VisA results: On VisA, zero-shot segmentation improves over WinCLIP by 14.6↑ AUROC-segm, 17.5↑ F1-max-segm, and 30↑ PRO-segm, while anomaly classification slightly lags.In few-shot VisA, segmentation slightly trails WinCLIP but has higher PRO, while classification metrics significantly outperform comparison methods.
4. Conclusion
The method adapts CLIP for zero-shot segmentation and classification, then extends it with memory banks for few-shot performance. It achieved first place in the zero-shot VAND track and fourth place in the few-shot track.
- Extra linear layers enable CLIP to perform zero-shot segmentation while retaining zero-shot classification.
- Memory banks incorporating a few reference images further improve the method for few-shot anomaly detection.
- First place in the zero-shot track and fourth place in the few-shot track demonstrate competitive challenge performance.