PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts

Jiawen Zhu, Guansong Pang

arXiv:2403.06495v3cs.CV

TL;DR

Generalist anomaly detection seeks one model that can detect anomalies across domains without target-data training, but CLIP-based defect methods rely on handcrafted prompts that may not transfer to other anomaly types. InCTRL uses few-shot normal images as prompts and learns residual-based anomaly detection on auxiliary data; it significantly and consistently outperforms competing methods across nine diverse datasets.

  • Problem

    Generalist anomaly detection asks whether one model can detect anomalies across diverse application domains without training on target data, while CLIP defect methods rely on handcrafted prompts that are difficult to extend to medical or semantic anomalies.

    Source: §1 · p. 2

  • Method

    InCTRL trains on auxiliary data to compare query images with few-shot normal prompts and identify residuals that distinguish anomalies from normal samples.

    Source: §1 · p. 2, §3.2 · pp. 3–4

  • Results

    Across nine datasets covering industrial, medical, and semantic anomaly detection, InCTRL significantly and consistently outperforms state-of-the-art competitors across multiple few-shot settings.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results support evaluating anomaly-detection generalization across application domains without training on the target datasets.

    Source: §5 · p. 9

    A few prompts can yield false positives when an anomalous query has texture differences from the normal prompts; authors note more similar normal prompts may remedy this case.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

This paper explores the problem of Generalist Anomaly Detection (GAD), aiming to train one single detection model that can generalize to detect anomalies in diverse datasets from different application domains without any further training on the target data. Some recent studies have shown that large pre-trained Visual-Language Models (VLMs) like CLIP have strong generalization capabilities on detecting industrial defects from various datasets, but their methods rely heavily on handcrafted text prompts about defects, making them difficult to generalize to anomalies in other applications, e.g., medical image anomalies or semantic anomalies in natural images. In this work, we propose to train a GAD model with few-shot normal images as sample prompts for AD on diverse datasets on the fly. To this end, we introduce a novel approach that learns an in-context residual learning model for GAD, termed InCTRL. It is trained on an auxiliary dataset to discriminate anomalies from normal samples based on a holistic evaluation of the residuals between query images and few-shot normal sample prompts. Regardless of the datasets, per definition of anomaly, larger residuals are expected for anomalies than normal samples, thereby enabling InCTRL to generalize across different domains without further training. Comprehensive experiments on nine AD datasets are performed to establish a GAD benchmark that encapsulate the detection of industrial defect anomalies, medical anomalies, and semantic anomalies in both one-vs-all and multi-class setting, on which InCTRL is the best performer and significantly outperforms state-of-the-art competing methods. Code is available at https://github.com/mala-lab/InCTRL.

1. Introduction

The paper targets generalist anomaly detection: one model should detect anomalies across application domains without training on target data. InCTRL uses few-shot normal images as prompts and learns to detect residuals between prompts and queries.

1. Introduction

  • Generalist anomaly detection aims for one model that detects anomalies across diverse application domains without training on target data.The paper motivates this setting by noting that per-dataset training can be infeasible when target data are scarce or cannot be used for training.Source: §1 · pp. 1–2
  • CLIP-based defect detection methods rely on handcrafted defect-specific prompts, limiting their applicability to medical and semantic anomalies.Source: §1 · p. 2
  • InCTRL learns to distinguish anomalies from normal samples by comparing query images with few-shot normal prompts, where larger residuals are expected for anomalies.The prompts serve as prototypes of normal patterns and are used for inference rather than model training or tuning.Source: §1 · p. 2
  • InCTRL models in-context residuals at both image and patch levels and incorporates normal or abnormal text-prompt knowledge from the text-image-aligned semantic space.Source: §1 · p. 2
  • Across nine diverse datasets, InCTRL significantly outperforms state-of-the-art methods in industrial, medical, and semantic anomaly detection settings.The benchmark includes semantic anomalies in both one-vs-all and multi-class settings.Source: §1 · p. 2

2. Related Work

Prior anomaly-detection approaches generally fit target datasets or rely on assumptions that constrain generalization. The paper distinguishes its instance-level residual approach from existing in-context vision methods that focus on task-level generalization.

2.1. Anomaly Detection

  • Conventional anomaly-detection methods describe normal data, reconstruct normal images, compare embeddings, or distill normal patterns, but typically fit one model per target dataset.Domain-shift approaches generally assume substantial source-target relevance, while concurrent VLM methods address weakly supervised or zero-shot settings.Source: §2.1 · pp. 2–3
  • Traditional few-shot anomaly detection models the target dataset's limited normal samples, often requiring retraining or fine-tuning that can hinder generalization to new domains.Source: §2.1 · p. 3
  • SPADE [16], PaDiM [17], and PatchCore [41] use pretrained representations without training, while RegAD [23] needs domain relevance and WinCLIP [25] depends on prompt semantics.WinCLIP's handcrafted text prompts work well for defect detection but can fail when they do not capture the required anomaly semantics.Source: §2.1 · p. 3

2.2. In-Context Learning

  • In-context learning uses minimal prompts to adapt language models to novel tasks, while vision adaptations include language or token prompts and image-inpainting approaches.Source: §2.2 · p. 3
  • Those vision methods target task-level generalization, whereas anomaly detection focuses on instance-level discrepancy; InCTRL instead treats prompts as dataset-specific normal patterns.It captures the residual between a query image and few-shot normal prompts.Source: §2.2 · p. 3

3. InCTRL: In-Context Residual Learning

InCTRL trains on auxiliary data to compare query images with few-shot normal prompts across domains, combining local, global, and text-guided signals into anomaly scores. Its design includes multi-layer patch residuals, adapted image-level residuals, and inference-time use of target-domain normal prompts.

3.1. Problem Statement

  • GAD trains one scoring function on auxiliary data for test domains drawn from different distributions, with few-shot normal prompts available only at inference.The target prompt set is not available during training.Source: §3.1 · p. 3

3.2. Overview of Our Approach InCTRL

  • InCTRL simulates query-and-prompt examples from auxiliary data, learns local and global residuals, and augments them with text-guided prior features while keeping CLIP encoders frozen.Projection and adaptation layers attached to the visual encoder are optimized to score training anomalies above normal samples.Source: §3.2 · pp. 3–4

3.3. Multi-Layer Patch-Level Residual Learning

  • Multi-layer patch residuals measure discrepancies between query-patch embeddings and their nearest prompt-patch embeddings, then average residual maps across layers.The resulting map supplies anomaly-discriminative features for subsequent scoring.Source: §3.3 · p. 4

3.4. Image-level Residual Learning

  • Image-level residual learning adapts the query class-token representation and compares it with a prototype of the few-shot prompt features.The image-level component complements local patch-level information because global discriminative information is also significant.Source: §3.4 · p. 5

3.5. Fusing Text Prompt-based Prior Knowledge

  • Text prompts provide normal and abnormal semantic features that complement visual residual features, rather than directly supplying InCTRL's anomaly score.InCTRL uses WinCLIP's state- and template-level prompt templates and ensemble strategy.Source: §3.5 · p. 5

3.6. Training and Inference

  • Training combines patch-, image-, and text-guided features for holistic scoring, while the maximum patch residual is added to retain sensitivity to local abnormal regions.At inference, the test image is processed with the target dataset's K-shot normal prompts and the text prompts to obtain its anomaly score.Source: §3.6 · pp. 5–6

4. Experiments

Experiments evaluate InCTRL across industrial, medical, and semantic anomaly detection settings using few-shot normal prompts. The results report broad gains over competing methods, examine component contributions, and show how prompt diversity relates to failures.

4.1. Experimental Setup

  • The evaluation spans nine datasets: five industrial, two medical, and two semantic, with MNIST and CIFAR-10 tested under one-vs-all and multi-class protocols.Models are trained on auxiliary data and evaluated on target datasets without further training; the target normal prompts are held constant across compared models.Source: §4.1 · p. 6

4.2. Main Results

  • InCTRL significantly outperforms competing methods across almost all industrial cases, all medical few-shot settings, and semantic anomaly detection settings.The evaluations cover nine real-world datasets and multiple few-shot settings.Source: §4.2 · p. 7
  • Up to 11.3%, 6.5%, and 3.7% AUROC and 6.4%, 5.6%, and 6% AUPRC gains over the best competing method are reported for industrial datasets.The passage reports these gains across the three few-shot settings without pairing each value to a specific setting.Source: §4.2 · p. 7
  • Over 97.3% AUPRC on BrainMRI is achieved with two normal prompts, despite no medical-data training.Average gains over the best competing model are 3.9%, 3.4%, and 3.9% AUROC and 0.6%, 1%, and 1% AUPRC for K = 2, 4, and 8, respectively.Source: §4.2 · p. 7
  • For semantic anomalies, InCTRL reaches 90+% AUROC where prior methods obtain 50%-65%, with up to 8.2%, 5.1%, and 4.4% AUROC and 2.3%, 1.9%, and 2.5% AUPRC improvement across K = 2, 4, and 8.WinCLIP performs well on CIFAR-10, while CoOp shows a notable decline there.Source: §4.2 · p. 7

4.3. Why Does InCTRL Generalize Well?

  • Visual residual features matter more on industrial datasets, visual and textual features contribute complementarily on medical datasets, and patch residuals and/or text prompts dominate semantic results.The three components are generally mutually complementary across datasets.Source: §4.3 · pp. 7–8
  • In-context residual learning generalizes better across three domains than replacing residuals with concatenation or averaging.The alternatives were tested with other components fixed.Source: §4.3 · p. 8

4.4. Failure Cases

  • On ELPV, a false positive has texture differences from the normal prompts and residuals as large as a true positive; performance improves as eight-shot prompts cover more normal classes.Prompt diversity is especially important when normal data contains multiple patterns.Source: §4.4 · p. 8

5. Conclusion

The paper introduces Generalist Anomaly Detection as an evaluation task and proposes InCTRL, which uses holistic in-context residual learning. Across nine datasets spanning industrial, medical, and semantic anomaly detection, InCTRL significantly and consistently outperforms competing models across few-shot settings.

5. Conclusion

  • The proposed GAD task evaluates whether anomaly detectors generalize across scenarios without training on target datasets.The benchmark encompasses industrial defects, medical anomalies, and semantic anomalies.Source: §5 · p. 9
  • InCTRL uses holistic in-context residual learning and significantly and consistently outperforms state-of-the-art competing models across multiple few-shot settings.The experiments establish a benchmark using nine AD datasets across the three task types.Source: §5 · p. 9

A.1. Data Statistics of Training and Testing

  • The evaluation trains on full MVTec AD data and uses full VisA data when evaluating on MVTec AD; few-shot target normal prompts are sampled from target training sets and are not used for further training.The other eight datasets are evaluated without further training.Source: §A.1 · p. 12

A.2. Industrial Defect Inspection Datasets

The appendix describes the industrial and medical datasets used for anomaly detection and distinguishes datasets used for training from those used only for inference.

A.2. Industrial Defect Inspection Datasets

  • MVTec AD contains over 5,000 images across 15 object and texture categories, with anomaly-free training images and test images containing both defective and defect-free examples.The dataset is used to benchmark industrial inspection anomaly detection.Source: §A.2 · p. 12
  • VisA comprises 10,821 high-resolution color images across 12 objects and three domains, including 9,621 normal and 1,200 anomalous samples.Its anomalies include surface and structural defects.Source: §A.2 · p. 12
  • For training on MVTec AD or VisA, the complete datasets, including training and test data, are used; inference uses only their test sets.Table 5 provides test-set statistics for seven datasets used exclusively for inference.Source: Table 4 · p. 12, Table 5 · p. 12
  • ELPV contains 2,624 grayscale solar-cell images, SDD has 52 defective and 347 defect-free product images, and AITEX includes 245 images across seven fabric types.Only the test sets of these three datasets are used for evaluation.Source: §A.2 · pp. 12–13

A.3. Medical Anomaly Detection Datasets

  • BrainMRI is used for brain-tumor detection, while HeadCT contains 100 normal slices and 100 slices with brain hemorrhage.Only the test sets of both medical datasets are used for evaluation.Source: §A.3 · p. 13

A.4. Semantic Anomaly Detection Datasets

MNIST and CIFAR-10 are repurposed as semantic anomaly detection datasets under one-vs-all and multi-class protocols; the paper also provides examples of normal and abnormal text prompts.

A.4. Semantic Anomaly Detection Datasets

  • MNIST uses one digit class as normal in one-vs-all testing and even-number classes as normal in the multi-class protocol.The multi-class category-level label is “even number.”Source: §A.4 · p. 13
  • CIFAR-10 uses one class as normal in one-vs-all testing and animal-related classes as normal in the multi-class protocol.The multi-class category-level label is “animal.”Source: §A.4 · p. 13
  • Table 6 gives examples of normal and abnormal text prompts used in InCTRL and WinCLIP, with [c] representing a class label.Source: Table 6 · p. 13

B. Implementation Details

The experiments use a shared CLIP setup with frozen encoders and specified preprocessing, learnable components, text prompts, and comparison-method implementations.

B.1. Data Pre-processing

  • All CLIP-based models use OpenCLIP ViT-B/16+ at 240×240, with channel-wise standardization and bicubic resizing; other baselines retain their preprocessing.The visual and text encoders are frozen; InCTRL's adapter, image-level classifier, and holistic scorer have specified dimensions.Source: §B.1 · p. 13, §B.2 · p. 13

B.2. Network Architectures

  • The image-level learner predicts from in-context image residuals, while the holistic scorer maps a 225-dimensional residual map to one anomaly score.The adapter has 896-dimensional input and output with a 224-unit hidden layer; the image-level learner uses hidden layers of 128 and 64 units.Source: §B.2 · p. 13

B.3. Details of Text prompts

  • CIFAR-10 uses category-level normal and abnormal text prompts because WinCLIP prompts fail on natural images.The prompts are “a photo of [c] for anomaly detection” and “a photo without [c] for anomaly detection,” respectively.Source: §B.3 · pp. 13–14

B.4. Implementation of Comparison Methods

  • SPADE, PaDiM, and WinCLIP were re-implemented; PatchCore, RegAD, and CoOp use official implementations, and results average three seeded runs.SPADE uses K = 2, 4, or 8 nearest neighbors, and all methods use the same image prompts as InCTRL.Source: §B.4 · p. 14

C.1. Complexity of InCTRL vs. Other CLIP-based Methods

InCTRL adds auxiliary-data training relative to training-free WinCLIP, but is faster at inference and its training can be performed offline.

C.1. Complexity of InCTRL vs. Other CLIP-based Methods

  • InCTRL achieves faster inference than WinCLIP’s multi-scale few-shot scoring, while its auxiliary-data training can be done offline with generally negligible computation overhead.InCTRL and CoOp add trainable parameters through auxiliary-data training compared with training-free WinCLIP.Source: §C.1 · p. 14

C.2. Comparison with Traditional Medical Anomaly Detection Methods

InCTRL is compared with traditional medical anomaly-detection methods on two medical datasets; it uses eight normal prompts and requires no training on medical data.

C.2. Comparison with Traditional Medical Anomaly Detection Methods

  • The traditional medical methods use full-shot one-class classification, and adding anomaly images can improve their performance but makes comparison unfair to InCTRL’s 8-shot setting.The cited traditional-method results come from Cai et al. [10].Source: §C.2 · p. 14
  • Using only 8-shot normal prompts and no medical-data training, InCTRL outperforms competing models on almost all cases.The comparison covers Brain Tumor MRI2 and LAG; the traditional methods require extensive training on normal medical images.Source: §C.2 · p. 14, Table 8 · p. 14

C.3. Full Results on VisA and MVTec AD

Across VisA and MVTec AD, InCTRL outperforms the baseline methods in every reported few-shot setting.

C.3. Full Results on VisA and MVTec AD

  • InCTRL markedly surpasses all six state-of-the-art methods in every reported few-shot setting on VisA.Source: §C.3 · p. 14
  • Performance generally improves across all methods as the number of few-shot image prompts increases.Source: §C.3 · p. 14
  • InCTRL consistently outperforms all six state-of-the-art methods across every reported few-shot setting on MVTec AD.Source: §C.3 · p. 14

C.4. Full Results on One-vs-all Setting of Sementic Anomaly Detection Datasets

InCTRL surpasses all five baseline methods across few-shot settings on MNIST and CIFAR-10 under the one-vs-all protocol.

C.4. Full Results on One-vs-all Setting of Sementic Anomaly Detection Datasets

Related papers

Loading 2403.06495v3…