PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models

Jiacong Xu, Shao-Yuan Lo, Bardia Safaei, Vishal M. Patel, Isht Dwivedi

arXiv:2502.07601v2cs.CVcs.CL

TL;DR

The paper addresses the lack of datasets, benchmarks, and reliable fine-grained reasoning for zero-shot visual anomaly detection. It introduces Anomaly-Instruct-125k, VisA-D&R, and Anomaly-OV, whose anomaly expert and Look-Twice Feature Matching mechanism improve detection and reasoning over generalist models. The authors also identify remaining accuracy and domain-coverage limitations.

  • Problem

    Fine-grained anomaly reasoning is underexplored because suitable datasets and benchmarks are lacking, while existing MLLMs can miss and misdescribe image abnormalities.

  • Method

    The paper builds Anomaly-Instruct-125k and VisA-D&R, then introduces Anomaly-OV with an anomaly expert and Look-Twice Feature Matching for suspicious visual-token selection.

  • Results

    Anomaly-OV shows significant improvements over existing methods on traditional ZSAD and text-based anomaly detection and reasoning tasks.

  • Takeaways & Limitations

    The resulting specialist assistant supports visual inspection with anomaly descriptions and rationales, while extensions to 3D and medical domains are demonstrated.

  • Takeaways & Limitations

    Anomaly-OV can misclassify target objects, describe anomalies too generally, hallucinate reasoning, and still has room to improve detection performance.

Abstract

from arXiv · show

Zero-Shot Anomaly Detection (ZSAD) is an emerging AD paradigm. Unlike the traditional unsupervised AD setting that requires a large number of normal samples to train a model, ZSAD is more practical for handling data-restricted real-world scenarios. Recently, Multimodal Large Language Models (MLLMs) have shown revolutionary reasoning capabilities in various vision tasks. However, the reasoning of image abnormalities remains underexplored due to the lack of corresponding datasets and benchmarks. To facilitate research in AD & reasoning, we establish the first visual instruction tuning dataset, Anomaly-Instruct-125k, and the evaluation benchmark, VisA-D&R. Through investigation with our benchmark, we reveal that current MLLMs like GPT-4o cannot accurately detect and describe fine-grained anomalous details in images. To address this, we propose Anomaly-OneVision (Anomaly-OV), the first specialist visual assistant for ZSAD and reasoning. Inspired by human behavior in visual inspection, Anomaly-OV leverages a Look-Twice Feature Matching (LTFM) mechanism to adaptively select and emphasize abnormal visual tokens. Extensive experiments demonstrate that Anomaly-OV achieves significant improvements over advanced generalist models in both detection and reasoning. Extensions to medical and 3D AD are provided for future study. The link to our project page: https://xujiacong.github.io/Anomaly-OV/

1. Introduction

Visual anomaly detection traditionally relies on many normal samples, while zero-shot methods address data-restricted settings. The paper introduces datasets and Anomaly-OV to support fine-grained anomaly reasoning and improve detection and explanations.

  • Motivation: Traditional unsupervised anomaly detection learns normal visual-feature distributions but requires substantial normal data, which privacy and labeling constraints can make impractical.These constraints motivate Zero-Shot Anomaly Detection as an emerging direction.
  • Motivation: Reasoning about image abnormalities remains underexplored because large-scale datasets and benchmarks are difficult to collect.Existing methods generally predict anomaly likelihood without rationales, despite the need for visual evidence and explanations.
  • Contributions: The authors establish Anomaly-Instruct-125k and VisA-D&R as a visual instruction-tuning dataset and evaluation benchmark for anomaly detection and reasoning.The resources are created through intensive human effort to support research in this area.
  • Contributions: Anomaly-OV is a specialist visual assistant that learns object-awareness abnormality embeddings and uses Look-Twice Feature Matching to emphasize suspicious visual tokens.The mechanism is designed to help the language model select anomaly-relevant visual information.
  • Results: Extensive experiments show significant improvements over advanced generalist models in both anomaly detection and reasoning.The paper also reports extensions from industrial defect detection to 3D inspection and medical image diagnosis.

2. Related Work

Prior work spans multimodal reasoning and anomaly detection, but existing approaches remain limited in fine-grained, object-agnostic anomaly representation. Anomaly-OV addresses this setting with a specialist architecture for anomaly-focused visual reasoning.

  • Multimodal Large Language Models: MLLMs combine vision encoders with language models to support text-based interaction and have demonstrated reasoning capability across several vision tasks.Related applications include video anomaly detection, affective computing, and visual commonsense reasoning.
  • Unsupervised Anomaly Detection: Traditional unsupervised anomaly detection methods use only normal data but generally assume that many normal samples are available.The related work surveys reconstruction-, student-teacher-, and augmentation-based approaches.
  • Zero-Shot Anomaly Detection: Zero-shot anomaly detection compares visual and textual features to estimate abnormality without target-object-specific data.Existing methods construct positive anomaly and negative normal prompts using CLIP encoders.
  • Zero-Shot Anomaly Detection: Prompt-based ZSAD methods using a generic object token cannot capture class-aware anomaly types and conflict with object-agnostic visual assistance.The limitation concerns representing different objects and keeping the image blind to the user.
  • Anomaly-OV: Anomaly-OV is presented as a specialist architecture with professional anomaly-expert training followed by visual instruction tuning for anomaly detection and reasoning.The architecture overview distinguishes text and visual tokens across its two training stages.

3. Method

Anomaly-OV augments a LLaVA-OneVision-based MLLM with multi-level visual features and an anomaly specialist that selects suspicious tokens for detection and reasoning. Its LTFM and VT Selector mechanisms emphasize anomalous regions while preserving original features for general instructions.

  • Architecture Overview: Anomaly-OV uses LLaVA-OneVision as its base MLLM, whose visual encoder, projector, and LLM process high-resolution images with AnyRes cropping.High-resolution inputs are divided into crops that the visual encoder processes independently before spatial pooling.
  • Architecture Overview: Features from four selected ViT layers are compressed with adapters to capture fine-grained local semantics at multiple levels.The extracted representations index both feature level and image crop.
  • Look-Twice Feature Matching: LTFM removes the heavy text encoder and uses global visual features, multi-level features, and anomaly-aware embeddings to recognize and localize suspicious tokens.The module follows a two-step inspection process inspired by humans identifying suspicious regions before examining them closely.
  • Text Generation: The method preserves original visual features alongside selected features so the base model retains generalization on regular instructions.This design contrasts anomaly-focused token selection with ordinary image-description instructions.
  • Visual Token Selector: The VT Selector weights visual tokens by their similarity to anomalous patterns, suppresses normal tokens, pools the emphasized features, and aggregates correlated tokens with a Q-Former.The pooled query tokens q_j have shape R^h×w×D, with h = w = 2 reported as an empirical trade-off.
  • Anomaly Prediction: Anomaly prediction aggregates crop-level anomaly information using significance-map weighting, while an MLP distinguishes normal and abnormal semantics with balanced BCE loss.The resulting image-level abnormal possibility is computed from a global anomaly-information vector.

4. Dataset and Benchmark

The paper introduces Anomaly-Instruct-125k and VisA-D&R to address scarce multimodal anomaly-reasoning data and evaluate detection, defect description, and complex reasoning. The dataset spans multiple domains, while the benchmark uses human-reviewed VisA-derived samples and structured metrics.

  • Anomaly-Instruct-125k: Anomaly-Instruct-125k addresses limited multimodal instruction-following data by providing large-scale anomaly detection and reasoning instructions.The motivation contrasts it with AnomalyGPT’s smaller, less diverse dataset focused only on anomaly localization.
  • Motivation: Existing MLLMs struggle to locate and describe anomalies without explicit human involvement, motivating the benchmark’s emphasis on fine-grained anomaly reasoning.The paper identifies missing captions and weak anomaly localization in GPT-4V and GPT-4o as obstacles to instruction-data construction.
  • Anomaly-Instruct-125k: The dataset combines industrial, medical, 3D, and in-the-wild images, including 72k WebAD images collected through an automatic GPT-4o and Google Image Search pipeline.Its conversations cover anomaly detection, visual description, potential causes, and future suggestions.
  • VisA-D&R: VisA-D&R selects 10 VisA classes and contains 761 normal and 1000 anomalous samples after human review and re-annotation of generated descriptions.VisA provides fine-grained anomaly types and segmentation for each image.
  • VisA-D&R: Detection evaluation uses one-word-answer prompts and Accuracy, Precision, Recall, and F1-score, while reasoning evaluation separates low-level descriptions from complex cause and improvement reasoning.Text similarity is measured with ROUGE-L, SBERT, and GPT-Score.

5. Experiments

Experiments evaluate Anomaly-OV across zero-shot detection, text-based anomaly detection, reasoning, and extensions to 3D and medical domains. Results show improvements over existing methods, while also identifying a limitation in the anomaly expert.

  • Training & Evaluation: Anomaly-OV evaluates image-level anomaly detection on nine industrial and medical ZSAD benchmarks using AUROC.The evaluation includes MVTec AD, VisA, AITEX, ELPV, BTAD, MPDD, BrainMRI, HeadCT, and Br35H.
  • Zero-Shot Anomaly Detection: Anomaly-OV achieves significant image-level AUROC improvements on most ZSAD benchmarks compared with existing methods.The authors attribute the success mainly to additional WebAD data, which supports learning generic normality and abnormality semantics without a text encoder.
  • Zero-Shot Anomaly Detection: Significance maps concentrate high intensities around anomalous areas without supervision from anomaly masks.This demonstrates the anomaly expert’s interpretable suspicious-token selection mechanism.
  • Anomaly Detection & Reasoning: Anomaly-OV improves text-based anomaly detection and reasoning over open-sourced generalist MLLMs.The authors observe that larger language models improve reasoning ability, while many existing MLLMs show lower recall than precision.
  • Anomaly Detection & Reasoning: Anomaly-OV-0.5B improves accuracy by 0.08 and F1-score by 0.06, while Anomaly-OV-7B achieves the best detection performance among existing MLLMs.Anomaly-OV-7B shows reasoning ability comparable to GPT-4o.
  • Anomaly Detection & Reasoning: A stronger anomaly expert is suggested because the current anomaly expert restricts Anomaly-OV’s detection performance.The paper also reports extensions to 3D inspection and medical diagnosis for future study.

6. Conclusion

The paper addresses data scarcity in visual anomaly detection and reasoning by introducing a dataset, benchmark, and specialist MLLM. Anomaly-OV improves ZSAD and text-based anomaly reasoning, with extensions to 3D and medical domains.

  • 6. Conclusion: Anomaly-Instruct-125k and VisA-D&R address data scarcity for visual anomaly detection and reasoning.They are presented as the first large-scale visual instruction tuning dataset and corresponding benchmark for this purpose.
  • 6. Conclusion: Anomaly-OV uses an anomaly expert to select suspicious visual tokens and improves traditional ZSAD and text-based anomaly detection and reasoning.The model is introduced as a specialist MLLM targeting visual inspection.
  • 6. Conclusion: Anomaly-OV is extended to 3D and medical anomaly detection and reasoning.The conclusion presents these extensions as demonstrations of the model’s broader application scope.

A1. Dataset Establishment

The dataset establishment process combines automatically collected web data with anomaly-focused prompts and descriptions. It uses GPT-4o, image search, visual hints, filtering, and existing anomaly masks to build training data.

  • How to highlight the anomaly?: GPT-4o often fails to detect anomalies without hints but can reason about tiny anomalies when visual or textual hints are supplied.A red bounding box enables detection of a tiny bubble inside a small capsule, motivating the token-picking mechanism.
  • How to highlight the anomaly?: Existing anomaly masks are dilated and merged to generate bounding boxes used as visual prompts for GPT-4o.The resulting prompts support generation of detailed image and anomaly descriptions for instruction-following data.
  • Motivation: Existing industrial and medical datasets provide fewer than 20 classes and limited anomaly types, restricting generic anomaly-pattern learning.The paper identifies extensive human involvement in collecting such images as a source of this limitation.
  • How to collect anomaly data: GPT-4o and Google Image Search form an automatic pipeline for collecting online normal and anomalous images.GPT-4o generates 400 common classes and ten normal and anomalous phrases for each class.
  • How to collect anomaly data: WebAD contains around 72k images from 380 classes with more than five anomaly types per class.CLIP similarity filtering removes near-duplicates, while GPT-4o classification removes incorrectly labeled and artificial images.

A2. Training Details

Training uses separate professional and instruction-tuning stages with different trainable components. The setup includes multi-image data for 3D detection and medical data from LLaVA-Med.

  • Professional training: The anomaly expert is trained for two epochs on eight H100 GPUs with a total batch size of 128.The professional stage uses AdamW, CosineAnnealingWarmRestarts, and an initial learning rate of 1e-4.
  • Domain extensions: The training setup incorporates multi-image understanding for 3D anomaly detection and LLaVA-Med data for medical-domain knowledge.These additions support the paper’s 3D and medical anomaly-detection extensions.

A3. Experimental Results

Anomaly-OV is evaluated against GPT-4o and LLaVA-OneVision on anomaly detection and reasoning, with additional results covering multiple VisA-D&R object categories and in-the-wild images.

  • Anomaly-OV shows better performance than GPT-4o and LLaVA-OneVision in detecting and describing visual anomalies.
  • Additional evaluations cover Candle, Capsules, Fryum, Cashew, PCB, and an unseen Road Sign object, while Table 14 examines low-level and complex reasoning.

A4. Limitation and Future Work

The paper identifies limitations in anomaly classification, description, reasoning, detection performance, and benchmark coverage, and proposes broader data and improved model designs as future directions.

  • Limitations: Anomaly-OV sometimes misclassifies target objects, uses overly general anomaly descriptions, produces hallucinated reasoning, and still has room to improve detection performance.
  • Limitations: VisA-D&R contains industrial images, so benchmarks for 3D and medical anomaly detection are needed to evaluate a unified model across domains.
  • Future Work: The anomaly expert strongly determines detection performance, motivating more advanced expert designs in future research.
  • Future Work: Limited industrial anomaly diversity, mainly cracks or broken parts, causes failures on fine-grained reasoning for unseen anomaly features.
  • Future Work: Forwarding multi-level visual features from different encoder layers to the LLM is proposed as a possible performance improvement because anomaly detection depends on low-level visual clues.
Loading 2502.07601v2…