Source-linked AI summary
InspectorGPT: A Comparative Reasoning Enhanced VLM for Comprehensive Industrial Anomaly Detection
Weifei Chen, Honghao Zhang, Zhiyuan You, Xinyi Le
TL;DR
Industrial anomaly detection needs detectors that transfer across categories while providing interpretable, pixel-level evidence, but existing normality-based and reasoning-oriented approaches have important limitations. InspectorGPT addresses this gap by comparing a normal reference with a query, training comparative reasoning with CoT and GRPO, and combining separately trained reasoning and segmentation branches through task-vector fusion. Experiments report superior multi-dimensional performance and generalization to unseen benchmarks.
Problem
Industrial anomaly detection methods have limited transfer to unknown categories, while recent reasoning-oriented VLMs can lose anomaly discrimination and often lack pixel-level localization.
Method
InspectorGPT trains a VLM for reference-conditioned comparative reasoning using CoT fine-tuning and GRPO, then fuses separately trained reasoning and segmentation branches for pixel-level masks.
Results
InspectorGPT demonstrates superior multi-dimensional performance and generalization to unseen benchmarks, outperforming commercial, open-source, and anomaly-detection-specific models.
Takeaways & Limitations
Comparative reasoning is presented as a robust paradigm for comprehensive industrial inspection across multiple tasks and unseen benchmarks.
Abstract
from arXiv · showhide
Industrial anomaly detection is a critical component of modern manufacturing. Most traditional unsupervised methods rely on modelling normal feature distributions, inherently limiting generalization to unknown categories. To improve generalizability, some recent methods incorporate vision-language models (VLMs) for zero-shot detection via text prompts. However, we observe that reasoning-oriented post-training can cause anomaly discrimination to collapse, with some fine-tuned models performing worse than their base VLMs. Existing methods also provide only textual decisions or coarse boxes, without pixel-level segmentation. A more explicit detection principle comes from human inspection: anomalies are identified by comparing a query image with a defect-free reference. Inspired by this, we propose InspectorGPT, a VLM framework centered on comparative reasoning. Given a normal reference and a query image, InspectorGPT compares them to identify discrepancies and perform multiple inspection tasks with detailed reasoning. We internalize this capability through Chain-of-Thought (CoT) fine-tuning and Group Relative Policy Optimization (GRPO) with tailored, verifiable rewards. We further introduce InspectorGPT-Seg for pixel-level anomaly masks. Segmentation supervision improves anomaly discrimination but weakens semantic reasoning, while joint training fails to balance them. We therefore train the two branches separately and combine them through task-vector fusion. Extensive experiments demonstrate superior multi-dimensional performance and generalization to unseen benchmarks, validating comparative reasoning for comprehensive industrial inspection.
Introduction
InspectorGPT addresses limited cross-category generalization, weak interpretability, and coarse localization in industrial anomaly detection through reference-conditioned comparative reasoning. It combines reasoning and segmentation branches to support multi-dimensional inspection and transfer to unseen benchmarks.
- Motivation: Traditional detectors rely on defect-free normal distributions, limiting transfer to new product types without retraining and providing binary, weakly interpretable anomaly scores.These methods also lack semantic descriptions and interpretable evidence for defect identification.
- Motivation: Reasoning-oriented post-training can cause anomaly discrimination to collapse, while existing VLM-based outputs often remain textual answers or coarse boxes.The paper motivates grounding decisions in observable discrepancies between a defect-free reference and a query image.
- InspectorGPT: InspectorGPT compares a normal reference with a query, performs structured semantic comparisons, and generates Chain-of-Thought reasoning across diverse inspection dimensions.The framework then supports anomaly discrimination, detection, classification, and segmentation.
- Training: Training combines CoT supervised fine-tuning and GRPO with task-specific rewards to internalize generalizable comparative detection principles.The framework is designed to produce interpretable outputs and transfer to novel categories without retraining.
- Segmentation: InspectorGPT-Seg adds a lightweight segmentation decoder for pixel-level anomaly masks, while separate reasoning and segmentation branches are combined through task-vector fusion.Dense segmentation supervision improves anomaly discrimination but weakens semantic reasoning, and joint optimization fails to balance both capabilities.
- Results: The framework generalizes to unseen benchmarks and demonstrates superior multi-dimensional performance over commercial, open-source, and anomaly-detection-specific models.The reported results are presented as validating comparative reasoning for comprehensive industrial inspection.
Related Work
Prior industrial anomaly detection methods model normality or use VLMs for zero-shot and interpretable detection, but reasoning-based systems still have discrimination and localization limitations. Task-vector fusion offers a training-free approach for composing capabilities without joint training.
- Traditional Industrial Anomaly Detection: Classical industrial anomaly detectors learn normality from defect-free samples using embedding-based or reconstruction-based paradigms.Embedding methods compare test features with stored normal representations, discrepancies, or one-class boundaries.
- Limitations of Recent Methods: Reasoning-oriented post-training can reduce anomaly discrimination below the base VLM, while outputs often remain textual answers or coarse boxes rather than pixel-level localization.Pixel-level segmentation remains rarely supported among VLM-based anomaly detectors.
- Task Vector Fusion: Task-vector fusion composes fine-tuning displacements approximately linearly, offering a training-free way to balance multiple abilities without joint training.Related work also explores averaging fine-tuned models and resolving parameter interference during merging.
Method
InspectorGPT uses a four-stage pipeline that combines comparative reasoning, reward-guided alignment, pixel-level segmentation, and task-vector fusion. Separate reasoning and segmentation branches are trained from a shared SFT checkpoint and fused to balance semantic reasoning with anomaly discrimination.
- Pipeline overview: The framework comprises CoT supervised fine-tuning, GRPO, segmentation grounding, and task-vector fusion.The branches are optimized in parallel after CoT-SFT to learn transferable comparative reasoning and grounded outputs.
- Stage 1: CoT supervised fine-tuning: CoT-SFT trains the VLM on reference-query pairs to generate structured comparative reasoning trajectories.The trajectories follow an Align-Contrast-Diagnose format, with masks used only as spatial hints during data construction.
- Stage 2: GRPO alignment: GRPO samples completion groups and optimizes them with format, conditional IoU, and verifiable answer rewards.The answer reward is gated so anomalous predictions require corresponding boxes, while normal samples must not receive boxes.
- Stage 3: Segmentation grounding: A segmentation branch upgrades coarse bounding boxes to pixel-level masks using multi-scale features, spatial context adapters, and a Mask2Former-derived decoder.Its loss combines classification with mask objectives, while total variation encourages spatial continuity and consistency aligns global classification with local activation.
- Stage 4: Task-vector fusion: Because joint optimization fails to balance reasoning and anomaly discrimination, the two branches are trained separately and combined through task-vector fusion.The shared backbone vectors are interpolated from the common SFT initialization, while the segmentation decoder is retained separately; the final fusion uses α = 0.3.
Experiments
InspectorGPT is evaluated across multi-dimensional VQA, pixel-level detection, unseen-benchmark transfer, and controlled ablations. Results show that reference-conditioned comparative reasoning, staged optimization, and task-vector fusion jointly preserve semantic performance while improving anomaly discrimination.
- Multi-dimensional inspection: InspectorGPT reaches 82.38% average MMAD accuracy, exceeding AD-FM at 82.03% and Qwen2.5-VL-72B at 76.96%.It also leads anomaly discrimination at 73.90% and defect analysis at 88.25%.
- Pixel-level detection: InspectorGPT-Seg achieves 96.9 image and 98.7 pixel AUROC on MVTec-AD, and ties the best pixel-level AUROC on VisA at 97.8.These results are evaluated against non-VLM few-shot detectors using dense anomaly masks.
- Input paradigm: Providing a normal reference raises average discrimination from 65.83% to 73.90% (+8.07), with gains of +13.49 on VisA and +10.16 on GoodsAD.MVTec-LOCO instead declines by 0.33 because its logical anomalies depend on global compositional rules that one reference cannot express.
- Training stages: CoT supervision is load-bearing: direct-answer retraining reduces A-Disc. to 52.66%, while combining SFT and GRPO reaches 82.50% versus 73.68% for SFT alone.The controlled ablation attributes the collapse to lacking an explicit comparison trace between the two images.
- Task-vector fusion: Segmentation and reasoning branches specialize in opposite directions, so joint training reaches only 69.08% A-Disc.; task-vector fusion improves discrimination by 4.82 points without updating weights.The reasoning branch has Sem-6 84.66% and A-Disc. 69.57%, whereas the segmentation branch has A-Disc. 79.08% and Sem-6 74.31%.
- Task-vector fusion: Setting α=0.3 raises A-Disc. from 69.57% to 73.90% versus the untuned base VLM at 70.47%, while costing 0.12 points of overall average.The first 0.3 of the sweep provides 4.33 A-Disc. points for 0.86 Sem-6 points, motivating the selected coefficient.
Conclusion
InspectorGPT combines semantic reasoning with precise localization through a staged pipeline, using reward design, segmentation, LoRA co-training, and task-vector fusion to balance the two capabilities.
- InspectorGPT uses a staged pipeline coupling large-model semantic reasoning with precise anomaly localization.Its design includes a GRPO reward engine, segmentation decoder, LoRA co-training, and task-vector fusion.