PaperlayerView exact revision on arXiv

Source-linked AI summary

IndusAgent: Reinforcing Open-Vocabulary Industrial Anomaly Detection with Agentic Tools

Rongbin Tan, Fangfang Lin, Zhenlong Yuan, Min Qiu, Kejin Cui, Mengmeng Wang, Yi Wang, Zijian Song, Zhiyuan Wang, Jiyuan Wang, Yue Wang, Shuhan Song§, Huawei Cao

arXiv:2605.20682v1cs.CV

TL;DR

Open-vocabulary industrial anomaly detection challenges MLLMs with domain-misaligned reasoning and structural hallucinations. IndusAgent combines Indus-CoT, supervised diagnostic alignment, dynamic tool orchestration, and accuracy-gated reinforcement learning. Across five benchmarks, it achieves state-of-the-art zero-shot performance while retaining an 8B-parameter footprint, although its current evaluation is mainly image-level and tool feedback may be unreliable.

  • Problem

    MLLM-based open-vocabulary IAD is limited by domain-misaligned reasoning and brittle generalization to novel anomalies or ambiguous instructions.

  • Method

    IndusAgent uses Indus-CoT, supervised fine-tuning, dynamically orchestrated diagnostic tools, and an accuracy-gated reinforcement learning objective.

  • Results

    IndusAgent achieves state-of-the-art zero-shot performance across five challenging benchmarks, with an average score of 83.4% using 8B parameters.

  • Takeaways & Limitations

    The framework provides an active, domain-aligned inspection paradigm that rewards tool use when it supports correct diagnosis.

  • Takeaways & Limitations

    The current experiments mainly evaluate image-level anomaly judgment, while tool feedback may be inaccurate and introduce additional inference overhead.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have shown remarkable capability in bridging visual perception and textual reasoning, enabling zero-shot understanding across diverse industrial scenarios. However, their performance in open-vocabulary industrial anomaly detection (IAD) is often limited by domain-misaligned reasoning and hallucinated structural inferences. To address these challenges, we propose \textbf{IndusAgent}, a tool-augmented agentic framework for open-vocabulary IAD. Specifically, we first construct \textbf{Indus-CoT}, a structured dataset that integrates global visual observations, high-resolution local patches, and expert normalcy priors, providing supervision for fine-tuning the model on rigorous industrial inspection trajectories. Building on this, IndusAgent dynamically orchestrates a set of external tools, including dynamic region cropping, high-frequency feature enhancement, and prior retrieval, thus enabling the agent to actively resolve visual ambiguities and disentangle subtle anomalies. Furthermore, we introduce a gated reinforcement learning objective that jointly optimizes anomaly classification, localization accuracy, anomaly type reasoning, and efficient tool usage, ensuring that tool invocation occurs only when beneficial. Extensive evaluations on five industrial anomaly benchmarks, including MVTec-AD, VisA, MPDD, DTD, and SDD, demonstrate that IndusAgent achieves state-of-the-art zero-shot performance among all existing methods, validating our robustness and generalization capacity.

1 State Key Lab of Processors, Institute of Computing Technology, CAS

The listed affiliations include Santa Clara University, LongCat Team, and an independent researcher.

  • Santa Clara University is listed among the authors’ affiliations.
  • LongCat Team is listed among the authors’ affiliations.
  • An independent researcher is listed among the authors’ affiliations.

9 University of Chinese Academy of Sciences, Beijing, China

The passage marks equal contribution, project leadership, and corresponding authorship.

  • The authors include equal-contribution designation.
  • The author list identifies a project lead.
  • The author list identifies a corresponding author.

1 Introduction

IndusAgent addresses open-vocabulary industrial anomaly detection by aligning MLLM reasoning with industrial protocols and enabling active tool use. Its accuracy-gated reinforcement learning further links tool utility to diagnostic correctness.

  • Motivation: Open-vocabulary IAD targets unseen defect classes and object categories beyond closed-set inspection systems.
  • Motivation: MLLMs offer zero-shot visual reasoning, but industrial use is limited by domain-misaligned reasoning and brittle open-vocabulary generalization.
  • IndusAgent: IndusAgent aligns model reasoning with expert industrial standards and dynamically inspects high-resolution patches while querying normalcy priors.
  • IndusAgent: Accuracy-gated reinforcement learning activates tool utility rewards only when the final diagnostic judgment is correct, discouraging tool abuse.
  • Contributions: Indus-CoT encodes global observations, localized evidence, normalcy priors, tool feedback, and final defect judgments for domain-aligned reasoning.
  • Contributions: IndusAgent establishes state-of-the-art results across five challenging industrial anomaly benchmarks.

2 Methodology

IndusAgent combines structured reasoning data, supervised alignment, tool-augmented inference, and gated reinforcement learning for industrial anomaly diagnosis. The framework uses complementary visual, semantic, and geometric evidence while rewarding useful rather than excessive tool invocation.

  • Overview: IndusAgent constructs Indus-CoT, applies supervised fine-tuning, and then optimizes tool-augmented reasoning with reinforcement learning.
  • Problem Formulation: Industrial anomaly diagnosis produces structured reasoning, localization, defect category, and binary judgment from an image and task instruction.
  • Unified Agentic Inference: The agent identifies uncertain regions, invokes selected tools, and fuses their observations with the original image and instruction.
  • Agentic Toolkit: The toolkit uses cropping, prior retrieval, enhancement, and geometric measurement to address fine-grained defects, acceptable variations, and ambiguous structures.
  • Indus-CoT Construction: Indus-CoT combines multimodal reasoning trajectories with explicit tool-execution traces to supervise active industrial inspection.
  • Supervised Fine-Tuning: SFT trains on global images and multi-round tool observations while supervising generated reasoning tokens for structured diagnostic behavior.
  • Agentic Reinforcement Learning: The accuracy-gated reward combines classification, localization, anomaly type, tool utility, and format compliance, with tool rewards gated by correct classification.
  • Agentic Reinforcement Learning: The cost term penalizes redundant calls, biasing the policy toward tools that provide additional local, textual, or geometric evidence.

3 Experiment

IndusAgent is evaluated zero-shot across five industrial anomaly benchmarks and achieves strong overall performance through domain-specific alignment, active tooling, and reinforcement learning. Ablations show that supervised alignment, reinforcement learning, tool augmentation, and gated rewards each contribute to performance.

  • Main Results: 83.4% average score establishes IndusAgent (8B) as the reported zero-shot SOTA across five industrial anomaly benchmarks.The evaluation covers industrial objects and surface textures, including MVTec-AD, VisA, MPDD, DTD, and SDD.
  • Main Results: 76.8% on VisA and 72.7% on MPDD show strong performance on structurally complex datasets with an 8B parameter footprint.These scores are reported as surpassing the best-performing VLM baselines.
  • Key Findings: Domain-specific diagnostic alignment improves industrial anomaly detection beyond open-ended MLLM reasoning alone.Qwen3-VL-Instruct performs poorly on VisA, whereas Agentic SFT and RL substantially improve performance.
  • Key Findings: Selective cropping, enhancement, measurement, and normalcy-prior retrieval isolate local evidence and verify structural cues.The findings characterize active tool use as a complement to passive MLLM perception for subtle defects and scale ambiguity.
  • Anomaly Recall: +17.4% on MPDD and +10.4% on DTD are the reported recall gains from the GRPO framework under severe background interference.The framework aligns reasoning policy with final diagnostic outcomes and improves recall across all evaluated datasets.
  • Ablation Studies: 55.5% on VisA follows removal of Agentic SFT from the 76.8% full-model result, while removing RL or tools also degrades performance.The ablation attributes these effects to protocol alignment, open-vocabulary generalization, and active tool orchestration.
  • Ablation Studies: 65.7% on VisA after removing format compliance is the largest reported gated-reward degradation.Removing localization, type, or gated tool-utility rewards also impairs diagnostic grounding or accuracy.

4 Related Work

Related work spans reconstruction, feature-embedding, multimodal reasoning, and tool-augmented agentic systems. IndusAgent positions itself as an active inspector using tool-augmented reinforcement learning for industrial anomaly detection.

  • Open-Vocabulary Industrial Anomaly Detection: Reconstruction-based and feature-embedding-based methods represent established paradigms for open-vocabulary industrial anomaly detection.The related-work discussion includes inpainting, synthetic anomaly generation, diffusion reconstruction, memory banks, and normalizing flows.
  • Open-Vocabulary Industrial Anomaly Detection: Diffusion-based reconstruction can improve fidelity but may reconstruct anomalies and miss subtle defects.This limitation is presented alongside methods such as AnoDDPM and DiffusionAD.
  • Positioning: IndusAgent introduces an active inspector paradigm with tool-augmented reinforcement learning for robust reasoning.This distinguishes the proposed approach from the related paradigms summarized in the section.
  • Reasoning in Multimodal LLMs: RL-based post-training has extended multimodal reasoning to mathematical VQA, reasoning segmentation, and general image understanding.The discussion cites OpenAI-o1, DeepSeek-R1, and multimodal extensions as examples.
  • Tool-Augmented Agentic Systems: Tool-augmented multimodal systems include supervised, program-derived, and reinforcement-learning approaches, but most rely on static pipelines or reward invocation without execution-cost consideration.The cited systems include MVoT, LLaVA-Plus, VPD, TACO, and PyVision.

5 Conclusion

IndusAgent combines domain-specific reasoning alignment, tool augmentation, and efficiency-aware reinforcement learning for zero-shot industrial anomaly detection. Its reward design jointly addresses diagnostic correctness, localization, anomaly categorization, tool utility, and output compliance while discouraging indiscriminate tool use.

  • Conclusion: IndusAgent grounds zero-shot industrial anomaly detection in expert diagnostic protocols through Agentic SFT and autonomous tool-augmented reinforcement learning.The framework uses tools for fine-grained patches, low-contrast textures, geometric measurements, and normalcy-prior retrieval.
  • Conclusion: The reward function balances binary anomaly accuracy, spatial localization, semantic anomaly categorization, tool utility, and valid output formatting.Racc gates task-level credit, Rloc measures predicted-ground-truth box overlap, Rtype evaluates hierarchical semantic distance, and Rformat stabilizes parseable trajectories.
  • Conclusion: Tool utility rewards beneficial evidence acquisition only when tool feedback improves the final diagnostic decision, while penalizing computational and reasoning cost.The confidence improvement compares decision probabilities before and after tool observations, and the cost term discourages redundant calls.
  • Conclusion: Confidence improvement is estimated from normalized binary decision-token margins rather than free-form verbalized confidence.This reduces sensitivity to response length, reasoning style, and formatting variations.
  • Conclusion: IndusAgent is evaluated under a shared binary anomaly-detection instruction without paired normal references, category-specific exemplars, or dataset-specific prompts.The same inference protocol is applied to baseline models and IndusAgent.

B.4 More Experimental results.

Across four shared datasets, IndusAgent is compared with general-purpose multimodal language models using aggregated F1 scores. The reported gap is attributed to the need for granular industrial-defect localization beyond macroscopic visual recognition.

  • More Experimental results: General-purpose MLLMs show a pronounced performance gap relative to IndusAgent in zero-shot industrial scenarios.The passage attributes this gap to insufficient granular resolution for micro-defect localization.
  • More Experimental results: Table 6 aggregates F1-Score comparisons over four shared datasets to enforce strict evaluation fairness.The table caption specifies aggregation over four shared datasets.

B.5 Tool Usage Analysis.

IndusAgent uses a selective, cost-aware tool policy, adapting invocation patterns to dataset characteristics while keeping average tool use near or below one call per query.

  • Tool Usage Analysis: IndusAgent maintains an average tool invocation rate near or below 1.0 per query across the analyzed benchmarks.The policy is described as selective and cost-aware rather than exhaustive.
  • Tool Usage Analysis: Tcrop dominates tool usage on MVTec and VisA, accounting for 62.4% and 54.8% of invocations, respectively.The passage links this pattern to isolating fine-grained structural defects in object-centric datasets.

B.6 More Ablation Studies.

The ablations examine individual tools and reinforcement-learning group size. Dynamic cropping is especially important on VisA, while group-size scaling improves accuracy up to six candidates before saturation and degradation.

  • Tool Ablations: Table 8 reports ablations of individual tools in the toolset.The ablation study evaluates how removing tools affects benchmark performance.
  • Tool Ablations: Removing Dynamic Region Cropping reduces VisA performance from 76.8% to 68.6%.The reported degradation underscores cropping’s role in isolating micro-defects from intricate normal backgrounds.
  • Group-Size Ablation: Table 9 evaluates the number of generated candidates per prompt during Agentic RL.This analysis studies sensitivity to GRPO group size.
  • Group-Size Ablation: 84.1%, 77.2%, and 96.0% are achieved on MVTec, VisA, and DTD, respectively, at group size 6.Scaling from group size 2 to 6 consistently refines GRPO advantage estimation, while group size 8 slightly degrades performance.

C Discussion on Novelty and Problem Setting

IndusAgent reformulates open-vocabulary industrial anomaly detection as reference-free, category-disjoint active inspection rather than generic tool-augmented reinforcement learning. Its novelty combines tool-integrated diagnostic supervision, accuracy-gated rewards, and dataset-specific tool selection.

  • Novelty and Problem Setting: IndusAgent targets reference-free, category-disjoint active inspection, where subtle defects must be separated from legitimate structural variation without paired normal references or target-category training samples.This setting makes naive tool use insufficient because additional tools can amplify noise, hallucinate locations, or overfit category-specific priors.
  • Novelty and Problem Setting: Accuracy-gated rewards prevent localization, anomaly-type prediction, or tool invocation from receiving positive credit when the final binary diagnosis is wrong.The design addresses hallucinated defect regions and redundant tool use that can arise under standard additive rewards.
  • Novelty and Problem Setting: Indus-CoT supervises inspection trajectories linking global hypotheses, tool decisions, tool feedback, localized or semantic evidence, and final diagnosis.It differs from ordinary chain-of-thought data and conventional IAD datasets by explicitly grounding reasoning in tool use.
  • Novelty and Problem Setting: The framework combines a stricter problem setting, active-inspection supervision, diagnostic correctness gating, and empirical selection of tools according to dataset-specific inspection demands.Together, these components distinguish IndusAgent from directly transferring generic agentic reinforcement learning to IAD.

D Tool Library Specifications

The tool library provides complementary mechanisms for inspecting suspicious regions, retrieving normalcy information, enhancing subtle textures, and measuring geometric relations. Training prompts constrain tool use to visually supported, image-grounded diagnostic verification.

  • Tool Library: Dynamic region cropping extracts high-resolution local patches to preserve fine spatial details such as microscopic scratches and subtle textural inconsistencies.The module acts as an active attention mechanism when the agent suspects a morphological deviation.
  • Tool Library: Normalcy-prior retrieval supplies semantic descriptions of legitimate category-specific structures to mitigate confusion between intricate geometries and true anomalies.The tool grounds judgments in verified structural baselines, such as expected component surfaces and joints.
  • Tool Library: Texture enhancement applies lightweight operators such as Canny edge detection or CLAHE to produce noise-suppressed, high-frequency maps for low-contrast or reflective surfaces.It is intended to clarify faint stains and other subtle surface anomalies.
  • Tool Library: Geometric verification computes physical or pixel distances and angles from reference coordinates to expose spatial deviations in components such as PCBs and threaded screws.This addresses the limited physical-scale awareness of standard vision-language models.
  • Training and Output Protocol: The inspection prompts require global description, visually supported suspicious regions, justified tool selection, and final answers consistent with tool observations and annotations.Normal samples must not receive fabricated defects, while anomalous samples require consistent locations and coarse defect types.

H Limitations

IndusAgent’s reported scope is bounded by inference overhead, dependence on tool-feedback reliability, limited fine-grained evaluation, and possible teacher or prompt-template bias.

  • Limitations: The active inspection process adds inference overhead because cropping, enhancement, and prior retrieval require extra computation.The limitation concerns efficiency relative to single-pass MLLM inference.
  • Limitations: Final diagnosis depends on reliable tool feedback, since inaccurate crops, noisy enhanced maps, or incomplete normalcy priors may mislead the agent.This identifies a practical robustness boundary for deployment.
  • Limitations: Current experiments mainly focus on image-level anomaly judgment, leaving pixel-level localization, region-level grounding, and tool-use efficiency for further evaluation.The paper explicitly calls for finer-grained analyses of diagnostic behavior.
  • Limitations: Indus-CoT generation uses a strong teacher model and rule-based validation, which may introduce teacher or prompt-template bias.Future work is directed toward more diverse expert supervision.

2. Limitations

The supplied material indicates that limitations are discussed in supplementary material, while no theoretical results are presented. It also states that the work does not release new assets and does not involve human subjects.

  • The proposed algorithm’s limitations are discussed in the supplementary material.
  • The paper reports no theoretical result and therefore provides no theoretical proof or assumption analysis.
  • The paper states that implementation details, experimental settings, and compute resources are provided in the main paper or supplementary material.
Loading 2605.20682v1…