PaperlayerView exact revision on arXiv

Source-linked AI summary

Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?

Xingwu Zhang, Duanyang Du, Huiling Zhu, Jiayue Dai, Yixiao Liu, Guozhi Liu, Zhihan Zhang, Zijun Long

arXiv:2610.12310v1cs.CV

TL;DR

MM-IAU requires fine-grained perception alongside semantic reasoning, and single MLLMs struggle to master both while different models share a perception weakness. SiGMA divides work among heterogeneous MLLMs and a reference-based visual defect expert, with label-free reliability weighting. On MMAD, it reaches 85.2% average accuracy, 4.0% above the strongest trained specialist and Gemini-2.5-Pro.

  • Problem

    MM-IAU requires detection, localization, description, and reasoning, but a single MLLM struggles to master this breadth and models share a fine-grained perception weakness.

    Source: §1 · p. 1

  • Method

    SiGMA combines heterogeneous MLLM agents with retrieved industrial knowledge and normal references, calibrated defect evidence, and a label-free controller that weights sources by task competence and query-level evidence quality.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 4

  • Results

    85.2% average accuracy on MMAD is 4.0% above the strongest trained specialist and Gemini-2.5-Pro; SiGMA reaches 84.4% with three agents of at most 9B parameters.

    Source: §5 · p. 10

  • Takeaways & Limitations

    With all model parameters frozen, new MLLMs can join SiGMA without retraining.

    Source: §1 · p. 2

    The controller estimates source reliability from the unlabeled evaluation batch, making the full result transductive.

    Source: §5 · p. 10

Abstract

from arXiv · show

A single multimodal large language model (MLLM) struggles to excel simultaneously at detection, localization, description, and reasoning in multimodal industrial anomaly understanding (MM-IAU). We show that in-domain training does not close this gap. On MMAD, a widely adopted MM-IAU benchmark, trained specialists reach at most 75.5% accuracy in defect localization, against 92.3% for human experts, and even detect anomalies less accurately than their untrained base model. Meanwhile, different MLLMs offer complementary strengths but share this weakness in fine-grained perception, so combining them alone cannot remove it. We therefore propose SiGMA, a spatially grounded multi-agent framework that divides labor between heterogeneous MLLM agents and a dedicated visual defect expert. A multimodal searcher supplies industrial knowledge and normal references, the defect expert turns query-reference comparison into calibrated anomaly evidence, and a label-free reliability controller weighs each source by task-wise competence and query-level evidence quality. SiGMA reaches 85.2% average accuracy on MMAD, 4.0% above the strongest trained specialist and Gemini-2.5-Pro and within 1.5% of human experts. Even with three agents of at most 9B parameters, it reaches 84.4%, and new MLLMs join without retraining.

1 INTRODUCTION

MM-IAU demands both fine-grained visual perception and semantic reasoning, capabilities that single MLLMs struggle to master. The introduction argues that in-domain training and model composition do not close the perception gap, motivating SiGMA’s division of labor.

1 INTRODUCTION

  • The strongest trained method reaches 75.5% accuracy in defect localization, versus 92.3% for human experts; OmniAD and JUDO also trail their untrained base model in anomaly detection.Their anomaly-detection accuracies are 68.8% and 65.0%, respectively, versus 70.5% for the base model.Source: §1 · p. 1
  • MLLMs offer complementary task strengths, but their shared fine-grained perception weakness limits what model composition alone can recover.Across ten compact MLLMs, defect localization averages 55.94%; even selecting the best model per task leaves detection and localization at 74.1% and 63.6%.Source: §1 · p. 2
  • SiGMA pairs heterogeneous MLLM agents with a visual defect expert that supplies localized evidence through comparison with normal references.Naively overlaying anomaly maps can lower anomaly-detection accuracy by up to 12.62%, motivating calibration and task- and query-dependent weighting.Source: §1 · p. 2
  • A label-free reliability controller weights sources by task-wise competence and query-level evidence quality, while frozen parameters let new MLLMs join without retraining.The searcher retrieves industrial knowledge and a normal reference; the defect expert calibrates query-reference discrepancies and converts them into answers through reader agents.Source: §1 · p. 2
  • 85.2% average accuracy on MMAD puts SiGMA 4.0% above both JUDO and Gemini-2.5-Pro.The largest gains over trained methods are 13.8% in anomaly detection and 4.1% in defect localization.Source: §1 · p. 2

2 RELATED WORK

Prior MM-IAU work often concentrates capabilities in one MLLM, while multi-model composition methods combine responses in several ways. The related-work discussion positions SiGMA as dividing capabilities across heterogeneous MLLMs and a visual defect expert, with label-free, task- and query-sensitive reliability weighting.

2 RELATED WORK

  • Prior MM-IAU methods specialize one model through fine-tuning, learned comparison, internal expert modules, or trained tool use; other approaches keep one MLLM frozen and guide it with retrieved knowledge and normal images.SiGMA instead divides capabilities among heterogeneous frozen MLLMs and a reference-based visual defect expert.Source: §2 · pp. 2–3
  • Multi-MLLM methods rank or route responses or use an aggregator, but the cited approaches require learned selectors, extra model calls, or cannot fix weaknesses shared across the pool.Label-free reliability estimation avoids labels but typically assigns each source one competence across all items and uses only votes; SiGMA adds task-wise competence and query-level evidence quality.Source: §2 · p. 3
  • Visual-evidence methods either train a single MLLM to consume expert evidence or provide evidence without training through overlays or attention guidance.SiGMA calibrates reference-based evidence against normal residual statistics, lets several MLLM readers interpret it, and weights answers by label-free reliability and evidence quality.Source: §2 · p. 3

3 METHOD

SiGMA divides MM-IAU among a multimodal searcher, heterogeneous frozen MLLM agents, a spatially grounded defect expert, and a reliability controller. Retrieved knowledge and normal references support the agents and expert, while the controller adapts source influence to task competence and query-level evidence.

3.1 PROBLEM FORMULATION AND MOTIVATION

  • Each MM-IAU instance contains a query image, inspection question, and candidate answers, and SiGMA divides labor because MLLM capabilities and the value of expert evidence vary across tasks and queries.The method addresses both distributed capabilities and the shared weakness in fine-grained perception.Source: §3.1 · p. 3

3.2 MULTIMODAL SEARCHER

  • The multimodal searcher uses a frozen CLIP encoder to retrieve industrial knowledge and a category-specific normal reference for each query.The retrieved knowledge grounds agents semantically, while the reference anchors normal appearance for both agents and the defect expert.Source: §3.2 · p. 4

3.3 HETEROGENEOUS MLLM AGENTS

  • Heterogeneous frozen MLLM agents independently predict candidate answers using each query, prompt, retrieved knowledge, and normal reference.The framework retains their distinct task-wise strengths rather than collapsing them into one specialized model.Source: §3.3 · p. 4

3.4 FINE-GRAINED DEFECT EXPERT

  • The defect expert compares query and normal-reference patch features to create a spatial anomaly map, calibrates it against normal residual statistics, and supplies localized evidence to reader agents.Readers interpret the calibrated overlay alongside the original image; localization readers receive available maps, while detection readers are admitted only when spatial-coherence criteria are met.Source: §3.4 · pp. 4–5

3.5 RELIABILITY CONTROLLER AND ADAPTIVE AGGREGATION

  • The reliability controller estimates global and task-wise source competence from unlabeled test-batch predictions, then weights reader influence using query-level spatial evidence.It uses the global estimate alone for task types with fewer than nmin instances and otherwise combines global and task-wise estimates; the procedure is transductive in the experiments.Source: §3.5 · pp. 5–7

4 EXPERIMENTS AND ANALYSIS

The experiments test whether in-domain training closes MM-IAU’s performance gap, isolate the roles of MLLM composition and defect evidence, and examine calibration, weighting, and team size on MMAD. Across these tests, SiGMA combines heterogeneous MLLMs with calibrated visual evidence and label-free reliability weighting.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses MMAD’s 39,670 questions across seven task types and reports per-task accuracy and their unweighted mean.For a fixed configuration, the controller uses no labels and is fit transductively on unlabeled evaluation predictions; each MLLM receives one reference and the expert uses a category-specific bank of at most 256 images.Source: §4.1 · p. 8

4.2 IS IN-DOMAIN TRAINING ENOUGH?

  • 4.0%↑ average accuracy over the strongest training-based method; SiGMA is 1.5%↓ below human experts on average and leads on four of seven tasks.Its largest remaining human gaps are anomaly detection (12.6%↓) and defect localization (12.7%↓).Source: §4.2 · p. 8

4.3 WHAT DOES EACH ROLE CONTRIBUTE?

  • On the two perception tasks, heterogeneous-agent composition improves accuracy by only 0.4%↑ and 0.6%↑, while it raises the other five tasks by 3.2%↑ on average.Adding the defect expert raises average accuracy by 2.0%↑ through anomaly detection and defect localization; reducing the base team to one agent lowers the other five tasks by 4.2%↓ on average.Source: §4.3 · p. 9

4.4 DOES EXPERT EVIDENCE NEED CALIBRATION?

  • 82.6% anomaly-detection accuracy with calibration and the coherence gate reduces the false-positive rate to 9.4%, from 24.2% without calibration.Task-wise competence adds 0.8%↑ on average and query-level evidence weighting adds 0.4%↑, mainly in anomaly detection (1.7%↑). Defect localization rises from 76.6% at K=1 to 79.6% at K=256, but this comparison also changes calibration availability.Source: §4.4 · p. 9

4.5 HOW MANY AGENTS ARE ENOUGH?

  • 84.4% accuracy with three agents is 0.8%↓ below the eight-agent peak; the full ten-agent pool reaches 84.9%, while retrieval alone reaches 78.9%.With one normal image, defect localization reaches 76.6%, above the best training-based method (75.5%).Source: §4.2 · p. 8, §4.5 · p. 9

5 CONCLUSION AND LIMITATIONS

The MMAD results suggest that in-domain training alone does not resolve MLLMs’ shared weakness in anomaly detection and defect localization. SiGMA combines heterogeneous MLLM agents, calibrated defect evidence, and label-free source weighting, while its evaluation and controller have defined scope boundaries.

5 CONCLUSION AND LIMITATIONS

  • On MMAD, SiGMA reaches 85.2% average accuracy, 4.0% above the strongest trained specialist and Gemini-2.5-Pro; a three-agent configuration with at most 9B parameters reaches 84.4%.Source: §5 · p. 10
  • SiGMA pairs frozen heterogeneous MLLM agents with a defect expert whose calibrated evidence improves detection and localization, then weighs sources by task and query without labels.The reported conclusion states that MLLMs improve semantic tasks while the defect expert improves detection and localization.Source: §5 · p. 10
  • Experiments use only MMAD, and controller reliability estimates come from the unlabeled evaluation batch, making the full result transductive.The benchmark spans four source datasets and seven tasks; majority voting over the same sources reaches 83.3% without batch-level statistics.Source: §5 · p. 10
  • SiGMA remains 12.6% below human experts in anomaly detection and 12.7% below in defect localization, and a three-agent team takes 1.5 s per question versus JUDO’s 1.2 s.Without the defect expert, defect localization drops by 13.8%.Source: §5 · p. 10

A EXPERIMENTAL SETUP DETAILS

The experimental setup evaluates MMAD question accuracy across seven task types, using a pool of open-source MLLMs and specified trained, large-scale, and optimization-based baselines. The reported hyperparameters are fixed after selection on MMAD answers.

A EXPERIMENTAL SETUP DETAILS

  • 39,670 MMAD questions span seven types, and results report accuracy per type plus the unweighted mean across types.Images come from MVTec-AD, VisA, MVTec-LOCO, and GoodsAD.Source: A EXPERIMENTAL SETUP DETAILS · p. 14
  • The candidate pool contains ten open-source MLLMs from eight model series; the main configuration uses eight base agents and three expert readers.Baselines include four training-based methods, four large-scale MLLMs, and one optimization-based method; pool models are also evaluated individually with one normal reference.Source: A EXPERIMENTAL SETUP DETAILS · p. 14
  • Hyperparameters were selected on MMAD answers with the agent team and then fixed for all reported questions.Table 3 lists the hyperparameters.Source: A EXPERIMENTAL SETUP DETAILS · p. 14, Table 3 · p. 14

B REFERENCE DATA AND EVALUATION PROTOCOL

SiGMA builds category-specific normal-reference galleries from training-partition images and uses retrieved context and references for its agents and expert. Team and hyperparameter choices use MMAD evaluation answers, while the fixed-configuration controller estimates reliability from unlabeled evaluation predictions.

B REFERENCE DATA AND EVALUATION PROTOCOL

  • The searcher’s text branch uses MMAD-provided domain knowledge for product semantics, normal structures, and defect descriptions; it is separate from the normal-image gallery and excludes evaluation answers.The gallery uses normal images from each category’s source-dataset training partition, with evaluation queries excluded.Source: B REFERENCE DATA AND EVALUATION PROTOCOL · pp. 14–15
  • Each MLLM receives one retrieved normal reference, while the defect expert uses a category-specific bank of at most 256 images.At K=1, the expert shares the MLLM’s reference and noise-robust calibration is skipped; for K>1, the expert uses a larger bank for calibration.Source: B REFERENCE DATA AND EVALUATION PROTOCOL · p. 15
  • The best agent subset for each team size was chosen by comparing MMAD evaluation answers, so the reported curve is not an automatic team-selection procedure for unseen data.The team choice is separate from the fixed-team controller’s source statistics and per-question predictions, which are computed without answer labels.Source: B REFERENCE DATA AND EVALUATION PROTOCOL · p. 15
  • Reported hyperparameters, including calibration and controller settings, were selected by comparing candidates using MMAD evaluation answers and then fixed.The resulting accuracy is not performance under label-free hyperparameter selection on an unseen benchmark.Source: B REFERENCE DATA AND EVALUATION PROTOCOL · p. 15
  • Controller statistics are computed from unlabeled MMAD evaluation predictions without a separate query-calibration split, so the reported controller operates on a transductive batch.A historical-batch protocol for sequential inference is described but unevaluated; reported MMAD accuracies use the transductive batch protocol.Source: B REFERENCE DATA AND EVALUATION PROTOCOL · p. 15

C RELIABILITY CONTROLLER: DERIVATION AND PROPERTIES

The reliability controller uses weighted, regularized updates and has stated convergence properties, with important limits on how its aggregation should be interpreted. In the reader-team ablation, three readers achieved the highest unrounded accuracy among the tested team sizes.

C RELIABILITY CONTROLLER: DERIVATION AND PROPERTIES

D EXPERT-READER TEAM SIZE

E PER-TASK COMPONENT ABLATION

The ablations show distinct task profiles for retrieval and composition, expert evidence, and reliability control. Removing expert evidence or base-agent diversity affects different groups of tasks.

E PER-TASK COMPONENT ABLATION

F USING EXPERT EVIDENCE

Raw anomaly maps can improve localization while harming detection, and calibration and spatial gating reduce that trade-off. With prediction sources fixed, task-wise competence and query-level evidence weighting increase average accuracy.

F USING EXPERT EVIDENCE

G TRAINING, INFERENCE, AND SERVING COST

SiGMA avoids in-domain parameter updates but incurs test-time costs from its multiple models and reference-based processing. Serving measurements show a lower-cost three-agent configuration with a modest accuracy decrease.

G TRAINING, INFERENCE, AND SERVING COST

H QUALITATIVE EFFECTS OF NOISE-ROBUST CALIBRATION

Noise-Robust Calibration suppresses query–reference differences associated with normal variation while preserving more localized defect evidence. On normal queries, it attenuates benign responses before the expert-reader route is considered.

H QUALITATIVE EFFECTS OF NOISE-ROBUST CALIBRATION

  • Calibration on eight defective queries suppresses responses typical of normal variation and retains more localized evidence for expert readers.Figure 4 shows each query through five comparison stages; raw residuals can reflect pose, placement, texture, or illumination differences beyond the defect.Source: H QUALITATIVE EFFECTS OF NOISE-ROBUST… · p. 20, Figure 4 · p. 20
  • Figure 4’s raw residual maps include query–reference differences beyond the defect, while calibrated maps and overlays concentrate evidence for expert readers.Source: Figure 4 · p. 20, H QUALITATIVE EFFECTS OF NOISE-ROBUST… · p. 20
  • On eight normal queries, calibration attenuates benign query–reference responses before the expert-reader route, while a spatial-coherence gate determines whether anomaly questions reach expert readers.The visual comparison illustrates suppression of spurious evidence, not the final answer for each query.Source: H QUALITATIVE EFFECTS OF NOISE-ROBUST… · p. 21, Figure 5 · p. 21

Related papers

Loading 2610.12310v1…