PaperlayerView exact revision on arXiv

Source-linked AI summary

LocalAgent: Collaborative Agentic Verification for Fine-Grained Instance-Level Consistency

Jiachen Guo, Xinshan Zhu, Boyun Wang, Yanyan Liang, Jun Wan, Feng Lin, Ajian Liu

arXiv:2610.29174v1cs.CV

TL;DR

Fine-grained instance verification asks whether images depict the same physical object despite changing viewpoints, illumination, and backgrounds, a task where MLLMs can overlook subtle local cues. LocalAgent combines global MLLM reasoning and region localization with specialist appearance and geometric verification, training the policy to invoke verification selectively. It improves mean accuracy over the strongest CoT-RFT baseline while using a verifier on a minority of samples.

  • Problem

    MLLMs can miss subtle local evidence when deciding whether images depict the same physical object across changes in viewpoint, illumination, and background.

    Source: §1 · pp. 1–2

  • Method

    LocalAgent has an MLLM localize suspicious regions, LEV perform appearance and geometric matching, and an auxiliary-capped RFT policy learn when to invoke verification.

    Source: §1 · pp. 2–3

  • Results

    LocalAgent improves mean accuracy by 4.9% over the strongest CoT-RFT baseline while invoking a verifier on only 12.13% of samples.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across six diverse data sources, LocalAgent improves mean accuracy while preserving general multimodal capabilities and narrowing the gap to closed-source counterparts.

    Source: §5 · p. 10

    LEV’s evidence depends on the MLLM’s proposed regions, and the current policy does not jointly aggregate evidence across multiple-choice candidates.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have made rapid progress in general visual understanding, yet they still struggle with fine-grained instance-level consistency verification, where the goal is to determine whether two images depict the exact same physical object under varying viewpoints, illumination, and backgrounds. This task requires both global semantic perception and precise examination of subtle local evidence, since semantically similar objects may differ only in small suspicious regions. To address this challenge, we propose LocalAgent, a collaborative agentic framework that integrates the complementary strengths of a generalist MLLM and lightweight specialist verifiers. The MLLM first reasons globally to identify suspicious local regions, which are then delegated to a Local Evidence Verifier (LEV) toolkit for fine-grained appearance and geometric matching. LEV comprises Local Appearance Consistency (LAC), which estimates appearance similarity under illumination and color changes, and Local Geometric Correspondence (LGC), which estimates spatial correspondence under viewpoint changes. LEV returns structured scalar evidence that the MLLM integrates for the final decision. We further introduce an auxiliary-capped reinforcement fine-tuning (RFT) strategy, in which answer correctness dominates auxiliary tool-use and format signals, enabling the policy to learn when verification is necessary. Experiments show that LocalAgent improves mean accuracy by 4.9% over the strongest CoT-RFT baseline while invoking a verifier on only 12.13% of samples and preserving the general multimodal capabilities of the underlying MLLM. The open-source repository is available at https://github.com/Firefly-zoo/LocalAgent-main.

1 Introduction

Fine-grained instance verification requires distinguishing the same physical object across changing views and conditions, but MLLMs can miss decisive local cues. LocalAgent delegates localized evidence checks to specialist verifiers and trains the MLLM to invoke them selectively.

1 Introduction

  • Instance-level verification asks whether images show the same physical object despite viewpoint, illumination, and background changes, often requiring subtle local distinctions.Global semantic similarity can overlook decisive texture, geometry, or brand-specific cues.Source: §1 · pp. 1–2
  • LocalAgent assigns global perception and suspicious-region localization to the MLLM, then uses LEV for appearance and geometric matching and returns structured evidence for the decision.The evidence is integrated through textual reasoning rather than re-fed as visual content.Source: §1 · pp. 2–3
  • LocalAgent improves mean accuracy by 4.9% over the strongest CoT-RFT baseline while invoking a verifier on only 12.13% of samples.The auxiliary-capped RFT strategy prioritizes answer correctness and preserves the underlying MLLM’s general multimodal capabilities.Source: §1 · p. 2
  • Auxiliary-capped RFT combines answer-correctness priority with tool-use and format signals to teach selective verification behavior.Weak behavioral priors guide tool invocation without additional human annotation.Source: §1 · pp. 2–3

2 Related Work

Prior work documents fine-grained visual weaknesses in MLLMs and explores reasoning and tool-use strategies. LocalAgent targets instance verification with invariant matching on localized regions and structured evidence.

2.1 Fine-Grained Recognition and MLLMs

  • Fine-grained recognition work reports that MLLMs miss visual details, while TWIN still relies on global perception without explicit invariant matching under complex transformations.Prior benchmarks describe missed patterns and correspondences, and vision encoders optimized for cross-modal alignment can attenuate instance-specific detail.Source: §2.1 · p. 3

2.2 Multimodal Reasoning and Tool-Augmented Inference

  • Reasoning and tool-augmented methods improve general decision-making, but LocalAgent instead delegates suspicious regions to LEV for raw-feature invariant matching and structured evidence.Its auxiliary-capped RFT strategy learns selective tool invocation without hand-crafted triggering rules.Source: §2.2 · p. 3

3 Methodology

LocalAgent divides instance verification among an MLLM controller, local appearance and geometric verifiers, and an RFT-trained policy that decides when to verify. The components exchange localized regions and structured scalar evidence rather than relying solely on global semantic representations.

3.1 Overview

  • The framework assigns global perception and suspicious-region localization to the MLLM, evidence extraction to LEV, and verification choice to an RFT-trained policy.LEV operates on vision-only features and returns structured scalar indicators.Source: §3.1 · p. 3

3.2 The MLLM as Spatial Controller

  • The MLLM answers directly when its global assessment is sufficient and localizes suspicious regions when it is inconclusive, then integrates LEV evidence with its initial reasoning state.The process closes the loop without re-feeding raw visual content through the vision encoder.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3 Local Evidence Verifier Toolkit

  • LEV uses vision-only features to perform complementary appearance and geometric matching, returning scalar evidence for integration by the MLLM.LAC targets photometric variation with DINO-based patch matching, while XFeat-based LGC targets viewpoint changes through keypoint geometry.Source: §3.3 · p. 4

3.3.1 Local Appearance Consistency (LAC)

  • LAC compares patch-level DINO features across context and crop views with color, grayscale, and flip variants, then aggregates appearance scores into a structured evidence triplet.The triplet summarizes favorable-view similarity, central tendency, and variation across views.Source: §3.3.1 · pp. 4–5

3.3.2 Local Geometric Correspondence (LGC)

  • LGC uses XFeat descriptors and robust geometric estimation to filter candidate matches into inliers, then summarizes correspondence strength, match quantity, spatial coverage, and stability.Its view variants emphasize contrast and orientation because geometric estimation is less sensitive to color distribution.Source: §3.3.2 · p. 5

3.4 Learning When to Verify

  • The verification policy uses GRPO-based RFT with answer correctness as the dominant reward and auxiliary tool-use and format signals to discourage indiscriminate invocation.Weak priors label samples as Use_tool, No_tool, or Uncertainty; auxiliary rewards are capped on incorrect answers and bounded below.Source: §3.4 · pp. 5–6

4 Experiments

Experiments evaluate LocalAgent across fine-grained verification datasets and general capability benchmarks. The results report improved mean accuracy over CoT-RFT with selective verifier use, alongside ablations and diagnostics that reveal where performance and transfer remain limited.

4.1 Experimental Setup

  • Evaluation covers TWIN and five additional data sources, 12k queries, pairwise and multiple-choice tasks, plus ten general vision-language and text-only benchmarks.RFT training emphasizes highly similar hard negatives and positive pairs with large viewpoint, illumination, or background changes.Source: §4.1 · pp. 6–7

4.2 Main Results

  • 4.9% higher mean accuracy than CoT (Qwen3-VL-4B), with verifier calls on 12.13% of samples; LocalAgent improves both mean task-format accuracies but transfers unevenly.It improves over CoT-RFT on four of five out-of-distribution sources, while performance on ILIAS and TWIN-Eval is mixed; on TWIN-Eval it remains below the untrained base model.Source: §4.2 · p. 7
  • 87.55% pairwise and 80.70% mean accuracy for LocalAgent, versus 76.81% and 75.64% with crop-based verification; multiple-choice accuracy is slightly lower.The comparison uses the same training and evaluation protocol; the paper notes that multiple-choice verification requires joint evidence comparison across three candidates.Source: §4.2 · p. 7, Table 4 · p. 8

4.3 Ablation Study

  • LAC and LGC each outperform CoT-RFT individually, while their integration reaches 80.70% mean accuracy; the weak prior and auxiliary cap also contribute to performance.Without the weak prior, pairwise invocation rises to 38.45% and mean accuracy drops by 1.31%; replacing the auxiliary cap with hard gating drops accuracy by 0.47%.Source: §4.3 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 7

4.4 Diagnostic Analysis

  • 7.85% pairwise and 16.40% multiple-choice tool-call rates accompany 336.86 ms mean latency, versus 610.41 ms for always-call; calls concentrate on harder cases.Selective invocation reduces latency by avoiding unnecessary verifier executions, while direct inference averages 258.13 ms.Source: §4.4 · pp. 8–9, Table 5 · p. 8, Table 6 · p. 9
  • 75.06% mean accuracy for localized DINO and 71.11% for localized XFeat, still below LocalAgent’s 80.70%; DINO reaches 75.36% multiple-choice accuracy with a post-hoc evaluation threshold.MLLM-localized matching improves the mean accuracy of DINO from 74.23% to 75.06% and XFeat from 69.23% to 71.11%.Source: §4.4 · p. 9, Table 7 · p. 9

4.5 General Capability Evaluation

  • LocalAgent matches or slightly exceeds Qwen3-VL-4B on every evaluated general-purpose benchmark across vision-language and text-only tasks.The evaluation spans ten benchmarks.Source: §4.5 · p. 9, Table 8 · p. 9

4.6 Qualitative Analysis

  • Qualitative cases show low LAC scores overturning deceptive global similarity, while high scores confirm identity despite packaging-induced appearance shifts.The examples illustrate how structured scalar evidence helps resolve ambiguous pairwise cases.Source: Figure 3 · p. 8, §4.6 · p. 9

5 Conclusion

LocalAgent combines MLLM-led global perception and suspicious-region localization with LEV appearance and geometric evidence, using auxiliary-capped RFT to learn when verification is needed. The authors report improved mean accuracy while preserving general multimodal capabilities, alongside limitations in localization dependence, multiple-choice evidence aggregation, TWIN-Eval performance, and speed.

5 Conclusion

  • LocalAgent pairs MLLM global perception and suspicious-region localization with LEV appearance and geometric evidence, which the MLLM integrates through textual reasoning.LEV returns structured scalar indicators rather than re-feeding regions through the vision encoder.Source: §5 · p. 10
  • Auxiliary-capped RFT enables the policy to learn when verification is necessary.Source: §5 · p. 10
  • The authors report improved mean accuracy while preserving general multimodal capabilities and narrowing the gap to closed-source counterparts.Source: §5 · p. 10
  • LEV evidence depends on MLLM-proposed regions, multiple-choice evidence is not jointly aggregated, and LocalAgent remains below the base model on TWIN-Eval and 30.5% slower than direct inference.Source: §5 · p. 10

A Localization Quality Analysis

The analysis evaluates whether the base Qwen3-VL-4B model localizes regions that cover identity-discriminative components and remain usable for verification. An independent judge accepts more predicted boxes in pairwise than multiple-choice queries.

A Localization Quality Analysis

  • Localization quality is assessed on 12k evaluation samples across pairwise and multiple-choice formats using base Qwen3-VL-4B predicted coordinates.Pairwise queries receive one box per image; multiple-choice queries first select a suspicious reference, then box it and the query image.Source: A Localization Quality Analysis · p. 13
  • An independent Qwen3-VL-30B-A3B-Instruct judge accepts 94.22% of pairwise boxes and 87.32% of multiple-choice boxes as covering discriminative components and remaining usable for verification.Source: A Localization Quality Analysis · p. 13
  • The lower multiple-choice acceptance rate reflects its additional candidate-selection step before localization.Source: A Localization Quality Analysis · p. 13

B TWIN-Eval Regression Analysis

The TWIN-Eval analysis finds that LocalAgent’s aggregate regression is concentrated in samples outside the training-emphasis-aligned group, while it leads the base model and CoT-RFT on aligned cases. The analysis describes this pattern but does not identify a single cause for the overall decrease.

B TWIN-Eval Regression Analysis

C Failure Mode Analysis

The failure-mode audit identifies localization and visibility problems, ambiguous verifier evidence, incomplete region coverage, and policy or evidence-integration errors. The authors relate these limitations to controller-bounded localization and the lack of joint evidence aggregation across multiple-choice candidates.

C Failure Mode Analysis

  • Localization and visibility failures occur when predicted boxes miss identifying regions or occlusion, blur, and tiny details leave insufficient evidence for verification.Source: C Failure Mode Analysis · p. 14
  • Repetitive textures, large viewpoint changes, or sparse keypoints can produce weak or conflicting LAC and LGC evidence.Source: C Failure Mode Analysis · p. 14
  • A single invocation may miss some inconsistent regions when several exist, consistent with the controller-bounded design and absent joint aggregation across multiple-choice candidates.Source: C Failure Mode Analysis · p. 14
  • The MLLM may omit useful verifier calls, invoke unnecessary verification, or misweight returned evidence against its global impression.Source: C Failure Mode Analysis · p. 14

D Implementation Details

The implementation details specify the GRPO/LoRA training setup and report compute resources; Figure 4 is captioned as illustrating failure modes.

D Implementation Details

  • GRPO training uses LoRA with rank 16 and αLoRA = 32 on the language backbone, while the visual encoder and multimodal aligner are frozen.The main configuration uses five NVIDIA H20 GPUs: one for rollout and four for DDP training.Source: D Implementation Details · pp. 14–15
  • Figure 4 is captioned as illustrating failure modes.Source: Figure 4 · p. 15
  • Training uses eight rollout responses per prompt, AdamW at 1 × 10−5 learning rate, cosine decay with 0.05 warmup, and gradient clipping at 0.5.Weight decay is 0.1, with β1 = 0.9 and β2 = 0.95.Source: D Implementation Details · p. 15

E Reward Sensitivity Analysis

The sensitivity analysis varies the auxiliary-reward bounds for incorrect predictions and finds that removing or loosening the bounds lowers mean accuracy.

E Reward Sensitivity Analysis

  • Removing the auxiliary reward on incorrect predictions lowers mean accuracy by 1.2%, while loosening the bounds lowers it by 3.6%.The authors conclude that bounding the auxiliary contribution on incorrect predictions matters for final performance.Source: E Reward Sensitivity Analysis · p. 15

F Statistical Reliability

Paired-bootstrap intervals and exact McNemar tests assess LocalAgent’s accuracy gain over CoT-RFT across three evaluation splits. The reported gain is positive and statistically significant on all three.

F Statistical Reliability

  • The paired mean accuracy difference is positive with 95% confidence intervals excluding zero, and exact McNemar tests are significant on all three splits.Table 11 reports paired-bootstrap confidence intervals and exact McNemar tests for LocalAgent’s accuracy gain over CoT-RFT.Source: Table 11 · p. 15, F Statistical Reliability · p. 15

NeurIPS Paper Checklist

The checklist reports that the paper’s contributions and main accuracy claim are supported by its experiments. It also records disclosures on reproducibility, statistical significance, limitations, ethics, and asset use.

NeurIPS Paper Checklist

  • The checklist says the abstract and introduction accurately reflect the framework, LEV toolkit, auxiliary-capped RFT strategy, and reported 4.9% mean-accuracy improvement over CoT-RFT.Source: NeurIPS Paper Checklist · p. 16
  • The authors identify a TWIN-Eval regression, dependence on localization quality, and lack of joint evidence aggregation in the multiple-choice task as limitations.Source: NeurIPS Paper Checklist · p. 16
  • The authors report that code is publicly available, evaluation data come from public benchmarks, and training, evaluation, and reproduction details are provided across the paper and appendices.The response identifies the repository and says Section 4 and Appendix D detail data, benchmarks, baselines, and hyperparameters.Source: NeurIPS Paper Checklist · pp. 17–18
  • Appendix F reports paired-bootstrap 95% confidence intervals and exact McNemar tests, with significant gains over CoT-RFT on overall, pairwise, and multiple-choice splits.Source: NeurIPS Paper Checklist · p. 18
  • The checklist reports that the work uses public models and benchmarks, involves no human subjects, and releases no new public assets.It also says external assets are credited and that the paper does not release models or datasets considered high risk for misuse.Source: NeurIPS Paper Checklist · pp. 19–21

Related papers

Loading 2610.29174v1…