PaperlayerView exact revision on arXiv

Source-linked AI summary

ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning

Shengyuan Ding, Xinyu Fang, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiangyu Zhao, Haodong Duan, Xiaoyi Dong, Jianze Liang, Bin Wang, Conghui He, Dahua Lin, Jiaqi Wang

arXiv:2512.05111v1cs.CV

TL;DR

Complex multimodal judgments can require multi-step retrieval and verification, capabilities that existing reward models lack. ARM-Thinker makes reward judgment an active tool-using process and reports gains across reward-modeling, tool-use, and general-reasoning benchmarks.

  • Problem

    Existing reward models often judge multimodal responses in one pass without tools, while complex tasks require multi-step evidence retrieval, visual localization, and verification.

    Source: §1 · pp. 1–2

  • Method

    ARM-Thinker uses an explicit think–act–verify loop to plan and invoke multimodal tools, then issue evidence-grounded scalar scores with interpretable rationales.

    Source: §1 · p. 2

  • Results

    Across benchmarks, ARM-Thinker reports +16.2% average reward-modeling gains, +9.6% tool-use gains, and +4.2% general-reasoning gains.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    The authors argue agentic capabilities are crucial for next-generation reward models and evaluate them through verifiable, multi-step evidence-grounded reasoning.

    Source: §6 · p. 11

Abstract

from arXiv · show

Reward models are critical for aligning vision-language systems with human preferences, yet current approaches suffer from hallucination, weak visual grounding, and an inability to use tools for verification, limiting their reliability on complex multimodal reasoning tasks. We present ARM-Thinker, an A}gentic multimodal Reward Model that autonomously invokes external tools (e.g., image cropping, doc page retrieval) to ground judgments in verifiable evidence, replacing static, non-interactive reward scoring. This enables the model to verify fine-grained visual details, cross-reference multi-page evidence, and validate reasoning claims, which are capabilities absent in existing reward models. We train ARM-Thinker with multi-stage reinforcement learning, jointly optimizing tool-calling decisions and judgment accuracy. To evaluate agentic reward modeling, we introduce ARMBench-VL, comprising three benchmarks that assess fine-grained visual grounding (image-level tools), multi-page document understanding (retrieval tools), and instruction following (text-level verification). ARM-Thinker achieves +16.2% average improvement on reward modeling benchmarks, +9.6% on tool-use tasks, and outperforms baselines on multimodal math and logical reasoning benchmarks. Our results demonstrate that agentic capabilities significantly enhance both accuracy and interpretability of reward models.

1 Introduction

ARM-Thinker addresses multimodal judgment that requires evidence retrieval, visual grounding, and verification by making reward scoring an active tool-using process. It introduces ARMBench-VL and reports gains across reward-modeling, tool-use, and general-reasoning benchmarks.

1 Introduction

  • Multimodal judgments can require cross-page evidence retrieval, fine-grained visual grounding, and planning when to call and parameterize tools.Existing rule-based verifiers and single-pass reward models struggle with complex tasks and cannot retrieve or verify cited content.Source: §1 · pp. 1–2
  • ARM-Thinker plans reasoning steps, invokes multimodal tools to gather evidence, and produces an evidence-grounded scalar score with an interpretable rationale.Its motivation is to judge answers by accessible evidence rather than surface fluency.Source: §1 · pp. 2–3
  • ARMBench-VL evaluates multi-step, verifiable reasoning across fine-grained perception, document navigation, and instruction following.Its tasks measure the ability to construct a verifiable chain of evidence.Source: §1 · pp. 2–3
  • A scalable data-generation pipeline constructs discriminative preference pairs through counterfactuals and perturbations, then filters them for evidence validity.The pipeline is designed to address scarce labels with scalable, evidence-grounded agentic data.Source: §1 · p. 3
  • ARM-Thinker reports +16.2% average reward-modeling, +9.6% tool-use, and +4.2% general-reasoning gains, with ARM-Thinker-7B competitive with or sometimes superior to proprietary models.The overview figure depicts gains across multiple benchmarks.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Work

Prior multimodal tool-use work and multimodal reward models leave gaps in tool diversity, task scope, and tool-assisted judgment. ARM-Thinker combines an agent loop with a staged training pipeline and flexible tool selection.

Multimodal Models with Tool Use

  • Multimodal tool-use research includes image zooming and region selection, but faces scarce data, narrow scenarios, and limited tool operations.Some work also combines tool use with long-range reasoning.Source: §2 · p. 3

Multimodal Reward Models

  • Existing multimodal reward models focus on training data and long-horizon reasoning, yet lack tool calls for challenging image, document, and instruction-following tasks.The cited work describes easier tasks with simple outputs as a reason tools have not been provided.Source: §2 · pp. 3–4
  • ARM-Thinker flexibly selects tools within an iterative think–act–observe loop, while its training pipeline pairs SFT and cold start with two GRPO stages.The first GRPO stage encourages correct tool calls; the second refines accuracy using verifiable rewards balancing correctness and tool efficiency.Source: Figure 2 · p. 4, §2 · p. 4

3 ARM-Thinker

ARM-Thinker is an active multimodal reward model that iteratively reasons with tools and observations before judging. Its training combines preference data, trajectory supervision, and staged rewards that shift from tool behavior to accuracy.

3.1 Multimodal Tools Integrated Agent Loop

  • ARM-Thinker iteratively plans, invokes multimodal tools, incorporates returned observations, and produces interpretable reasoning trajectories before judging.Its tools include instruction checks, image crop and zoom, and document retrieval.Source: §3.1 · pp. 4–5

3.2 Data Gathering for ARM-Thinker Training

  • Preference training combines general QA supervision with tool-specific data; generated flawed responses provide diverse negatives, and overly similar pairs are removed.The tool-specific sources cover image inspection, instruction checks, and document retrieval.Source: §3.2 · p. 5

3.3.1 SFT & Cold Start

  • Filtered, tool-augmented multimodal reasoning trajectories provide SFT and cold-start supervision for structured reasoning and tool use.Trajectories are filtered for format, accuracy, and tool-calling behavior; trivial samples are removed before trajectory generation.Source: §3.2 · p. 5, §3.3.1 · p. 6

3.3.2 Two-Staged GRPO training

  • Two-stage GRPO first rewards valid tool exploration, then shifts toward factual accuracy and credits successful tool use when it contributes to a correct prediction.The staged design separates learning appropriate tool use from accurate and efficient reasoning.Source: §3.3.2 · p. 6

4 ARMBench-VL

ARMBench-VL evaluates agentic reasoning and tool-calling across multimodal tasks. It combines three tool-requiring task families with reconstructed preference examples and a freely selectable toolkit.

4 ARMBench-VL

  • ARMBench-VL targets agentic reasoning and tool-calling behaviors across modalities, addressing limitations of existing LVLM-based verifiers.The benchmark is designed to assess verifiable, multi-step reasoning.Source: §4 · p. 7

4.1 Benchmark Motivation

  • The three tasks require crop or zoom for local image details, page retrieval for long-document QA, and instruction checks for multiple response constraints.Instruction-following examples may require checking output format or required keywords.Source: §4.1 · pp. 7–8

4.2 Construction of ARMBench-VL

  • For perception and document QA, construction expands answers and generates style-matched incorrect responses, then filters pairs for hallucinated correctness cues and overly simple items.Instruction-following responses are filtered from Qwen2.5-VL-7B outputs for the reward model to judge against stated constraints.Source: §4.2 · p. 8

4.3 Benchmark statistics

  • 1,499 questions span three tasks and diverse visual domains and capabilities; ARMBench-VL is the first reward benchmark to provide a freely selectable toolkit.The evaluated capabilities include perception, OCR, recognition, and long-document understanding.Source: §4.3 · p. 8

5 Experiments

Experiments evaluate ARM-Thinker on reward modeling, visual tool use, and multimodal reasoning, then examine how tool use and reward design affect performance. The reported results show gains across these benchmarks, with ablations highlighting the role of adaptive tool-use rewards.

5.1 Experimental Setup

  • Experiments cover reward modeling, tool-assisted visual use, and general reasoning benchmarks, comparing ARM-Thinker-7B with general-purpose, reward, and visual-reasoning models.The reward-modeling evaluation includes RewardBench-2, VL-RewardBench, and ARMBench-VL; tool-use evaluation includes four visual benchmarks, and general reasoning covers six benchmarks.Source: §5.1 · p. 9

5.2 Results on Reward Benchmarks

  • ARM-Thinker-7B reaches 67.8% accuracy on VL-RewardBench, surpassing the baseline by 17.7%, and scores 64.6% on ARMBench-VL (+18.5% on baseline).It also yields a 12.5% gain on RewardBench-2, with balanced ARMBench-VL gains across fine-grained perception, instruction following, and document understanding.Source: Table 2 · p. 8, §5.2 · p. 9

5.3 Results on Tool Use (Think-with-Images) Benchmarks

  • ARM-Thinker-7B improves over Qwen2.5-VL-7B by an average of 9.6% and attains 76.5% overall accuracy on think-with-images benchmarks, matching or exceeding Mini-o3.The model also outperforms Qwen3-VL-8B across all benchmarks and acquires tool-calling ability without explicit tool-use supervision.Source: Table 3 · p. 9, §5.3 · p. 10

5.4 Results on General Benchmarks

  • ARM-Thinker-7B improves over Qwen2.5-VL-7B by 10.9% on WeMath and 8.7% on LogicVista.The authors suggest that verification-task reward training may improve general reasoning through logical analysis and error detection.Source: Table 4 · p. 9, §5.4 · p. 10

5.5 Ablation Studies

  • With tools enabled, ARM-Thinker-7B gains substantially over its tool-disabled performance while remaining comparable to Qwen2.5-VL-7B without tools.The baseline rarely invokes tools without explicit supervision, whereas ARM-Thinker learns when tools are useful, including for zoom-in and page-retriever functions.Source: §5.5 · pp. 10–11, Table 5 · p. 10
  • ARM-Thinker’s adaptive reward attains the highest final accuracy while stabilizing tool calls near 1.12; accuracy-only rewards underuse tools, while fixed tool rewards drive overuse.The accuracy-only design finishes at 77.5% with a call rate near 0.7, while the fixed-tool design reaches 78.5% with a call rate rising to about 1.15.Source: §5.5 · p. 11, Figure 4 · p. 11

6 Conclusion

ARM-Thinker reports average gains across reward modeling, tool-assisted reasoning, and multimodal reasoning, and the authors plan to extend it to more tools.

  • +16.2% average improvement on reward modeling benchmarks is reported for ARM-Thinker.Source: §6 · p. 11
  • +9.6% on tool-assisted reasoning and +4.2% on multimodal reasoning are reported on average.Source: §6 · p. 11
  • The authors plan to extend the method to a broader set of tools and expand its applicability.Source: §6 · p. 12

Outline

The appendix outlines materials on the models and data, ARMBench-VL statistics, qualitative cases, tool implementation, and experimental prompts.

  • Section A covers the models, datasets, benchmark statistics, and multimodal reward-modeling training data.Source: Outline · p. 16
  • Section E reports ARMBench-VL statistics and describes implementation details for the multimodal tools.Source: Outline · p. 16
  • Section C presents qualitative response cases, while Section F provides the full experimental prompt list.Source: Outline · p. 16

A.1 Models

The study uses a 7B Qwen2.5-VL backbone for ARM-Thinker and compares it with general LVLMs, reward models, visual tool-use models, and larger reasoning systems.

  • ARM-Thinker-7B adds a think–act–observe loop and multi-stage GRPO training to Qwen2.5-VL-7B without changing its 7B-parameter size.Source: §A.1 · p. 16
  • General-purpose comparison models include Qwen3-VL-8B, InternVL3-8B, InternVL3.5-8B, and GPT-4o.Source: §A.1 · p. 16
  • ARMBench-VL includes 2-way and 4-way judging; its evaluation templates specify reasoning and structured responses, including one correct and three plausible incorrect long responses.Source: Figure 6 · p. 17, Figure 7 · p. 17, Figure 8 · p. 18
  • UnifiedReward-7B is evaluated on RewardBench-2, VL-RewardBench, and ARMBench-VL to isolate the agent loop and tool-use contribution.Source: §A.1 · p. 17
  • Without curated tool-use demonstrations, ARM-Thinker-7B achieves accuracy comparable to or better than specialized visual tool-use models.Source: §A.1 · p. 18
  • Gemma-3-27B and InternVL3-8B provide larger or specialized reasoning references across six multimodal math and logic benchmarks.Source: §A.1 · p. 18

A.2 Training Data

The appendix describes ARM-Thinker training data, ARMBench-VL task composition and sample counts, qualitative examples, and broader directions for the framework.

A.2 Training Data

  • SFT combines multimodal QA preference data with image zoom, instruction-checking, and document-retrieval data; GRPO samples about 20k DeepEyes and 4k MP-DocVQA queries.Instruction-following tool-use data is omitted from GRPO; the authors report that the trained model generalizes to those tasks.Source: §A.2 · pp. 18–20
  • The ARMBench-VL generation template rewrites image questions as descriptive queries and pairs correct scene descriptions with subtly misleading alternatives.Source: §A.2 · p. 19, Figure 9 · p. 19

B ARMBench-VL Statistics

  • ARMBench-VL contains 550 fine-grained perception samples, 460 long-document QA samples, and 489 instruction-following samples.Perception includes 295 general pairwise, 163 2-way, and 92 4-way items; document QA has 173 single-response and 287 2-way items, while instruction following is single-response only.Source: B ARMBench-VL Statistics · p. 20

C Qualitative Case Study

D Broader Impact and Future Directions

E Implementation Details of Multimodal Tools

The implementation provides function-calling tools for text validation, image zooming, and document-page retrieval, alongside standardized evaluation and data-construction prompts.

E Implementation Details of Multimodal Tools

  • Document retrieval combines query-based top-k page search, returned as a stitched image, with direct indexed page access for sequential inspection.The search tool retrieves up to five pages by default; the index tool bypasses dense retrieval.Source: E Implementation Details of Multimodal… · pp. 22–23
  • Image zooming crops normalized-coordinate bounding boxes, validates and expands small regions, and can upsample tiny crops before returning them for continued reasoning.The minimum crop width and height are 28 pixels in the experiments; upsampling is optional and uses bicubic interpolation.Source: E Implementation Details of Multimodal… · p. 23
  • Textual validators check structural, lexical, keyword, and numeric-format constraints, returning a boolean check result through the shared tool interface.The checks include length and segmentation bounds, required or forbidden strings, keyword counts, and decimal precision.Source: E Implementation Details of Multimodal… · pp. 23–24

F Prompts

  • Evaluation uses standardized prompts for single-response constraint judgments, pairwise comparisons, and agent-style reasoning with controlled tool use and explicit answer formatting.The pairwise prompt structure extends to four or more candidates, although the figure illustrates two-way comparison.Source: F Prompts · p. 24
  • Data-construction prompts generate a solution-faithful response with plausible incorrect alternatives and image-grounded descriptions contrasting consistent and inconsistent details.The image-centric prompts rewrite questions around solution-relevant regions to construct contrastive pairs for judging description quality.Source: F Prompts · pp. 24–25
Loading 2512.05111v1…