PaperlayerView exact revision on arXiv

Source-linked AI summary

Beacon: Knowing When and How to Perform Agentic Visual Reasoning

Qixun Wang, Yang Shi, Letian Cheng, Zhuoran Zhang, Yan He, Yuqi Tang, Qi Zhang, Xinlei Yu, Ruizhe Chen, Tianrun Xu, Yuanxing Zhang, Pengfei Wan, Haotian Wang, Xianghua Ying

arXiv:2607.28595v1cs.CV

TL;DR

Agentic visual reasoning lacks reliable evidence that models know when tools are necessary or use them without offsetting harms. Beacon evaluates these gaps with Mode Adaptiveness and Tool Effect, then trains an adaptive tool-using model that achieves stronger benchmark performance and genuine tool-induced gains.

  • Problem

    Existing agentic visual reasoning models often use tools without reliably adapting to task necessity, while tool-induced gains can be offset by errors on tool-free-solvable problems.

  • Method

    Beacon combines Mode Adaptiveness and Tool Effect metrics with a data synthesis pipeline and reinforcement learning using Necessity-Aware Adaptive Reward and Hint-Guided Capability Expansion.

  • Results

    Beacon-RL-8B ranks first on 11 of 13 benchmarks and outperforms its base model by an average of 6.07 points.

  • Takeaways & Limitations

    Agentic visual reasoning should be evaluated by whether models know when tools are necessary and use them to produce genuine capability gains, not merely by tool-use frequency.

Abstract

from arXiv · show

The fundamental goal of agentic visual reasoning is to improve the success rate of multimodal large language models (MLLMs) on complex tasks, rather than merely equipping them with a sophisticated yet inefficient reasoning paradigm. In this work, we rethink agentic visual reasoning through two key dimensions of tool use: Mode Adaptiveness (MA) and Tool Effect (TE). Mode Adaptiveness characterizes whether an MLLM can recognize when tools are truly necessary and invoke them accordingly, thereby avoiding unnecessary computational overhead while improving performance on challenging problems that require tool assistance. Tool Effect characterizes the actual impact of tool use: tools should extend the model's capabilities on problems unsolvable through text-only reasoning, while avoiding additional errors on problems that the model can already solve without tools. We conduct a comprehensive analysis to quantify these two properties and empirically reveal that existing agentic visual reasoning models exhibit limited Mode Adaptiveness, while the gains produced by tool use on hard examples are largely offset by the harm introduced on easy examples that the models can already solve. Motivated by these observations, we propose Beacon, a novel agentic visual reasoning model that achieves stronger overall performance, improved Mode Adaptiveness, and genuine tool-induced performance gains. At the core of Beacon are the Necessity-Aware Adaptive Reward and the Hint-Guided Capability Expansion mechanism in the reinforcement learning stage, which respectively encourage adaptive tool invocation based on task necessity and strengthen the model's tool-use capability on the most challenging problems. Extensive experiments across diverse benchmarks demonstrate the strong overall performance of Beacon and its substantial improvements in both Mode Adaptiveness and Tool Effect.

1 INTRODUCTION

The paper argues that effective agentic visual reasoning requires adaptive tool invocation and tool use that provides genuine gains without introducing avoidable errors. It presents Beacon, whose training framework improves both properties and yields strong average performance across 13 benchmarks.

  • Contributions: The paper contributes systematic analyses of tool-invocation adaptiveness and tool effects, alongside SFT data synthesis and RL data curation strategies.These contributions target adaptive tool use and stronger performance on challenging problems.
  • Motivation: Existing agentic visual reasoning models generally invoke tools inadequately, with gains on hard problems largely offset by errors on text-solvable problems.This produces little improvement over text-only reasoning.
  • Method: Beacon is proposed to improve tool-invocation adaptiveness and the effectiveness of tool use.Its reinforcement learning framework includes a necessity-aware adaptive reward and a capability-expansion mechanism.
  • Method: Beacon can autonomously generate Python code for visual and numerical operations, including cropping, annotation, enhancement, rotation, stitching, and pixel-level computation.The system also supports complex numerical calculations during inference.
  • Results: 13 benchmarks show that Beacon achieves the best average performance.Further analysis finds stronger tool-invocation adaptiveness and the largest gap between tool-induced gain and harm.

2 RELATED WORK

Prior work develops agentic visual reasoning and adaptive tool-use methods, but diagnostic studies reveal unresolved questions about when tools are necessary and what benefits they provide. This work addresses these gaps with a unified, multi-run analysis framework and an RL approach targeting both adaptiveness and tool effect.

  • Agentic Visual Reasoning: Agentic visual reasoning methods interleave textual reasoning with visual interactions such as revisiting regions, cropping or editing images, generating visual representations, and executing code.These tools target fine-grained perception, visual manipulation, and numerical computation.
  • Adaptive and Efficient Tool Use: Recent RL studies encourage adaptive tool use because indiscriminate invocation can add unnecessary computation, but their adaptive objectives have limitations.The supplied discussion identifies CodeDance as one example of an approach using group-level accuracy to guide adaptive rewards.
  • Understanding the Actual Benefit of Visual Tools: Diagnostic studies question whether tool use itself drives performance gains, reporting marginal tool contributions, error reduction, and limited correction of problems unsolved without tools.Other analyses find that tool-induced gains and harms each contribute only marginally to overall performance.
  • Understanding the Actual Benefit of Visual Tools: Existing analyses lack a unified characterization of reasoning-mode adaptiveness and tool effect, and often evaluate tool-free solvability from a single inference run.The paper identifies this binary, single-run evaluation as a limitation of prior analyses.
  • Our Approach: The paper introduces a multi-run framework covering adaptiveness and tool effect, plus an RL framework designed to improve both aspects.Its adaptive reward uses mode-conditioned labeling based on tool-free and tool-assisted performance and online labeling based on the current policy.

3 ANALYSIS

The analysis evaluates agentic visual reasoning along Mode Adaptiveness and Tool Effect, measuring whether models invoke tools when necessary and whether tools extend hard-problem performance without harming easy-problem performance. Existing models show limited adaptiveness and little net benefit from tool use, motivating explicit adaptive objectives and capability expansion beyond pretraining.

  • Analysis framework: Mode Adaptiveness measures avoiding tools for text-easy problems while invoking them when text-only reasoning is insufficient.Text-easy and text-hard samples are identified using five text-only responses, requiring at least four or at most one correct response, respectively.
  • Analysis framework: Tool Effect measures gains on text-hard samples and harms on text-easy samples relative to tool-free reasoning.Tool-Gain captures text-hard samples solved with tools, whereas Tool-Harm captures text-easy samples answered incorrectly with tools.
  • Empirical observations: Tool-enabled reasoning provides little improvement over tool-free reasoning for most existing agentic visual reasoning models.Averaging over five runs shows that this limited gain is robust to sampling stochasticity.
  • Empirical observations: Existing models exhibit limited reasoning-mode adaptiveness, with most MAmean values close to or below the 50% always-tool or never-tool baseline.Many models consistently favor either tool-free or tool-assisted reasoning instead of adapting mode selection to problem difficulty.
  • Empirical observations: Most existing models have low Tool-Gain, and Tool-Gain does not substantially exceed Tool-Harm, yielding little net performance benefit.Tool use therefore provides limited benefit on text-hard samples despite also introducing limited harm on text-easy samples.
  • Issue attribution: The analysis attributes these issues to lacking an explicit adaptive objective and to RL with verifiable rewards struggling to expand capabilities beyond pretraining.These factors respectively limit adaptive behavior and tool-induced capability expansion.

4 METHOD

Beacon uses an SFT-then-RL paradigm to teach code use and improve reasoning-mode adaptiveness and genuine tool-induced gains. Its RL framework combines Necessity-Aware Adaptive Reward, which balances efficient tool use with task success, and Hint-Guided Capability Expansion, which supplies answer-free guidance on especially difficult problems.

  • Training Paradigm: Beacon first uses synthesized code-assisted trajectories in SFT, then applies adaptive reward shaping and hint-guided rollouts during RL.The training paradigm is designed to establish fundamental code-use capabilities before improving mode adaptiveness and tool-use gains.
  • Inference Procedure: Given an image-question pair, Beacon autonomously decides whether code is necessary and can iteratively execute Python before producing an answer.Code is emitted inside <tool_call>...</tool_call>, and execution results return as <observation>...</observation>.
  • Necessity-Aware Adaptive Reward: NAAR rewards correct text-only responses most when they suffice, reduces—but does not eliminate—reward for correct code responses, and fully rewards code when text-only reasoning fails.This soft preference discourages unnecessary tool calls while preserving the ability to use tools when necessary.
  • Hint-Guided Capability Expansion: HCE targets hard examples that conventional RLVR may waste, using an answer-free hint to expose reasoning and tool-use behaviors beyond the policy’s original distribution.When none of N = 8 original-policy responses is correct, Gemini-3.1-Pro generates the hint.
  • Hint-Guided Capability Expansion: Hints contain intermediate instructions, expected subgoals, and tool-use strategies without the final answer, and are removed from the prompt during policy optimization.This allows hinted trajectories to guide exploration without making the model rely on hints at inference time.

5 EXPERIMENTS

Experiments show that Beacon delivers the strongest open-source performance across diverse visual reasoning benchmarks while improving Mode Adaptiveness and Tool Effect. Ablations and training dynamics attribute these gains to necessity-aware mode selection and hint-guided capability expansion.

  • Overall Performance: Beacon-RL-8B ranks first on 11 of 13 benchmarks and exceeds Qwen3-VL-8B-Instruct by an average of 6.07 points.It achieves the highest average performance among all open-source models across perception, general visual understanding, and reasoning benchmarks.
  • Tool Effect: Beacon consistently outperforms its tool-free reasoning when tools are available, achieving the largest performance gain ∆Acc of +1.96%.Other models exhibit weaker tool-assisted ∆Acc values below +1%, while Beacon’s tool-free accuracy reaches 51.57% versus 49.75% for its base model.
  • Mode Adaptiveness: Beacon achieves the best average MAmean despite a relatively low MAtext, indicating adaptive selection of reasoning modes across problems.The model’s mode adaptiveness reflects a preference toward code use while still choosing appropriate reasoning modes overall.
  • Tool Effect: Beacon attains the largest tool-gain-versus-harm gap, with ∆TE of +3.14%, and the highest Text-Retain among evaluated models.This indicates effective tool use on problems difficult for text-only reasoning while limiting degradation from ineffective tool use.
  • Ablation Study: The full method achieves the best overall performance, while Necessity-Aware Adaptive Reward best improves Mode Adaptiveness and HCE improves ∆TE over GRPO.Their combination yields the best overall accuracy and the largest positive gap between tool-induced gains and harms.
  • Training Dynamics: During RL training, reasoning-mode alignment accuracy steadily improves without a clear shift toward pure-text or code-assisted responses.The dynamics suggest that Beacon learns increasingly accurate adaptive tool invocation rather than merely preferring one reasoning mode.

6 CONCLUSION … A.1 DETAILS OF THE METRIC ANALYSIS EXPERIMENTS

The paper frames agentic visual reasoning around using tools appropriately and effectively, introducing Mode Adaptiveness and Tool Effect as complementary dimensions. Beacon addresses these concerns through tailored training and demonstrates strong performance, improved adaptiveness, and greater relative tool-use gains across 13 benchmarks.

  • 6 CONCLUSION: Mode Adaptiveness and Tool Effect measure whether models select appropriate reasoning modes and whether tools extend capabilities beyond tool-free reasoning.The paper argues that tool use should be evaluated for appropriateness and effectiveness, not merely frequency.
  • 6 CONCLUSION: Beacon combines a high-quality data synthesis pipeline with a tailored reinforcement learning framework.Its training framework includes the Necessity-Aware Adaptive Reward and Hint-guided Capability Expansion mechanisms.
  • 6 CONCLUSION: The Necessity-Aware Adaptive Reward discourages unnecessary tool calls without suppressing useful tool-assisted reasoning.This mechanism is designed to encourage tool use according to task necessity.
  • 6 CONCLUSION: 13 diverse benchmarks show that Beacon achieves strong overall performance while improving reasoning-mode adaptiveness and the relative gain of tool use.These results are presented as substantial improvements over the evaluated agentic visual reasoning models.
  • A.1 DETAILS OF THE METRIC ANALYSIS EXPERIMENTS: The metric analysis enforces text-only answers by requiring direct reasoning from the provided image and question.The prompt begins with “You are a helpful assistant” and specifies step-by-step reasoning.
  • A.1 DETAILS OF THE METRIC ANALYSIS EXPERIMENTS: The text-only prompt prohibits external tools, code execution, Python, search, code blocks, and tool-call formats.It requires the output format: <think>...</think> <answer>...</answer>.

A.2 DETAILS OF THE EVALUATION EXPERIMENTS · B TRAINING DATA DETAILS · B.1 OVERVIEW

The evaluation uses VLMEvalKit with model-specific tool-use pipelines and bounded retries and tool-call rounds. Training-data statistics report raw samples alongside samples retained after filtering.

  • A.2 DETAILS OF THE EVALUATION EXPERIMENTS: Evaluation is built on the open-source VLMEvalKit toolkit.The framework follows the toolkit’s evaluation setup.
  • A.2 DETAILS OF THE EVALUATION EXPERIMENTS: Each model uses an inference pipeline implementing official prompts, tool-call formats, and execution procedures.This applies to all evaluated models, including Beacon.
  • A.2 DETAILS OF THE EVALUATION EXPERIMENTS: Agentic models may retry up to three times when code execution errors or missing final-answer tags prevent a valid answer.The passage specifically mentions missing or \boxed{} tags.
  • A.2 DETAILS OF THE EVALUATION EXPERIMENTS: Each model may perform at most 20 rounds of tool calls.This is the stated upper bound in the evaluation procedure.
  • A.2 DETAILS OF THE EVALUATION EXPERIMENTS: Answer evaluation first applies rule-based matching between the final prediction and ground-truth answer.Semantic judgment is used only if rule-based matching fails.
  • A.2 DETAILS OF THE EVALUATION EXPERIMENTS: When rule-based matching fails, Gemini 3.1 Pro or Gemini 3 Flash performs semantic answer judgment.The model receives both the prediction and ground-truth answer.
  • B TRAINING DATA DETAILS: Training-data statistics report both total raw samples and samples retained after data filtering.Table 5 is identified as the summary of these statistics.
  • B.1 OVERVIEW: The training-data overview summarizes the statistics presented in Table 5.The supplied passage provides no numerical values for the table.

B.2 DATA SOURCES

Beacon trains on eight publicly available multimodal datasets spanning diverse visual reasoning domains, task formats, complexity levels, and tool requirements. This mixture includes both tool-necessary and tool-optional scenarios.

  • Task coverage: The corpus covers mathematical reasoning, chart understanding, embodied interaction, spatial perception, game reasoning, and general visual reasoning.Its sources span broad task formats and visual reasoning requirements.
  • Training corpus: Beacon’s training data comprise Geometry3K, OlympiadBench, AgentVista, MuirBench, HRScene, CV-Bench, MMMU, and Vero.Together, these sources expose models to heterogeneous scenarios with different levels of tool dependency.
  • Specialized reasoning: Geometry3K and OlympiadBench provide visual mathematical reasoning, geometric diagram understanding, symbolic manipulation, and multi-step scientific problem solving.HRScene and CV-Bench add high-resolution image understanding, region-sensitive perception, object counting, spatial relations, and 2D/3D visual understanding.
  • Agentic and broad reasoning: AgentVista contributes tool-oriented, agentic data for long-horizon reasoning with intermediate visual operations, while MMMU, MuirBench, and Vero broaden multimodal scenarios.These latter datasets cover academic, multi-image, and general visual reasoning tasks.
  • Tool requirements: The resulting corpus includes samples solvable through direct multimodal reasoning and others benefiting from tools for visual manipulation, geometric analysis, or structured information extraction.This diversity provides both tool-necessary and tool-optional scenarios for learning tool use.

B.3 SFT DATA SYNTHESIS

SFT data are synthesized by generating correct code-based reasoning trajectories and refining them with Gemini 3.1 Pro. The process enforces tool-grounded, structured reasoning and removes trajectories whose code is invalid, redundant, uninformative, or unnecessary.

  • Two-step synthesis: SFT synthesis first generates code-based reasoning trajectories with Gemini 3.1 Pro and retains only trajectories whose final answers match ground-truth labels.The retained trajectories are then refined using the same model.
  • Prompt design: Five representative code actions—crop, draw line, draw box, numeric calculation, and rotation—guide common visual reasoning operations without constraining the generated code.Guidance specifies each action’s purpose, recommended library, imports, input convention, and example.
  • Prompt design: Trajectories must invoke valid code to obtain visual evidence or useful intermediate information, so examples without valid tool execution are discarded.This requirement prevents the model from directly answering questions without using code.
  • Trajectory format and sandboxing: Structured <tool_call>, <tool_response>, <observation>, and <answer> tags improve interpretability, automatic parsing, and explicit reasoning over tool outputs.Sandbox protocols also use stable image identifiers, executable final expressions, and img.size to ground coordinate-based operations in image resolution.
  • Trajectory refinement: Refinement removes trajectories that bypass tool outputs, redundant or uninformative code calls, and excess reasoning, producing more concise, informative, and structurally compliant trajectories.Manual inspection reports substantially higher quality after refinement.

C TRAINING DETAILS … D ADDITIONAL EXPERIMENTAL RESULTS

The paper details SFT and RL configurations, aligns prompts across training and evaluation, and defines hints as concise, answer-free plans grounded in useful reasoning and tool evidence.

  • C.1 DETAILS OF COLD-START SFT: SFT trains Qwen3-VL-8B-Instruct for 4 epochs with Adam, a 1×10−5 peak learning rate, 0.1 weight decay, cosine scheduling, and 5% warm-up.Gradients accumulate over 128 micro-batches before each parameter update.
  • C.2 DETAILS OF RL: RL runs for 1 epoch on 64 NVIDIA H200 GPUs, generating 8 rollouts per prompt with batch size 128 and PPO mini-batch size 128.The setup uses a 1 × 10−6 actor learning rate, token-mean PPO with symmetric clipping bounds of 0.2, and no KL or entropy regularization.
  • C.3 SYSTEM PROMPTS FOR SFT, RL, AND EVALUATION: The same system prompt is used for SFT, RL training, and evaluation, while mandatory code use is changed to optional decision-based code use.The prompt instructs the model to write code only when necessary and avoid it when the problem can be solved directly.
  • C.4 DETAILS FOR HINT GENERATION: Hint generation converts correct multimodal trajectories into concise reasoning plans containing instructions and expected subgoals without revealing final answers.The implementation retains only the hint field as the training hint.
  • C.4 DETAILS FOR HINT GENERATION: Hints retain only necessary reasoning, visual observations, and tool steps, omitting incidental trajectory content and unhelpful tool calls.The extraction process identifies steps truly needed to reach the answer and prefers actionable tool steps over vague advice.
  • C.4 DETAILS FOR HINT GENERATION: Each hint item follows a numbered instruction-and-subgoal format, while tool labels use canonical semantic operations such as crop, draw_line, draw_box, numeric_calculation, and rotation.The output schema also includes high-level actions, expected subgoals, factual subgoal results, and ordered subgoals.
  • C.4 DETAILS FOR HINT GENERATION: Subgoal results must report factual, answer-free evidence actually obtained after tool execution, including exact numeric, coordinate, count, OCR, or table-cell details when available.The subgoals field reproduces the ordered subgoal_result strings from the tool steps.

D.1 DETAILED RESULTS OF ABLATION STUDY … F.1 REASONING TRAJECTORY EXAMPLES

The paper reports detailed ablations of Mode Adaptiveness and Tool Effect, analyzes why an aggressive adaptive-reward variant fails, and illustrates Beacon’s use of diverse visual tools on complex reasoning tasks.

  • D.1 DETAILED RESULTS OF ABLATION STUDY: Table 6 reports detailed ablation results for Mode Adaptiveness and Tool Effect, using the metric definitions and notation from Table 3.The reported results are percentages.
  • D.1 DETAILED RESULTS OF ABLATION STUDY: The ablation study’s detailed results are provided in Table 6.
  • E DISCUSSION: The aggressive adaptive reward forces pure-text reasoning trajectories alongside standard rollouts and labels a problem “text” if either group contains a correct pure-text trajectory.
  • E.1 FAILED ATTEMPTS: This aggressive strategy biases the policy toward pure-text reasoning, collapsing the code-use ratio and substantially degrading performance.
  • E.1 FAILED ATTEMPTS: The failure arises because forced-text rollouts inflate the chance of observing a successful pure-text trajectory, even when pure-text reasoning is not most reliable on average.
  • E.1 FAILED ATTEMPTS: The resulting group-level label reflects the best outcome across attempts rather than the reasoning mode most likely to succeed in a single rollout, creating a training–evaluation mismatch.
  • F CASE STUDY: Tables 7, 8, and 9 illustrate that Beacon effectively leverages a diverse set of visual tools to solve complex reasoning tasks.
  • F.1 REASONING TRAJECTORY EXAMPLES: The reasoning trajectory examples demonstrate Beacon’s tool-assisted approach on complex visual reasoning tasks.

F.2 HINT EXAMPLES

Training-time hint supervision enables Beacon to extend its reasoning capabilities and tackle more challenging visual tasks. The examples show hints decomposing problems into targeted visual operations and subgoals.

  • Overall finding: Training-time hint supervision enables Beacon to extend its reasoning capabilities and tackle more challenging tasks.This conclusion is demonstrated by Tables 10, 11, and 12.
  • Hint Example: Case 1: For a badminton-score problem, hints direct Beacon to crop relevant regions, determine the score and boundary status, then calculate the updated score and event year.Beacon answers IND 1 20, TPE 0 16 and identifies 2023 as the event year.
  • Hint Example: Case 2: For hidden-text identification, the hint instructs Beacon to resize the image to 1/10th or 1/20th scale to reveal text or patterns.The expected subgoal is to reveal hidden content that becomes visible at lower resolution.
  • Hint Example: Case 3: For a reflection puzzle, hints explain that two reflections equal a 180-degree rotation, direct rotating the number 5 with a tool, and require comparison with options A through E.The hint sequence moves from understanding the transformation rule to obtaining and comparing the visual result.
Loading 2607.28595v1…