PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Explorative Policy Optimization for Multimodal Agentic Reasoning

Minki Kang, Shizhe Diao, Ryo Hachiuma, Sung Ju Hwang, Pavlo Molchanov, Yu-Chiang Frank Wang, Byung-Kwan Lee

arXiv:2605.28774v1cs.CL

TL;DR

External tools are essential for many multimodal problems, yet standard RL undertrains tool use because tool-using rollouts are sparse and often jointly fail. AXPO fixes the thinking prefix and resamples uncertain tool calls, outperforming SFT+GRPO across nine benchmarks, including a +1.8pp Pass@4 gain at 8B and surpassing the 32B Base on Pass@4 with 4× fewer parameters.

  • Problem

    Standard group-based RL undertrains tool use because tool calls are infrequent and tool-using rollouts disproportionately fail together, weakening their learning signal.

  • Method

    AXPO fixes the thinking prefix at uncertain tool calls and resamples the tool call and its continuation during RL.

  • Results

    +1.8pp Pass@4 at 8B on average across nine multimodal benchmarks, with SFT+AXPO surpassing the 32B Base on Pass@4 using 4× fewer parameters.

  • Takeaways & Limitations

    AXPO consistently outperforms SFT+GRPO across three model scales and narrows the performance gap between smaller agentic models and larger baselines.

  • Takeaways & Limitations

    The study assumes verifiable outcome rewards for RL and trains models only up to 8B parameters.

Abstract

from arXiv · show

Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.

1. Introduction

The introduction identifies a Thinking-Acting Gap: internal thinking is the model’s default, while tool use is fragile and underexplored in RL. AXPO addresses this gap by fixing failed thinking prefixes and resampling uncertain tool calls, improving multimodal reasoning performance over SFT + GRPO.

  • Motivation: The Thinking-Acting Gap separates self-contained thinking from fragile tool use, whose external responses the model cannot predict or control.Thinking tokens remain under the model’s control, whereas short tool-call sequences trigger external responses and admit diverse implementations.
  • Method: AXPO fixes the thinking prefix of a failed tool-using trajectory and resamples the tool call and continuation to explore alternative actions.The method concentrates exploration at the boundary where a committed intent becomes a high-variance external action, rather than re-deriving the full trajectory.
  • Method: AXPO resamples only all-wrong tool-using subgroups, selects highest-uncertainty prefixes, and uses continuation advantages plus a binary recovery indicator.These design choices operate with a small extra rollout budget and restructure the rollout distribution at the Thinking-Acting Gap.
  • Results: +2.8/+2.3/+1.8 pp Pass@4 gains at Qwen3-VL-Thinking 2B/4B/8B show SFT + AXPO outperforming SFT + GRPO across nine multimodal benchmarks.At 8B, SFT + AXPO surpasses the 32B Base on Pass@4 and reaches 99% of its Pass@1 with 4× fewer parameters.
  • Motivation: 30% of rollouts attempt tool use, while tool-using subgroups are all-wrong on 40% of questions, weakening GRPO’s tool-call learning signal.These two symptoms leave tool use underrepresented and often without a positive advantage at the tool-call tokens.

2. Analysis of RL in Agentic Reasoning

The analysis identifies a Thinking-Acting Gap in GRPO: optional tool use remains infrequent, attempted tool-using rollouts often fail together, and tool-call resampling at a fixed thinking prefix yields diverse alternatives. These diagnostics indicate that GRPO under-trains tool calls.

  • RL training with GRPO: GRPO assigns each rollout a group-normalized advantage from binary answer-correctness rewards and applies it uniformly across the trajectory.The pipeline starts with supervised fine-tuning on expert trajectories, followed by reinforcement learning with verifiable outcome rewards.
  • Tool calls admit substantive resampling at a fixed thinking: 2.9–3.4 distinct semantic tool-call classes emerge from 16 continuations sampled after fixing the first thinking prefix.The diversity shows that the tool-call boundary is a substantive divergence point under the policy.
  • Tool use is under-attempted: ∼20–35% of rollouts use tools across RL steps, leaving thinking-only rollouts as the majority.Tool use is optional: the model can emit a tool call or proceed directly to an answer.
  • When attempted, the tool-using subgroup is prone to being all-wrong: ∼40% of questions with attempted tool use have an all-wrong tool-using subgroup, versus 25% for the no-tool subgroup.Tool-call tokens receive negative advantage in mixed groups and zero advantage when the entire group is wrong.

3. Agent eXplorative Policy Optimization (AXPO)

AXPO targets the under-trained tool-use behavior by fixing a thinking prefix that initiates a tool call and resampling its continuations, concentrating stochasticity where useful learning signal is otherwise sparse. It triggers this process on entirely wrong tool-using subgroups, prioritizes low-confidence prefixes, and separates prefix from continuation updates to avoid gradient conflict.

  • Tool-call resampling: AXPO fixes a thinking prefix through the opening <tool_call> tag, then samples K tool-using continuations and rolls each forward to an answer or turn limit.All resampled trajectories share the source prefix and diverge at the tool call, receiving the same binary outcome reward.
  • Tool-call resampling: Every resampled continuation is tool-using by construction, so the full resampling budget targets tool-call learning signal instead of non-tool rollouts.The method addresses the tool-use rate q that bounds coverage of correct tool-using rollouts under raw sampling.
  • Trigger and budget allocation: AXPO resamples groups whose non-empty tool-using subgroup is entirely wrong, where tool-call tokens receive zero or negative advantage under GRPO.A recovered continuation therefore provides the largest gradient lift per resampled continuation.
  • Uncertainty-based prefix ranking: ∼30% of prefixes collapse to one cluster across 16 resamples, so AXPO ranks candidates by mean policy probability on tool-call tokens and samples lowest-confidence prefixes first.The mean policy probability serves as a tractable uncertainty proxy.
  • Advantage decomposition: AXPO masks shared prefix tokens in the resampled advantage group and updates them through the source trajectory with a binary recovery reward when any continuation succeeds.This decomposition assigns each token one source, avoiding contradictory advantages on the shared prefix while preserving standard GRPO for non-triggered groups.

4. Experiments

Experiments across nine multimodal benchmarks show that SFT + AXPO consistently improves over SFT + GRPO and alternative RL recipes, while reversing the training symptoms of the Thinking-Acting Gap. Its gains arise from more effective tool interaction, with 8B SFT + AXPO surpassing the 32B Base on Pass@4 despite using four times fewer parameters.

  • Experimental setup: The evaluation spans nine multimodal benchmarks across Reasoning, Perception, and Search, reporting Pass@1 and Pass@4 averaged over four rollouts per question.All benchmarks use the agentic setting and the unified tool interface used during training.
  • Main results: +1.1 / +1.4 / +1.8 pp are SFT + AXPO’s Pass@1 average gains over SFT + GRPO at 2B/4B/8B, while Pass@4 gains are +2.8 / +2.3 / +1.8 pp.At 8B, SFT + AXPO reaches 99% of the 32B Base’s Pass@1 and achieves Pass@4 of 75.8 versus 75.1, despite 4× fewer parameters.
  • Main results: +0.6 pp, +2.9 pp, and +1.3 pp are SFT + AXPO’s 8B Pass@1 gains over SFT + GRPO on Reasoning, Perception, and Search, respectively.The relative gain is largest in Perception (+4%) and smallest in Reasoning (+1%).
  • Alternative recipes: SFT + AXPO exceeds reward shaping, doubled rollout budget, RLTF, CISPO, and ARPO on both Pass@1 and Pass@4, showing gains from compute allocation rather than quantity alone.The 2× rollout control underperforms AXPO, and reward-shaping methods fall below the GRPO baseline of 51.9.
  • Training dynamics: Under SFT + GRPO, tool-use rate remains near ∼30% and the all-wrong tool-using rate near ∼40%, whereas AXPO raises tool use by +28 pp and lowers all-wrong rate by ∼17 pp.Resampling targets all-wrong tool-using subgroups, restoring at least one correct tool-call continuation and the associated learning signal.
  • Test-time dynamics: Only SFT + AXPO advances simultaneously in tool-attempt rate and conditional Pass@1, and matched-tool analysis shows higher Pass@1 across all three tool families.This indicates that AXPO improves tool interaction quality, not merely tool-use frequency.

5. Related Works

Related work studies agentic VLM reasoning, group-relative RL adaptations for tool use, and three strategies for handling tool-using rollouts. AXPO differs by restoring signal in all-wrong tool-using subgroups and exploring the tool call itself rather than only its continuation.

  • Agentic reasoning for VLMs: Agentic reasoning extends VLM reasoning beyond token space through tool calls, external outputs, and conditioned subsequent reasoning.Tools support knowledge search, computation, and visual perception; recent systems combine image manipulation with code and search under an SFT-then-RL recipe.
  • RL for agentic reasoning: Related RL methods include direct outcome-reward ports, mid-trajectory continuation branching, and tool-use reward shaping.Group-relative methods such as GRPO normalize rewards across N rollouts and assign the resulting advantage uniformly.
  • Our contribution: AXPO restores learning signal on all-wrong tool-using subgroups that uniform sampling cannot reach.This distinguishes AXPO from direct use of GRPO.
  • Our contribution: AXPO resamples at the tool-call boundary, exploring the tool call itself rather than the continuation after a tool observation.This contrasts with mid-trajectory branching methods that resample the continuation after a tool call.
  • Our contribution: AXPO acts on the rollout distribution while leaving the reward intact, contrasting with reward-shaping methods.The supplied passage states this contrast but ends mid-word after “intac.”

6. Conclusion … A.2. System Prompt and Tool Interface

The paper identifies a Thinking-Acting Gap in agentic RL and addresses it with AXPO, which resamples tool calls after fixing thinking prefixes and selects prefixes by uncertainty. The appendices detail the experimental setup, training data and objectives, and a shared three-tool agent interface used across training and evaluation.

  • 6. Conclusion: AXPO targets the Thinking-Acting Gap, where thinking is the default behavior but tool use is a high-variance auxiliary behavior.The gap appears as infrequent tool-use attempts and disproportionately all-wrong tool-using subgroups, producing non-positive tool-call advantages when tools would help.
  • 6. Conclusion: AXPO fixes the thinking prefix, resamples the tool call and continuation, and uses uncertainty-based prefix selection to concentrate exploration on tool use.The study assumes verifiable outcome rewards and trains models up to 8B parameters.
  • A. Experimental Setup: The experimental appendix covers training data, hyperparameters, the agentic system prompt, tool schemas, and per-benchmark descriptions.These materials expand on Section 4.1 of the paper.
  • A.1. Training: 64,274 SFT trajectories come from ViRL, fvqa, and PyVision-RL, with approximately 25% containing tool calls and all retained trajectories having correct final answers.All trajectories are generated by Qwen3-VL-32B-Thinking acting as the teacher model.
  • A.1. Training: 15,591 RL problems are filtered to remove trivial and unreachable cases, with 22,000 hard MMFineReason-hard questions additionally included.The filtering excludes problems solved by the SFT checkpoint on all four rollouts or failed by the 32B teacher on all four rollouts.
  • A.1. Training: Both methods maximize the PPO-clip surrogate with KL regularization, while AXPO applies per-prefix advantages to continuations and recovery-indicator advantages to source prefixes.The shared clipping settings are 𝜖low = 0.2, 𝜖high = 0.4 and KL coefficient 𝛽= 1 × 10−3; AXPO additionally uses branching budget 𝑟= 0.25 and 𝐾= 4 continuations per selected prefix.
  • A.2. System Prompt and Tool Interface: The shared system prompt requires agents to continue until the query is resolved and permits Python, tavily-search, and image_zoom_in through a unified <tool_call> interface.The prompt requires Python for image analysis when useful and web search when information is insufficient; the same prompt is used for SFT, RL, and evaluation across model sizes.
  • A.2. System Prompt and Tool Interface: The tavily-search tool excludes huggingface.co uniformly during training and evaluation to prevent benchmark-answer leakage.The tool schemas use OpenAI function-calling format, while Python runs in a persistent Jupyter kernel and image_zoom_in accepts a four-number bounding box.

A.3. Benchmark Details · B. Additional Experimental Results

The paper evaluates agentic multimodal reasoning across nine benchmarks organized by their expected tool family, while allowing models to use any subset of three tools or none. Supplementary experiments further analyze utilization, benchmark-level performance, variance, training stages, generalization, ablations, prior systems, and confidence-based prefix ranking.

  • A.3. Benchmark Details: Nine multimodal benchmarks are grouped into Reasoning, Perception, and Search according to whether Python, image zoom-in, or web search is most expected.All benchmarks use the same agentic tool interface as training, and each question permits any subset of the three tools or no tools.
  • A.3. Benchmark Details: Reasoning benchmarks test multi-step text, math, and figure problems for which Python supports symbolic and numerical computation.MathVision uses 304 diagram-paired competition problems, while DynaMath evaluates a single-variant slice of 501 seed problems.
  • A.3. Benchmark Details: Perception benchmarks target high-resolution visual details that are difficult to read at default encoder resolution, making image zoom-in the recovery mechanism.They include fine-grained visual search, targeted small-region probing, and HR-Bench-4K and HR-Bench-8K resolutions.
  • A.3. Benchmark Details: Search benchmarks require external or multi-hop knowledge, using the Tavily web search API for lookup and composition.HR-MMSearch combines image-based entity identification with chained web lookups, while MMSearch depends on post-cutoff web content.
  • A.3. Benchmark Details: Four rollouts per question yield Pass@1 as averaged per-question accuracy and Pass@4 as correctness when any rollout succeeds.Math and multiple-choice answers use exact match, search answers use F1, and difficult exact-match cases may use a gpt-5-mini LLM judge.
  • A.3. Benchmark Details: Evaluation uses temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5, four rollouts, five agentic turns, and a 32,768-token response limit.These settings define the sampling and trajectory constraints for benchmark evaluation.
  • B. Additional Experimental Results: Supplementary experiments report tool utilization, per-benchmark Pass@4, per-rollout Pass@1 variance, training-stage decomposition, unseen-tool generalization, ablations, prior-system comparisons, and confidence-ranking justification.The analyses span Appendices B.1 through B.9, including per-row examinations of ablation and comparison tables.

B.1. Tool Utilization Rates at Evaluation … B.5. Generalization to an Unseen Tool: Image Search

Across evaluation, AXPO increases tool use and improves Pass@4 beyond GRPO, while variance remains below the reported gains. The decomposition supports SFT before RL and shows AXPO transfers more effectively to an unseen image-search tool, especially for Pass@1.

  • B.1. Tool Utilization Rates at Evaluation: 81.9 / 86.9 / 82.9% average tool utilization makes SFT + AXPO the highest-utilizing trained method at 2B / 4B / 8B.At 8B, AXPO’s utilization advantage over SFT + GRPO reaches +30 / +33 / +22 pp on MathVision, DynaMath, and Math-VR, while both methods approach 100% on tool-saturated benchmarks.
  • B.2. Per-Benchmark Pass@4 Results: Table 5 reports Pass@4 across nine multimodal benchmarks, four Qwen3-VL-Thinking sizes, and Reasoning, Perception, and Search categories, with AXPO gains shown against SFT + GRPO.The shared Average column underlies the headline Pass@4 comparisons and the per-stage decomposition.
  • B.4. Decomposing the SFT-then-RL Recipe: −1.3 / +1.1, +5.2 / +9.2, +4.7 / +9.8, and +6.7 / +10.1 are the Non-agent-to-Base Pass@1 / Pass@4 deltas at 2B, 4B, 8B, and 32B.At 2B, enabling tools without training reduces Pass@1, and even the 32B Base reaches only 75.1 Pass@4 versus 75.8 for 8B SFT + AXPO.
  • B.4. Decomposing the SFT-then-RL Recipe: +6.8, +4.9, and +2.4 Pass@4 points are the SFT + GRPO gains over GRPO at 2B, 4B, and 8B, showing that SFT initialization preserves diversity for RL.Pass@1 gaps shrink with scale, but Pass@4 gaps remain positive at every scale.
  • B.4. Decomposing the SFT-then-RL Recipe: +1.1 / +1.4 / +1.8 Pass@1 and +2.8 / +2.3 / +1.8 Pass@4 are AXPO’s gains over SFT + GRPO at 2B / 4B / 8B.AXPO recovers all-wrong tool-using subgroups, turning otherwise vanishing gradient signals into trainable continuations and expanding the reachable correct-trajectory set.
  • B.5. Generalization to an Unseen Tool: Image Search: +4.6 pp versus +2.8 pp are the MMSearch Pass@1 gains from adding unseen image search for AXPO versus GRPO.Without image search, the methods are nearly tied; with it, Pass@4 gains are similar (+6.0 and +5.7), and the evidence is bounded to one benchmark and one new tool type.

B.6. Detailed Ablation Analysis … B.9. Confidence as a Tractable Proxy for Tool-Call Entropy

Across ablations and comparisons, AXPO’s gains depend on preserving tool-using prefixes, targeting under-covered failures, and selecting and crediting prefixes effectively. Confidence provides a practical proxy for tool-call entropy because it is strongly correlated with exact entropy while avoiding additional rollout cost.

  • B.6. Detailed Ablation Analysis: 52.1 without prefix fixing nearly matches the 51.9 SFT + GRPO baseline because fresh resamples reintroduce the (1−q) waste factor.Only a q-fraction of fresh resamples are tool-using when the thinking prefix is not fixed.
  • B.6. Detailed Ablation Analysis: 51.7 results from targeting all-no-tool failures, whereas resampling helps only when aimed at under-covered tool-using subgroups.The policy already covers the no-tool subgroup under GRPO, so additional samples there cannot recover missing tool-using signal.
  • B.6. Detailed Ablation Analysis: 51.0 with random prefix ranking, 51.4 without prefix credit, and 51.2 with merged advantage groups show that selection, credit, and per-prefix grouping are all necessary.These variants respectively waste resamples on determined commitments, remove positive recovery signals, or dilute successful resampling with contradictory source-group advantages.
  • B.7. Detailed Comparison Analysis: 46.0 with a tool penalty and 50.8 with a tool bonus both trail the 51.9 GRPO baseline, so reward shaping does not recover AXPO’s gain.Penalty suppresses tool use globally, while bonus rewards tool calls regardless of whether they restore a correct continuation.
  • B.7. Detailed Comparison Analysis: 52.4 from doubling GRPO’s rollout count remains 1.5 pp below SFT + AXPO despite ×4 exceeding AXPO’s extra compute.More rollouts re-derive non-tool prefixes but do not break all-wrong tool-using subgroups.
  • B.7. Detailed Comparison Analysis: 43.3 Avg for RLTF at 40% tool utilization and 47.8 Avg for CISPO show that alternative RL algorithms do not resolve the tool-call coverage problem.RLTF suffers tool-call collapse, while CISPO’s clipping reduces gradient bias without addressing all-wrong tool-using subgroups.
  • B.8. Comparison with Prior Agentic VLMs: AXPO’s largest prior-system gaps occur on MathVision and DynaMath, while pixel-level reasoning systems remain within ∼1 pp on zoom-in-heavy benchmarks.The comparison is not strictly controlled because prior systems use different base models, training-data scales, and benchmark subsets.
  • B.9. Confidence as a Tractable Proxy for Tool-Call Entropy: Pearson ρ=0.843 and Spearman ρ=0.835 show that mean tool-token confidence strongly preserves exact-entropy rankings.Confidence is emitted essentially for free by the rollout engine, whereas predictive entropy requires additional computation in the training loop.

C. Extended Related Works

Recent agentic VLMs typically use SFT on teacher-distilled tool trajectories followed by outcome-reward RL, while prior RL analyses and branching methods address only parts of tool-using exploration. AXPO’s uncertainty proxy is supported by strong confidence–policy-entropy correlations, and combining tool-call with post-observation branching remains future work.

  • Uncertainty proxy: 0.843 Pearson ρ and 0.835 Spearman ρ measure strong correlations between rollout confidence and exact policy entropy at tool-call positions.The figure uses mean confidence from free vLLM rollouts and exact full-vocabulary entropy from FSDP forward passes on failed tool-using rollouts.
  • Two-stage SFT-then-RL recipe: Agentic VLMs commonly combine SFT on teacher-distilled tool trajectories with outcome-reward RL, differing in distillation data, tool interfaces, and base models.Most prior agentic-RL VLM studies focus on one tool family, whereas the described setup combines code, search, and image zoom-in.
  • Group-relative RL and BroRL: GRPO samples N rollouts per question, normalizes rewards within each group, and assigns the resulting advantage uniformly to every token; BroRL links larger N to fewer unsampled blind spots.These analyses were developed for thinking-only trajectories with uniform rollout structure.
  • Mid-trajectory branching for tool-using RL: ARPO and AEPO branch after tool observations when post-observation entropy is high, exploring continuations with a fixed tool call but not recovering wrong tool calls.Their continuation sampling absorbs variance from tool outputs, but cannot correct errors such as selecting the wrong sub-image.
  • Future extensions: Combining tool-call branching with post-observation branching is described as a natural extension left for future work.The two strategies are presented as compatible.

D. Coverage Argument for Tool-call resampling

The coverage argument shows that prefix-fixed resampling improves recovery of correct tool-using rollouts when the selected prefix exceeds the threshold q p_tool. Because resampling makes every continuation tool-using, it removes raw sampling’s non-tool waste and restores positive learning signal in sparse tool-use regimes.

  • Proposition 1: Prefix-fixed resampling dominates raw sampling when p(t_1) ≥ q p_tool, strictly doing so when p(t_1) > q p_tool.The threshold is lower than the average per-tool-using success rate because resampling drives q toward 1 by construction.
  • Coverage comparison: 1 − (1 − p_eff)^N gives coverage of at least one correct tool-using rollout, with p_eff = q p_tool for raw sampling and p_eff = p(t_src_1) for resampling.Resampled continuations are tool-using by construction, whereas raw samples include the policy’s non-tool rollouts.
  • Consequence for GRPO training: Under GRPO with binary rewards, tool-call tokens receive no positive signal when the tool-using subgroup contains no correct rollout.This occurs when the subgroup is empty or all-wrong; fully all-wrong groups have zero group-normalized advantage, while mixed-success groups can produce negative advantage.
  • Threshold satisfaction: q < 1 guarantees that the expected tool-committed prefix has p(t_1) = p_tool > q p_tool, so all prefixes at or above the conditional mean satisfy the proposition.Only strongly subpar prefixes with p(t_1) < q p_tool can underperform raw sampling.
  • Relation to BroRL: q ∈ [0.2, 0.35] characterizes the rare-tool-use regime where naive rollout scaling leaves tool-call tokens difficult to cover.AXPO instead targets this low-probability tool-use region, whereas BroRL’s guarantee is uniform across token types.

E. Limitations and Future Work … G.2. Reasoning: geometry disambiguation via python visualization

The paper identifies verifiability, training scale, tool-environment breadth, and base-model coverage as limitations and future-work directions, while illustrating AXPO’s advantages through qualitative perception and geometry examples. It also notes sandboxing for Python-based tool use and shows how tool calls can correct reasoning failures that persist under GRPO.

  • E. Limitations and Future Work: AXPO’s trigger and per-prefix advantage require binary, automatically verifiable outcomes, limiting direct application to partially verifiable or subjective tasks.Open-ended generation and multi-criterion evaluation require a different definition of a failed subgroup, although resampling itself is reward-agnostic.
  • E. Limitations and Future Work: Training covers models up to 8B parameters, leaving AXPO’s generalization to 32B+ trainable scales unresolved.The 32B model is reported only as an inference-time SFT teacher, and the boundary reflects available compute rather than methodology.
  • E. Limitations and Future Work: Future work should extend AXPO to longer-horizon environments and tools with longer latency or higher per-call cost.The current inventory spans Python, web search, and image zoom-in, while browser agents, GUI control, terminal sessions, and LLM-as-tool settings remain unaddressed.
  • E. Limitations and Future Work: All experiments use Qwen3-VL-Thinking, so replication on newer Qwen3.5 models is identified as future work.The authors characterize the Thinking-Acting Gap as a property of reinforcement learning on agentic reasoning models rather than a specific architecture and expect transfer.
  • F. Broader Impacts: Python tool use introduces standard code-execution security concerns, which the authors address by sandboxing tool calls during training and evaluation.AXPO is presented as an additive RL improvement on Qwen3-VL-Thinking rather than a new capability class.
  • G. Qualitative Examples: GRPO vs. AXPO: At 8B, three side-by-side trajectories show SFT + AXPO succeeding 4/4 while SFT + GRPO fails 0/4 across perception, reasoning, and search.The examples provide the question, correct answer, and one representative trajectory per method, with abbreviated thoughts and verbatim tool interactions.
  • G.1. Perception: tiny-text reading on a Hong Kong street scene: All four GRPO rollouts guess 999 without invoking image_zoom_in, whereas AXPO uses the identified candidate region to answer 2563.Both methods’ thinking prefixes identify the relevant region, but only AXPO commits to the tool call.
  • G.2. Reasoning: geometry disambiguation via python visualization: AXPO’s Python visualization resolves a geometric misinterpretation: matching bottom quarter-disks and the top half-disk yield the full rectangle’s area, 8.Both rollouts initially derive 4, but that value corresponds to the wrong sub-region; the error is geometric rather than arithmetic.

G.3. Search: multi-hop decomposition into two sequential queries

The example shows that solving HR-MMSearch’s multi-hop question requires a second search after identifying the designer’s birthplace. AXPO issues this follow-up, while all four GRPO rollouts stop after the first hop and guess incorrectly.

  • G.3. Search: multi-hop decomposition into two sequential queries: 1984 is the ground-truth answer: Ray McAdam, Lord Mayor of Dublin since June 2025, was born in 1984.The successful reasoning connects the designer’s birthplace, Dublin, to the serving mayor and then retrieves his birth year.
  • G.3. Search: multi-hop decomposition into two sequential queries: AXPO consistently issues the second search needed to complete the designer → city → mayor → birth-year chain.The decisive action is whether to search again after the first hop returns only the designer’s information.
  • G.3. Search: multi-hop decomposition into two sequential queries: All four GRPO rollouts stop after the first hop and fall back to incorrect riddle-style guesses, including 1945, 1963, and 1968.One rollout explicitly commits to 1945 without conducting a second-hop search.
  • G.3. Search: multi-hop decomposition into two sequential queries: A single-query collapse of the two-hop structure works only by accident, making the follow-up search the load-bearing decision.The structure requires identifying the designer’s city, then using the city and date to identify the mayor and birth year.
Loading 2605.28774v1…