PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization

Xinhai Hou, Shaoyuan Xu, Manan Biyani, Moyan Li, Jia Liu, Todd C. Hollon, Bryan Wang

arXiv:2511.19661v2cs.CV

TL;DR

Agentic visual models can answer correctly while using tools on irrelevant regions or ignoring their outputs, exposing a gap between accuracy and faithful reasoning. The paper evaluates evidence alignment and introduces CodeV with TAPO, a process-level training method that rewards useful tool outputs alongside correctness. CodeV improves faithful tool use while maintaining strong benchmark performance, though the current judge remains costly and specialized for image-centric operations.

  • Problem

    Visual agents can achieve high final-answer accuracy while using image tools unfaithfully, so accuracy alone does not establish evidence-grounded visual reasoning.

  • Method

    CodeV uses executable visual-tool code and a two-stage SFT+RL pipeline, with TAPO assigning dense rewards from the question and tool output for faithfulness and answer correctness.

  • Results

    CodeV achieves strong performance across visual search and reasoning benchmarks while substantially increasing faithful tool use; it reaches 46.6 on VLMBlinds and 84.8 on V*.

  • Takeaways & Limitations

    Explicitly supervising intermediate tool behavior is important for developing trustworthy agentic multimodal systems beyond outcome-only RLVR.

  • Takeaways & Limitations

    TAPO relies on a static judge model with added deployment cost, and its current faithfulness metric is mainly tailored to image-centric operations such as cropping.

Abstract

from arXiv · show

Agentic vision-language models are increasingly trained to "think with images" by calling image operations. However, we show that high final-answer accuracy often hides unfaithful visual reasoning: models may invoke tools on irrelevant regions or ignore tool outputs entirely, yet still guess the correct answer. In this work, we first propose a faithfulness evaluation protocol that measures whether intermediate visual tool outputs (e.g., crops) actually contain the queried evidence. This reveals that recent visual agents achieve high final-answer accuracy but exhibit low rates of faithful tool-use on visual search benchmarks. We then introduce CodeV, a code-based visual agent trained with Tool-Aware Policy Optimization (TAPO). TAPO is a process-level RL framework that augments GRPO with dense rewards defined directly on visual tool inputs and outputs, rather than on chain-of-thought tokens, making supervision easier to verify and less susceptible to reward hacking. CodeV represents visual tools as executable Python code, and TAPO assigns step-wise rewards based solely on the question and tool output, encouraging both necessary and evidence-consistent tool use. In a two-stage SFT+RL pipeline, CodeV achieves competitive or superior accuracy while substantially increasing faithful tool-use rates on related visual search benchmarks. Beyond visual search, CodeV attains strong performance on a range of multimodal reasoning and math benchmarks, suggesting that explicitly supervising intermediate tool behavior is crucial for building trustworthy, agentic visual reasoning systems.

1. Introduction

Existing visual agents can achieve correct answers while using image tools unfaithfully, motivating direct evaluation and process-level supervision of tool behavior. CodeV addresses this gap by training open-source VLMs to ground visual reasoning in relevant tool outputs.

  • Motivation: High final-answer accuracy can coexist with unfaithful visual tool use, including crops that miss the queried object while the model still answers correctly.This disconnect makes accuracy alone insufficient for evaluating agentic visual reasoning.
  • Contribution: TAPO augments GRPO with dense, step-level rewards for tool necessity and output consistency rather than supervising unverifiable chain-of-thought tokens.The approach uses question context and tool outputs to provide process-aware signals while retaining GRPO scalability.
  • Contribution: CodeV is trained through a two-stage SFT-plus-RL curriculum and achieves strong performance across 10 challenging benchmarks while improving tool faithfulness.The model is built on Qwen2.5-VL-7B and is intended to narrow the gap to proprietary systems.
  • Faithfulness Evaluation: Faithfulness evaluation checks whether intermediate outputs remain relevant to the question, avoiding direct inspection of model-generated chain-of-thought.The protocol treats observable tool consequences, such as crops, as easier-to-verify evidence of grounded reasoning.
  • Evidence of Unfaithful Tool Use: Only 57% of DeepEyes and 43% of Pixel-Reasoner correct answers are accompanied by faithful intermediate tool calls.Faithful use is assessed by whether cropped outputs capture a target object, conditioned on correct answers.

4. Methodology

CodeV combines executable image-operation code with a two-stage SFT and TAPO reinforcement-learning pipeline. TAPO jointly rewards answer correctness and evidence-bearing tool outputs while discouraging unnecessary, invalid, or uninformative operations.

  • Training Pipeline: The training pipeline begins with SFT on image-operation and refinement traces before on-policy TAPO reinforcement learning guides faithful tool steps.SFT supplies a cold-start tool-use prior, while TAPO uses group-based rollouts and relative advantages to update the policy.
  • CodeV Pipeline: CodeV represents visual tool use as executable Python code whose sandbox outputs are appended to the model context alongside reasoning and final-answer actions.Completed code blocks operate on the input image using deterministic image or math utilities, while think and answer actions produce no observation.
  • Reward Design: TAPO combines final-answer correctness with tool faithfulness, rewarding relevant outputs and penalizing lazy crops or incorrect operations.The tool reward evaluates whether an output provides useful evidence for the question, while answer reward measures final correctness.
  • Tool-Aware Policy Optimization: A judge evaluates only the question, sandbox output, and metadata, rewarding crops that contain at least one relevant object or region without requiring all targets.This avoids judging hidden reasoning or code and reduces dependence on dense bounding-box annotations.
  • Reward Design: For tasks without localized visual search, TAPO keeps tool rewards near zero by default and penalizes only clear misuse such as invalid coordinates or repeated no-op crops.This design prevents dense rewards from encouraging unnecessary tool calls.

5. Experiments

CodeV is evaluated against open-source and proprietary baselines across perception, visual search, reasoning, and math benchmarks, while additional analyses measure faithful tool use and training-design effects. It achieves strong benchmark performance and the highest reported faithfulness, while ablations support dense step-level reward supervision.

  • 5.2. Performance of CodeV: CodeV achieves the strongest reported overall benchmark performance, including 46.7 on VLMBlinds and 84.8 on V*, while narrowing the gap with GPT-4o.It improves over Qwen2.5-VL-7B, Pixel-Reasoner-7B, and Thyme-7B-RL on VLMBlinds, and surpasses GPT-4o on V*.
  • 5.3. Evaluating Reasoning Faithfulness: CodeV consistently attains the highest faithful tool-use rates on V* and HRBench-4k, whereas Thyme shows single-digit faithfulness because it rarely executes visual tools.The evaluation reports faithful, unfaithful, and no-tool-use rates, linking higher faithfulness to better alignment with question intent.
  • 5.4. Ablation Studies: Zero-RL improves average performance by roughly 3–4 points but quickly collapses to text-only reasoning, motivating the two-stage SFT-plus-RL curriculum.The training-stage ablation starts from Qwen2.5-VL-7B and compares removing the cold-start SFT stage.
  • 5.4. Ablation Studies: Accuracy-only rewards yield small gains but drive text-only reasoning, while GPT-5-nano judging and the full TAPO reward add approximately 1–2 points across reasoning and perception.The full TAPO reward produces the strongest overall performance, supporting step-level supervision over purely outcome-based rewards.

6. Related Work

Related work develops agentic visual reasoning through executable image operations, visual chains of thought, and code-based tool interfaces. CodeV builds on these approaches while focusing on evaluating and training faithful visual tool outputs and addressing limitations of outcome-only reward supervision.

  • Agentic visual reasoning: Prior systems express visual reasoning as executable programs, unified Python APIs, or interleaved visual actions and intermediate sketches.Examples include VisProg, ViperGPT, MM-ReAct, and Visual Sketchpad, followed by image–code and flexible tool-interface approaches.
  • Faithful tool use: CodeV extends tool-based visual agents by exposing their faithfulness limitations and emphasizing evaluation and training for evidence-consistent tool outputs.This focus distinguishes the work from approaches centered primarily on composing visual operations or expanding tool interfaces.
  • Reinforcement learning supervision: Process-level feedback addresses credit-assignment limitations associated with sparse outcome rewards, while outcome-only GRPO can still induce unfaithful behaviors.Earlier and newer RL frameworks motivate combining verifiable signals with denser process supervision.

7. Conclusion

The paper presents CodeV and TAPO for training agentic multimodal models with faithful visual reasoning, reporting strong performance and high faithful tool-use rates. It concludes that outcome-only RLVR is limited and that explicit tool-use supervision matters, while noting costs and scope limitations of the current judge.

  • 7. Conclusion: CodeV uses TAPO to reward faithful visual tool use directly rather than focusing solely on final-answer accuracy.The model is trained as a code-based visual agent within the paper’s process-level reinforcement-learning framework.
  • 7. Conclusion: CodeV attains strong performance and high faithful tool-use rates across visual search and reasoning benchmarks.The conclusion presents these as the paper’s empirical outcomes across the evaluated tasks.
  • 7. Conclusion: The results expose a limitation of outcome-only RLVR and support explicitly supervising tool use in agentic multimodal systems.The paper frames this supervision as important for trustworthy visual reasoning.
  • Limitations and Future Work: TAPO relies on a static judge that adds deployment cost and is mainly tailored to image-centric operations such as cropping.Broader tool ecosystems will require scalable, verifiable faithfulness metrics and task-specific reward signals.

9. Data preparation

CodeV combines cold-start SFT with TAPO-based RL, using curated visual reasoning data and process-level supervision to improve benchmark performance and faithfulness.

  • Data preparation: Cold-start SFT teaches CodeV to write code, inspect visual evidence, and answer questions before RL training.Direct Zero-RL training is unstable: policies may hack tool rewards or avoid code entirely.
  • Data preparation: 333K SFT samples are drawn from Thyme-SFT traces featuring code-based image operations and mathematical computation.The corpus was distilled from more than 4M raw multimodal examples.
  • Main benchmark results: CodeV reaches 84.8 on V* and 91.0 on HRBench-4K-FSP, while improving over Qwen2.5-VL-7B by +9.8 on V*.It matches or exceeds tool-based baselines across large-image benchmarks.
  • Main benchmark results: TAPO improves beyond shared tool-use training foundations, producing more stable and generalizable gains across perception and high-resolution visual search tasks.The reported advantage is attributed to process-level reinforcement learning rather than isolated benchmark overfitting.
  • Math and reasoning benchmarks: CodeV achieves 71.8 on MathVista, 49.2 versus GPT-4o's 50.2 on MathVerse-Mini, and 33.6 on MathVision-Mini.It remains competitive on MMMU and CharXiv while surpassing 7B peers.
  • Faithful tool use: Faithfulness declines as resolution and task difficulty increase, although CodeV shows stronger robustness under this progression.The evaluation covers V*, HRBench-4K, and HRBench-8K and combines faithful tool use with correct answers.

11. Perturbation Analysis

Perturbation tests examine whether models depend on returned visual evidence, and CodeV is more behaviorally sensitive to altered tool outputs than baseline agents.

  • Faithfulness analysis: Faithfulness analysis evaluates whether intermediate crops contain the queried evidence jointly with final-answer correctness.This distinguishes evidence-aligned tool use from merely producing a correct crop or answer.
  • Perturbation analysis: CodeV changes its reasoning and subsequent actions more substantially under tool-output perturbations than Pixel-Reasoner and DeepEyes.The perturbations mask, replace, randomize, or remove returned tool-output images.
  • Perturbation analysis: CodeV's final-answer change rate is around 8–10%, compared with roughly 1–5% for the baselines.The answer response is weaker than the changes observed in internal reasoning and action sequences.
  • Perturbation analysis: Perturbed CodeV trajectories often switch to alternative reasoning paths, so the final answer can remain unchanged despite altered internal reasoning and actions.The authors interpret the broader behavioral changes as evidence that CodeV relies on tool outputs during inference.

12. Tool Use analysis

Figure 8 examines how tool-call frequency relates to accuracy across V*, HRBench-4K, and HRBench-8K.

  • Figure 8 buckets episodes by 0, 1, 2, or 3+ tool calls, reporting each bucket’s share of test examples and within-bucket accuracy across three datasets.The green bar portion represents the bucket’s fraction of all examples multiplied by its accuracy.

13. Python Sandbox Design

The Python sandbox makes generated visual-tool code safer, easier to execute, and more useful by returning artifacts and errors as observations.

  • Security and isolation: Sandboxed execution isolates file I/O, blocks dangerous filesystem operations, and enforces wall-clock time limits.These controls prevent code from affecting external files or processes and abort overlong runs.
  • Usability and code normalization: The sandbox normalizes paths, patches minor code issues, predefines common variables, and preserves imports across tool calls.These conveniences reduce routine coding burden and allow later segments to reuse earlier results.
  • Visualization and artifacts as observations: Newly created figures are captured as image observations, turning visualizations into explicit evidence for the agent.The system intercepts plotting calls such as plt.show().
  • Error handling and iterative repair: Execution errors are returned to the model so it can revise code or bypass the tool and continue reasoning.Exception handling therefore supports iterative repair within a broad but controlled action space.

14. RL Prompt Design

The RL prompt defines image-question answering with optional executable Python image processing and strict think/answer formatting. A judge prompt scores whether generated images contain the queried object, including partial and mismatched matches.

  • Prompt and execution: The prompt lets the agent answer an image question directly or execute Python to crop, resize, rotate, or adjust contrast before answering.Tool outputs are returned through sandbox_output, enabling another reasoning or coding round.
  • Prompt and execution: The interaction requires Python code in a marked block, saved processed-image filenames, and explicit variable definitions with printed outputs for calculations.
  • Output format: The output format requires detailed reasoning inside <think> and the final response inside <answer>.
  • Reward judging: The perception reward judge assigns 1 for a clear object match, 0.5 for a partial match, and 0.25 when the generated image does not match the question.

15. Training Dynamics

During TAPO training, format reward saturates quickly while accuracy and tool-consistency rewards continue rising. Response length and average tool calls decrease as rewards improve.

  • Training rewards: Format reward quickly reaches approximately 0.3, while accuracy and tool-consistency rewards continue increasing throughout training.
  • Training behavior: Response length and mean tool calls per step gradually decrease even as TAPO rewards increase.

16. Qualitative Examples

The qualitative examples show faithful answers produced after targeted crops and zooms, alongside unfaithful cases where initial crops omit the queried context. Corrected crops restore the relevant spatial evidence and support the final answers.

  • 16.1. Faithful Example 1: In the slippers example, cropping and zooming the boat region makes the slippers visible and supports the answer that they are blue.The rollout identifies the slippers near the boat’s center before returning option C.
  • 16.2. Faithful Example 2: In the candles example, a targeted zoom of the background table makes the candles distinguishable and supports the answer that they are yellow.The example returns option B after describing the cropped and zoomed view.
  • 16.3. Unfaithful Example 1: The mailbox example begins with a crop showing only the woman’s face and wall, so the range is adjusted to include both objects before answering right.
  • 16.4. Unfaithful Example 2: The recycle-bin example similarly expands an initially inadequate crop, after which both objects are visible and the person is identified as standing in front.
Loading 2511.19661v2…