PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded Reinforcement Learning for Visual Reasoning

Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J. Tarr, Aviral Kumar, Katerina Fragkiadaki

arXiv:2505.23678v3cs.CV

TL;DR

Visual reasoning models often lack grounded, adaptive visual strategies, limiting how they connect language reasoning to spatial evidence. ViGoRL trains each reasoning step with explicit image coordinates and uses multi-turn RL to obtain zoomed visual feedback. Across visual reasoning benchmarks, it consistently outperforms supervised fine-tuning and conventional RL baselines, including 86.4% accuracy on V*Bench. The results associate grounding with stronger exploration, subgoal setting, verification, and interpretable reasoning, while final-answer-only rewards remain vulnerable to reward hacking.

  • Problem

    Many VLMs and naïve RL methods produce abstract, ungrounded reasoning, despite visual tasks requiring attention to spatial evidence and adaptive search.

  • Method

    ViGoRL anchors each reasoning step to image coordinates, bootstraps grounded trajectories with MCTS and SFT, and applies GRPO with multi-turn zoomed visual feedback.

  • Results

    ViGoRL consistently outperforms existing methods across visual reasoning benchmarks, achieving 86.4% accuracy on V*Bench with multi-turn RL.

  • Takeaways & Limitations

    Grounding amplifies region exploration, subgoal setting, visual verification, and backtracking, while making reasoning spatially accurate and helpful to understand.

  • Takeaways & Limitations

    Final-answer-only rewards can permit reward hacking, allowing positive reward for partially incorrect or underspecified reasoning.

Abstract

from arXiv · show

While reinforcement learning (RL) over chains of thought has significantly advanced language models in tasks such as mathematics and coding, visual reasoning introduces added complexity by requiring models to direct visual attention, interpret perceptual inputs, and ground abstract reasoning in spatial evidence. We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with RL to explicitly anchor each reasoning step to specific visual coordinates. Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces, guiding visual attention to task-relevant regions at each step. When fine-grained exploration is required, our novel multi-turn RL framework enables the model to dynamically zoom into predicted coordinates as reasoning unfolds. Across a diverse set of visual reasoning benchmarks--including SAT-2 and BLINK for spatial reasoning, V*bench for visual search, and ScreenSpot and VisualWebArena for web-based grounding--ViGoRL consistently outperforms both supervised fine-tuning and conventional RL baselines that lack explicit grounding mechanisms. Incorporating multi-turn RL with zoomed-in visual feedback significantly improves ViGoRL's performance on localizing small GUI elements and visual search, achieving 86.4% on V*Bench. Additionally, we find that grounding amplifies other visual behaviors such as region exploration, grounded subgoal setting, and visual verification. Finally, human evaluations show that the model's visual references are not only spatially accurate but also helpful for understanding model reasoning steps. Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.

1 Introduction

Visual reasoning tasks require adaptive visual attention and intermediate reasoning, yet standard VLMs and naïve RL often produce abstract, ungrounded reasoning. ViGoRL addresses this by explicitly grounding reasoning steps in image coordinates and refining them with multi-turn visual feedback.

  • Motivation: Visual reasoning ranges from salient-cue recognition to sequential search, but many VLMs answer end-to-end without adaptive strategies or exposed intermediate reasoning.Prompt-based methods explicitly decompose visual reasoning, unlike typical single-pass systems.
  • Motivation: Naïve RL fine-tuning typically yields abstract, ungrounded reasoning rather than richer visual cognitive behaviors.This pattern is consistent with evidence that spatial references improve VLM performance and interpretability.
  • Approach: ViGoRL anchors each reasoning step to an image coordinate, allowing attention to be progressively refined as reasoning unfolds.The model learns to propose and use spatial grounding without explicit human-provided grounding supervision.
  • Approach: The method bootstraps grounded trajectories with MCTS and SFT, then applies GRPO to reinforce grounded sequences leading to correct answers.A multi-turn extension lets the model request zoomed-in visual feedback for fine-grained inspection.
  • Results: 12.9 points on SAT-2 and 2.0 points on BLINK are ViGoRL's accuracy gains over vanilla GRPO.Across the evaluated benchmarks, ViGoRL consistently outperforms existing methods, while multi-turn RL improves fine-grained grounding and visual search.
  • Results: Grounding amplifies region exploration, goal setting, and visual verification, while human evaluations find the references spatially accurate and helpful.Ablations show that removing spatial anchoring significantly worsens performance.

2 Related Work

Prior VLM research uses prompting, distillation, supervised fine-tuning, and reinforcement learning to elicit explicit reasoning, but each line of work leaves room for visually grounded RL.

  • Programmatic Reasoning in VLMs: VLMs perform multimodal tasks through large-scale pretraining but struggle with counting, spatial reasoning, and compositional understanding.Prompting methods such as CoT, mm-CoT, IoT, Mind's Eye, V*, Sketchpad, and VisProg guide explicit visual reasoning.
  • Distillation and Supervised Fine-Tuning: Supervised methods instill reasoning skills by training models on curated or distilled reasoning trajectories.Examples include STaR and S1 for text, and LLAVA-CoT, ICAL, VPD, and Mulberry for VLMs.
  • Reinforcement Learning on Chains of Thought: RL over chains of thought improves reasoning in verifiable domains such as mathematics and coding.Recent approaches use online RL with outcome-based rewards, while analyses suggest RL mainly amplifies existing capabilities.

3 Preliminaries

The paper frames VLM reasoning as generating textual traces and answers from visual inputs, then examines whether standard RL amplifies useful visual behaviors or instead reinforces ungrounded shortcuts.

  • Reasoning Formulation: A VLM policy maps an image and query to a sequential textual reasoning trace ending in a verifiable answer.The answer may be a class label, bounding box, or discrete action such as clicking an element.
  • RL Objective: RL optimizes reasoning-trace policies using expected returns based on task performance and structured-format adherence.The reward typically includes final-answer correctness and compliance with the required reasoning format.
  • Observed Behaviors: Current VLMs often fail to reference fine-grained image inputs, examining only 1.44 regions per task with minimal visual verification and no backtracking.These behaviors indicate that vision is treated largely as static context rather than actively referenced evidence.
  • Observed Behaviors: Task-level RL slightly increases region exploration to 1.8 but eliminates visual subgoal setting at 0.00 and retains no backtracking at 0.00.Optimizing correctness without grounded-reasoning incentives therefore fails to instill the targeted visual reasoning skills.
  • Observed Behaviors: Figure 2 contrasts vague, often incorrect reasoning from Standard CoT and Vanilla GRPO with more precise spatial references and correct outcomes from Visually Grounded RL.The figure uses yellow for vague references, red for incorrect answers, blue for spatial grounding, and green for correct outcomes.
  • Interpretation: The hypothesized failure reflects both abstract initial sampling by pretrained VLMs and reward amplification of strategies that achieve final correctness regardless of process.The passage presents these as interrelated explanations for standard RL's behavior.

4 Visually Grounded Reinforcement Learning (ViGoRL)

ViGoRL makes each reasoning step spatially explicit, bootstraps grounded trajectories with MCTS and SFT, and refines them with GRPO, including multi-turn crop feedback for fine-grained inspection.

  • Grounded Reasoning: Each reasoning step is a textual thought paired with an (x, y) image location, and the full trajectory ends with an answer.This grounding guides exploration, visual subgoals, verification, and revisiting regions when uncertainty arises.
  • Pipeline: The pipeline warm-starts models on structured grounded chains before using reinforcement learning to refine those behaviors.This two-stage design explicitly addresses ungrounded reasoning patterns observed in VLMs.
  • Warm-Start Data Generation via MCTS: MCTS with a teacher generates grounded reasoning trees, whose selected paths are linearized for SFT before GRPO refinement with outcome-based rewards.Figure 3 presents this as the sequence of tree generation, SFT, and GRPO.
  • Warm-Start Data Generation via MCTS: MCTS is used because linear rollouts cannot enforce the desired iterative exploration and corrective loops, while purely human-curated traces are costly to scale.Distillation without MCTS also degrades out-of-distribution spatial generalization after GRPO training.
  • Warm-Start Data Generation via MCTS: SFT on MCTS chains commonly yields dense visual subgoals, verification, broad region exploration, and visual backtracking.These behaviors are observed after fine-tuning the base model on selected MCTS paths.
  • Reinforcement Learning with Spatially-Grounded Reasoning Steps: The spatially grounded reward combines format and task rewards, granting format credit only when coordinate references are valid.This couples output validity with explicit spatial grounding and task correctness.
  • Multi-Turn Reinforcement Learning for Visual Feedback: Multi-turn RL lets the model call a crop tool to receive a higher-resolution view centered on a selected coordinate during reasoning.Dialog traces alternate thoughts, tool calls, observations, and continued reasoning or an answer.
  • Multi-Turn Reinforcement Learning for Visual Feedback: The multi-turn format reward combines strict dialog grammar with a diversity bonus for sufficiently distinct tool-call coordinates.The grammar requires ordered tags and a complete final answer; distinct coordinates must be at least 10px apart, with up to four bonuses.

5 Experiments

ViGoRL is evaluated across spatial reasoning, GUI grounding, live visual web interaction, visual search, ablations, behavioral analyses, and human interpretability. Across these experiments, explicit grounding, MCTS warm starts, GRPO, and multi-turn visual feedback each contribute to stronger performance and richer visual reasoning behaviors.

  • 5.1 Results: ViGoRL is evaluated on spatial reasoning, GUI understanding, live visual web interaction, and visual search benchmarks.The experiments compare against base models, direct SFT, vanilla GRPO, and other specialized systems.
  • 5.1.2 Results: 62.9% SAT-2 accuracy for ViGoRL-3B exceeds the base model by 16.8 points and Vanilla GRPO by 12.9 points.ViGoRL-3B also reaches 67.1% on RoboSpatial and 48.5% on BLINK.
  • 5.1.2 Results: 91.0% on ScreenSpot-V2 and 33.1% on ScreenSpot-Pro are achieved by ViGoRL-7B, outperforming comparable open-source VLMs trained with up to 13M GUI examples.On ScreenSpot-Pro, ViGoRL-3B improves 21.8 points over the base model, 6.1 over direct SFT, and 2.1 over vanilla GRPO.
  • 5.1.2 Results: ViGoRL surpasses ICAL by 3.0% on VisualWebArena while using only image-based set-of-marks inputs rather than HTML-derived textual inputs.VisualWebArena requires live interaction with web pages without access to HTML or underlying text.
  • 5.1.2 Results: 86.4% on V∗Bench exceeds GPT-4o at 66.0%, Gemini-Pro at 48.2%, VisProg at 41.4%, and Sketchpad-GPT-4o at 80.3%.The result comes from multi-turn RL with visual feedback for visual search and small-element detection.
  • 5.1.2 Results: Multi-turn zoomed-in feedback raises ScreenSpot-Pro accuracy to 32.3%, 1.2 points above the best non-feedback variant, and improves low-resolution performance by 2.4%.The zooming capability is intended for more precise inspection of small elements.
  • 5.3 RQ3: Visual Reasoning Behaviors: Explicit grounding increases region exploration, grounded subgoals, visual verification, and visual backtracking, with a 3B model reaching accuracy comparable to a 72B model.The reported averages are 3.5 regions, 1.1 grounded subgoals, 0.39 verification instances, and 0.47 backtracking instances per example.

6 Conclusion and Discussion

The paper argues that visual grounding structures model reasoning around specific visual regions, mirroring human spatial attention and enabling iterative visual inspection. It presents grounding as a central principle for building more queryable and perceptually aligned visual reasoning systems.

  • Discussion: Spatially anchoring reasoning steps helps models iteratively reference, inspect, and verify specific visual regions.The authors associate this behavior with subgoal formulation, visual verification, and backtracking.
  • Discussion: Grounding scaffolds reasoning with external visual structure rather than serving only to reduce computational load.The discussion connects this design to human spatial attention and visual routines.
  • Discussion: The authors propose pointing, zooming, and verifying as a central architectural and algorithmic principle for future visual reasoning agents.They link these behaviors to systems that are queryable, adaptable, and aligned with perceptual experience.

A1 Appendix Overview

The appendix organizes limitations, broader impacts, cognitive-science discussion, methods, behavioral analysis, human evaluation, experiments, and example outputs into separate sections.

  • A1 Appendix Overview: Section A2 covers limitations and future work, while Section A3 covers broader impacts.The appendix overview identifies these as the first two substantive sections.
  • A1 Appendix Overview: Sections A4 through A9 provide cognitive-science discussion, methods details, behavioral analysis, human evaluation, additional results, and example model outputs.These sections extend the paper’s discussion and empirical documentation.

A2 Limitations and Future Work

The appendix identifies limitations involving sparse reward supervision, restricted tool use, inefficient reasoning depth, and incomplete understanding of grounding’s internal mechanisms.

  • Intermediate Reward: Final-answer-only rewards can permit reward hacking when spatial reasoning is partially incorrect or underspecified.The authors suggest dense rewards for correct reasoning steps and accurate grounding as future work.
  • Expanded Tool Use and Adaptive Control: The current toolset supports pointing, cropping, and zooming, but not richer actions such as highlighting, region comparison, or 3D navigation.Future systems may need to learn how to balance visual actions dynamically.
  • Learning When and How Much to Reason: The model often produces long reasoning chains even for relatively simple questions, motivating adaptive control over when and how much to reason.The proposed direction is to adapt chain-of-thought depth to task complexity.
  • Interpreting Attention Patterns: The mechanisms by which explicit grounding changes internal attention dynamics remain underexplored.The authors call for comparisons with human visual routines, task-directed attention, and deictic planning.

A3 Broader Impacts

The broader-impacts discussion highlights improved transparency and human oversight alongside risks of misuse, over-trust, and bias amplification. It proposes uncertainty marking, audit tools, corrections, and more diverse evaluation as mitigations.

  • Positive Societal Impacts: Grounded reasoning can improve transparency by producing interpretable chains that reference specific image regions.The authors identify potential value in medical imaging, accessibility, assistive robotics, and scientific image analysis.
  • Potential Negative Impacts: Detailed visual reasoning traces could support surveillance or profiling and cause users to over-trust superficially plausible but incorrect outputs.The risk is especially relevant where visual ambiguity or annotation bias affects ground truth.
  • Potential Negative Impacts: Spatial grounding may propagate dataset biases by locking attention onto overrepresented or stereotyped visual regions.The paper recommends uncertainty estimation, trace auditing, user corrections, and diversified data and evaluation.

A4 Additional Discussion

The discussion presents explicit visual grounding as both a cognitive scaffold and a structural inductive bias for visual reasoning. It connects spatially localized attention with improved compositionality, verification, goal-directed reasoning, and generalization.

  • Grounding as a Cognitive Scaffold: Human visual cognition motivates sequentially directing attention to localized regions while solving goal-driven subproblems.Eye-movement patterns vary with task demands, linking visual attention to internal reasoning.
  • Grounding Reduces Hallucination and Enhances Generalization: Grounding visual reasoning has been associated with reduced hallucination and stronger generalization in prior machine-learning studies.Related methods use grounded descriptions or localized actions such as zooming, cropping, and verification.
  • Object-centric and Compositional Inductive Biases: Spatial cues may encourage object-centric and modular representations that improve structured and compositional visual reasoning.Prior work links learned object embeddings and language-guided modules conditioned on spatial cues to these benefits.
  • Grounding as a Cognitive Scaffold: Explicit grounding is framed as a mechanism that supports compositionality, verification, and goal-directed reasoning in visual models.The discussion draws this conclusion from converging cognitive-science and machine-learning perspectives.
  • Grounding as a Cognitive Scaffold: Overall, explicit grounding is characterized as a structural inductive bias rather than merely a way to reduce the search space.The proposed role encompasses compositionality, verification, and goal-directed reasoning.

A5.1 Group Relative Policy Optimization (GRPO)

GRPO stabilizes learning from long multimodal reasoning trajectories by comparing sampled trajectories within groups and applying clipped token-level updates with KL regularization.

  • GRPO overview: GRPO stabilizes policy learning from long-form trajectories using group-normalized advantages and clipped token-level PPO-style updates.The method is designed for long-horizon, multimodal reasoning settings.
  • Group-relative advantage: For each trajectory, GRPO computes a centered advantage by subtracting the group mean reward from that trajectory’s scalar reward.This makes each trajectory’s update relative to the other sampled trajectories in its group.
  • Clipped optimization: The objective applies a clipped surrogate loss at the token level to limit the effect of importance-weighted policy updates.The clipping parameter is ε = 0.2, with β controlling the KL penalty.
  • Clipped optimization: A KL penalty is included alongside clipping to stabilize optimization in long-horizon multimodal reasoning.β denotes the KL penalty coefficient.

A5.2 RL Reward Functions

The reward and training pipeline combines benchmark-specific task rewards with MCTS-generated grounded traces, SFT warm-starting, and GRPO over single- or multi-turn trajectories. Results show that iterative visual feedback helps especially in difficult low-resolution grounding, while reward design and task composition affect outcomes.

  • Task Rewards: Benchmark-specific rewards evaluate answer correctness, coordinate containment, or decomposed action type and argument correctness.SAT-2 uses binary answer accuracy; OS-Atlas uses bounding-box containment; ICAL splits reward equally between action type and argument.
  • Multi-turn Reinforcement Learning Details: Multi-turn training reformats grounded reasoning paths into alternating thought-and-coordinate tool calls with cropped visual feedback until a terminal answer.Backtracking is included, and observation tokens are masked from the GRPO loss while visual inputs remain available to the encoder.
  • Multi-turn Reinforcement Learning Details: A diversity turn bonus prevents convergence to single-turn outputs and enables the model to produce more than one turn on average.The bonus is intended to stabilize multi-turn behavior during RL training.
  • Warm Start Reasoning Traces: MCTS uses VLM-generated reasoning steps and a judge to explore grounded paths, while SFT and GRPO respectively embed and reinforce successful reasoning sequences.The MCTS tree tracks visited coordinates, expected reward, and visit counts, then expands, rolls out, and backpropagates judged rewards.
  • Additional Experimental Results: 96% accuracy is reached by MCTS on SAT-2, while the full training pipeline improves ScreenSpot-Pro low-resolution accuracy from 1.96% for the base model to 21.32%.Multi-turn ViGoRL reaches 23.72% on ScreenSpot-Pro-LR, where iterative visual feedback is especially beneficial.
  • Additional Experimental Results: A Euclidean grounding reward reduces accuracy by 2.2% on ScreenSpot-V2 and 0.7% on ScreenSpot-Pro compared with the binary reward.The result indicates that the tested reward formulation materially affects grounding performance.
Loading 2505.23678v3…