PaperlayerView exact revision on arXiv

Source-linked AI summary

iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning

Chang-Bin Zhang, Yujie Zhong, Qiang Zhang, Kai Han

arXiv:2605.31096v1cs.CV

TL;DR

The inference-time efficacy of explicit visual grounding in multimodal reasoning remains underexplored, while textual Chain-of-Thought can outperform grounded reasoning across most benchmarks. iVGR uses dual-stream reinforcement learning and a consistency reward to internalize localization in textual reasoning, achieving consistent improvements across benchmarks while retaining tool-assisted flexibility.

  • Problem

    The inference-time efficacy of visually grounded Chain-of-Thought with explicit crops or bounding boxes remains insufficiently scrutinized.

  • Method

    iVGR uses dual-stream reinforcement learning and a consistency reward to align textual reasoning with high-quality grounded rollouts.

  • Results

    Textual Chain-of-Thought outperforms grounded reasoning across most benchmarks, while iVGR achieves significant improvements across multiple benchmarks and model sizes.

  • Takeaways & Limitations

    Visual localization can be internalized into textual reasoning without explicit grounding outputs, while preserving compatibility with tool-assisted workflows.

  • Takeaways & Limitations

    The method requires bounding-box annotations, imposing additional demands on data labeling.

Abstract

from arXiv · show

While visually grounded Chain-of-Thought (CoT) has emerged as a promising paradigm to enhance fine-grained perception in multimodal large language models (MLLMs), its efficacy during the inference phase remains underexplored. In this work, we empirically find that mandating explicit object boxes in visually grounded CoT during inference often degrades performance compared to standard textual CoT, which reasons without explicit visual grounding. We hypothesize that the visual localization capability can be internalized into the textual CoT and that the mandatory explicit grounding introduces unnecessary interference with the model's primary objective of answer prediction. To address this problem, we propose Internalizing Visually Grounded Reasoning (\textbf{iVGR}), a novel reinforcement learning framework that transfers localization capabilities into the textual reasoning process. We employ a dual-stream training strategy, where a textual stream is aligned with a high-quality visually grounded stream via a proposed consistency reward, enabling the model to localize accurately without explicit grounding during inference. Extensive experiments demonstrate that our method significantly outperforms existing baselines on fine-grained benchmarks, while maintaining the flexibility to support tool-assisted inference workflows.

1. Introduction

The introduction finds that explicit visual grounding during inference can hurt answer prediction, motivating iVGR to internalize localization within textual reasoning. iVGR uses dual-stream reinforcement learning and consistency rewards to transfer grounded reasoning while retaining tool-assisted flexibility.

  • Motivation: Explicit grounding during inference lacks rigorous scrutiny and can impose unnecessary interference with answer prediction.TreeVGR’s visually grounded CoT consistently underperforms textual CoT across all IoU intervals, while inaccurate crops may introduce visual noise.
  • Motivation: The analysis suggests localization capability can be implicitly transferred into textual CoT, removing the need for explicit grounding outputs at test time.This insight is presented as the basis for avoiding mandatory grounding during inference.
  • Method: iVGR trains grounded and textual rollout streams jointly with a consistency reward that aligns textual reasoning to high-quality grounded trajectories.The grounded stream predicts explicit boxes, whereas the textual stream performs standard reasoning.
  • Results: Experiments on Qwen2.5-VL and Qwen3-VL report significant improvements across multiple benchmarks while maintaining compatibility with tool-assisted workflows at test time.The contribution summary attributes these results to the proposed models across multiple benchmarks.

2. Related Work

Prior work establishes CoT and RL post-training as foundations for reasoning in LLMs and MLLMs, while visually grounded reasoning addresses fine-grained visual understanding. iVGR differs by transferring grounded reasoning into pure textual CoT while remaining compatible with tool-assisted workflows.

  • Reasoning MLLMs: CoT has become a cornerstone for complex reasoning in both LLMs and MLLMs.The passage cites work establishing CoT across language and multimodal models.
  • Reasoning MLLMs: DeepSeek-R1 and OpenAI-o1 demonstrate the effectiveness of RL post-training for scaling reasoning capabilities.These models are presented as recent milestones in RL-based reasoning.
  • Visually Grounded Reasoning: Visually grounded reasoning research targets the challenges MLLMs face with high-resolution, fine-grained images.The passage situates this research within broader progress in multimodal understanding.
  • Visually Grounded Reasoning: Unlike approaches requiring mandatory explicit tool calls or coordinate generation, iVGR transfers grounded reasoning capability into pure textual CoT.This distinction defines the method’s position relative to existing visually grounded reasoning paradigms.
  • Visually Grounded Reasoning: The method remains compatible with tool-assisted workflows for further progress on fine-grained benchmarks.Tool assistance is retained as an optional workflow despite textual inference.

3. Method

iVGR uses dual-stream reinforcement learning to transfer visual localization from explicit grounded reasoning into textual CoT. It aligns textual rollouts with high-quality grounded references through consistency rewards while preserving optional box-generation for tool-assisted inference.

  • Dual-Stream Training: iVGR trains grounded and textual rollout streams in parallel, using explicit boxes in one stream and standard natural-language reasoning in the other.The grounded stream predicts bounding boxes for referenced objects, while the textual stream requests only reasoning and an answer.
  • Reward Design: The grounded stream combines format, answer-accuracy, and localization-quality rewards, while the textual stream adds a consistency reward to standard format and accuracy rewards.Grounded rollouts must follow the required structure and answer correctly; textual rollouts are additionally rewarded for semantic alignment with grounded references.
  • Reference Selection: A grounded rollout becomes a reference only when it has valid format, a correct answer, and localization quality above the predefined IoU threshold τ.This filtering prevents low-quality visual perceptions from supervising the textual stream.
  • Reference Selection: The rollout archive retains the highest-localization-quality valid grounded CoT found for each query, stabilizing consistency supervision as the policy changes.The archive ensures consistency rewards use the best available visual guide rather than only the current batch’s rollouts.
  • Consistency Scoring: An external LLM scores whether textual and grounded CoTs describe the same visual content, assigning consistency reward α and setting it to 0 when no valid reference exists.The rubric ranges from α = 1.0 for identical content to α = 0.0 for explicit contradiction.
  • Tool-Assisted Inference: Despite internalizing grounding into textual reasoning, the model retains explicit box generation and can combine it with cropping tools in a tool-assisted test-time workflow.This preserves flexibility for handling fine-grained details during inference.

4. Experiments

Experiments evaluate iVGR across fine-grained, general VQA, chart-understanding, and multidisciplinary reasoning benchmarks. The method outperforms existing approaches, scales across model sizes, supports tool-assisted test-time scaling, and benefits from consistency rewards and rollout archives.

  • Evaluation Benchmarks: Evaluations cover fine-grained perception, general VQA, chart understanding, and multidisciplinary reasoning benchmarks.Fine-grained datasets include V*, HR4K, and HR8K; broader evaluations include MME-RW-Lite, POPE, RealWorldQA, CV-2D/3D, ChartQA, AI2D, WeMath, MMStar, MMMU, and MMK12.
  • Main Results: iVGR surpasses leading tool-based and explicit grounding-based baselines, improving state-of-the-art performance by 2.6% on V* and 1.7% on HR8K.The comparison uses Qwen2.5-VL-7B and reports continued superiority on general VQA benchmarks.
  • Main Results: Average improvements reach 4.4% with Qwen3-VL-8B and 3.3% with Qwen3-VL-32B, demonstrating scalability across model sizes.These experiments extend beyond the Qwen2.5-VL-7B setting to larger Qwen3-VL models.
  • Test-Time Scaling: Predicted-box crops improve HR4K performance by 1.1% for Qwen2.5-VL-7B and 1.5% for Qwen3-VL-8B.Adding a union crop further improves performance by 3.5% and 2.3%, respectively, supporting tool-assisted test-time scaling.
  • Analysis of Training Dynamics: Fully consistent rollouts rise to approximately 50% by training’s end, while rollout-archive utilization increases from zero initially to approximately 50% in the third epoch.Intermediate consistency scores show a rise-and-fall pattern, and archive utilization reaches approximately 30% in the second epoch.
  • Ablation of Components: The consistency reward improves performance by 7.3% over the Qwen2.5-VL-7B baseline, while adding the rollout archive raises average performance to 72.4 for a total gain of 8.2%.The ablation compares RL variants initialized from the same cold-start model and finds limited gains from the dual-stream architecture alone.

5. Conclusion and Limitations · A. Prompts

The paper concludes that iVGR internalizes visual localization through dual-stream reinforcement learning, while identifying annotation, judge-model, and high-resolution-image limitations. Its prompts define separate grounded and textual training streams plus a consistency judge that scores semantic agreement between their reasoning chains.

  • 5. Conclusion and Limitations: iVGR internalizes visual localization into textual reasoning through reinforcement learning and aligns textual CoT with grounded reference rollouts using a consistency reward.The approach uses dual-stream training to enforce semantic alignment between the textual and grounded reasoning processes.
  • 5. Conclusion and Limitations: Bounding-box annotations impose additional labeling demands, which existing detection datasets and off-the-shelf detectors for pseudo-labeling can help reduce.The paper proposes repurposing detection datasets and using detector-generated pseudo-labels to reduce human effort.
  • 5. Conclusion and Limitations: External LLM scoring can be affected by hallucinations, while scaling the judge model and ensembling multiple judge scores may improve consistency-reward quality.The paper identifies judge accuracy as an ongoing challenge and discusses larger or ensemble judges as remedies.
  • 5. Conclusion and Limitations: Without crop-tool training, the grounded stream may hallucinate or miss details in very high-resolution or extremely fine-grained images.The resulting reference-CoT quality can limit the consistency reward, while crop-tool use at test time can enhance performance.
  • A. Prompts: The evaluation prompt requires textual reasoning inside <think>...</think> and a concise answer inside <answer>...</answer>.This prompt is used to evaluate off-the-shelf DeepEyes and TreeVGR models with textual CoT.
  • A. Prompts: The grounded-stream training prompt requires assistants to localize referenced objects with bounding-box coordinates during reasoning.The example identifies the left ball with coordinates [50,80,120,150] before answering Blue.
  • A. Prompts: The textual-stream training prompt retains the <think> and <answer> structure but omits the grounded-stream requirement for bounding boxes.It directs the assistant to reason mentally before providing the answer.

B. Training Data

The training data combines 37K grounded natural-image samples with 14K general reasoning samples, totaling 51K examples. General reasoning data is difficulty-filtered, while batches emphasize duplicated grounded samples for dual-stream training.

  • Training Data: The dataset contains TreeVGR-RL-37K with 37K natural-image samples and target bounding-box annotations, plus 14K general reasoning samples from OpenMMReasoner and ArxivQA.The general reasoning samples lack bounding-box annotations.
  • Training Data: The three training-data components—TreeVGR-RL, OpenMMReasoner, and ArxivQA—contain 51K samples in total.
  • Data Construction: Filtering with five cold-start Qwen2.5-VL-7B rollouts retains samples with pass rates between 20% and 40%, selecting 12K OpenMMReasoner and 2K ArxivQA samples.This corresponds to retaining queries with 1 or 2 correct answers out of 5.
  • Batch Sampling Strategy: Each batch uses 45% TreeVGR-RL-37K samples, duplicated across grounded and textual streams to occupy 90% of the batch, with the remaining 10% from general reasoning data.The general reasoning subset consists of OpenMMReasoner and ArxivQA samples.

C. Implementation Details

The implementation uses a two-stage cold-start and reinforcement-learning pipeline with distinct optimization settings and model-specific schedules. Training uses A100 GPUs, while a separate judge model evaluates accuracy and consistency.

  • Training Pipeline: The pipeline comprises cold-start and reinforcement-learning stages.The paper explicitly identifies these as the two training stages.
  • Cold-Start Stage: Cold-start fine-tunes a constructed 35K dataset for one epoch with AdamW, global batch size 64, cosine scheduling, and initial learning rate 1e-5.The stage uses the Llama-Factory framework.
  • RL Stage: RL experiments use VeRL, AdamW, constant learning rate 1e-6, KL penalty β = 0.01, rollout sample count N = 5, maximum length 4,096, and temperature 1.0.The RL implementation details are split across the optimization and rollout settings passages.
  • RL Stage: Qwen2.5-VL-7B and Qwen3-VL-8B train for 360 steps with batch size 512, while the 32B model uses 500 steps and batch size 128.The smaller batch size for the 32B model accommodates hardware constraints.
  • Evaluation: Qwen2.5-72B-Instruct serves as the universal judge for accuracy and consistency metrics through vLLM at temperature 0.01.The judge model is used for both evaluation metrics.
  • Training Cost: Training uses 8 NVIDIA A100 GPUs, with 4 additional A100 GPUs serving the Qwen2.5-72B judge model via vLLM.The judge’s overhead is described as minimal because it outputs only scalar scores and supports batch inference.

D. Experimental Results · D.1. Ablation Study on the Training Data Split

Under identical training data, iVGR outperforms TreeVGR, while adding general reasoning samples preserves comparable benchmark performance and robust cross-domain generalization. The final configuration therefore retains the additional 14K samples.

  • D. Experimental Results: Under identical data conditions, iVGR outperforms TreeVGR-7B by 2.0% in average accuracy.Both models use Qwen2.5-VL-7B fine-tuned exclusively on TreeVGR-RL-37K.
  • D.1. Ablation Study on the Training Data Split: The comparison strictly aligns iVGR’s training setup with the TreeVGR baseline.The aligned setup uses the same model and TreeVGR-RL-37K dataset.
  • D.1. Ablation Study on the Training Data Split: The 2.0% average-accuracy improvement validates the superior efficacy of iVGR’s reinforcement-learning framework.This result is reported under identical data conditions.
  • D.1. Ablation Study on the Training Data Split: Adding 14K general reasoning samples yields comparable performance on the evaluated natural-image VQA benchmarks.The passage does not report a numerical difference for this comparison.
  • D.1. Ablation Study on the Training Data Split: The final configuration retains the additional 14K samples despite comparable benchmark performance.Retention is motivated by preserving robust generalization across diverse visual domains.
  • D.1. Ablation Study on the Training Data Split: The retained training subset is intended to preserve robust generalization across diverse visual domains.This motivation accompanies the comparable results on natural-image VQA benchmarks.

D.2. Comparison with the Cold-Start Model

The method consistently outperforms baseline and cold-start models in textual-CoT evaluation, despite mixed cold-start results potentially caused by forgetting in general domains. It achieves the highest average accuracy across all evaluated model sizes, improving fine-grained VQA without compromising general reasoning capabilities.

  • Comparison with the Cold-Start Model: The method consistently outperforms both baseline and cold-start models in textual-CoT evaluation.Cold-start models show mixed results, potentially due to forgetting in general domains.
  • Comparison with the Cold-Start Model: It achieves the highest average accuracy across Qwen2.5-VL-7B, Qwen3-VL-8B, and Qwen3-VL-32B.The comparison is reported in Table 13 across all three model sizes.
  • Comparison with the Cold-Start Model: The RL framework enhances fine-grained VQA without compromising general reasoning capabilities.This conclusion follows from the method’s strongest average accuracy across model sizes alongside its reported general-domain behavior.

D.3. Ablation Study on the Grounding Capability

The ablation evaluates iVGR’s grounding capability through localization quality and answer accuracy, comparing it with a grounded-stream-only baseline and adding TreeBench for more challenging fine-grained localization.

  • Grounding capability: iVGR’s grounding capability is assessed using localization quality (IoU) and answer accuracy of its grounded Chain-of-Thought.The evaluation compares iVGR against a grounded-stream-only baseline.
  • Grounding capability: TreeBench is included as a more challenging benchmark for fine-grained localization.The passage identifies TreeBench as an additional evaluation beyond the main comparison.
  • Grounding capability: On HR8K, iVGR achieves an IoU of 24.1 versus 27.0 for the grounded-stream-only baseline.The reported values indicate comparable localization quality between the two methods on HR8K.

E. Qualitative Results

Qualitative comparisons show that the consistency reward improves implicit localization in iVGR, while textual CoT can avoid grounded-CoT localization failures that propagate to incorrect answers.

  • Effect of the Consistency Reward: Without the consistency reward, the model exhibits clear localization deviation, whereas iVGR attends to the correct region and identifies the trailer color accurately.The variant associates the trailer with a nearby blue object and reports an incorrect color; iVGR identifies the actual trailer as orange.
  • Effect of the Consistency Reward: The consistency reward sharpens the textual stream’s implicit localization, producing more reliable visual descriptions in the reasoning trace.
  • Grounded CoT vs. Textual CoT in iVGR: Grounded CoT can produce partial bounding boxes that cause incorrect counting, while iVGR’s textual CoT enumerates the computers instead.This represents a localization error that propagates to the final answer.
Loading 2605.31096v1…