Source-linked AI summary
VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought
Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang
TL;DR
Reasoning-based MLLMs can produce long textual chains but struggle to revisit visual regions for grounded reasoning. VLM-R3 combines VLIR supervision with R-GRPO to interleave region-focused visual evidence and reasoning, and reports state-of-the-art results across multiple benchmarks, including a 14.33% improvement on ScienceQA (87.90% vs. 73.57%).
Problem
Many MLLMs keep reasoning textual after static initial visual grounding, leaving region selection, transformation, and its contribution to reasoning insufficiently learned.
Source: §1 · pp. 1–2Method
VLM-R3 combines VLIR supervision with R-GRPO to train MLLMs to select and transform visual regions and integrate the evidence into interleaved reasoning.
Source: §1 · p. 2, §3.3 · p. 6Results
VLM-R3 achieves state-of-the-art results across multiple benchmarks; on ScienceQA, it improves 14.33% (87.90% vs. 73.57%).
Source: §4.3 · p. 7, §5 · p. 10Takeaways & Limitations
The authors identify adaptively focusing on relevant regions during multi-step inference as a promising direction for visually grounded reasoning systems.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Recently, reasoning-based MLLMs have achieved a degree of success in generating long-form textual reasoning chains. However, they still struggle with complex tasks that necessitate dynamic and iterative focusing on and revisiting of visual regions to achieve precise grounding of textual reasoning in visual evidence. We introduce \textbf{VLM-R$^3$} (\textbf{V}isual \textbf{L}anguage \textbf{M}odel with \textbf{R}egion \textbf{R}ecognition and \textbf{R}easoning), a framework that equips an MLLM with the ability to (i) decide \emph{when} additional visual evidence is needed, (ii) determine \emph{where} to ground within the image, and (iii) seamlessly weave the relevant sub-image content back into an interleaved chain-of-thought. The core of our method is \textbf{Region-Conditioned Reinforcement Policy Optimization (R-GRPO)}, a training paradigm that rewards the model for selecting informative regions, formulating appropriate transformations (e.g.\ crop, zoom), and integrating the resulting visual context into subsequent reasoning steps. To bootstrap this policy, we compile a modest but carefully curated Visuo-Lingual Interleaved Rationale (VLIR) corpus that provides step-level supervision on region selection and textual justification. Extensive experiments on MathVista, ScienceQA, and other benchmarks show that VLM-R$^3$ sets a new state of the art in zero-shot and few-shot settings, with the largest gains appearing on questions demanding subtle spatial reasoning or fine-grained visual cue extraction.
1 Introduction
The introduction identifies a gap in MLLMs’ ability to revisit and reason over visual regions, then presents VLIR and VLM-R3 as a data-and-training response. It reports superior performance on challenging multimodal reasoning tasks.
1 Introduction
- Current MLLMs often keep CoT reasoning textual after static initial visual grounding, limiting dynamic, iterative interaction with specific image regions.The paper highlights region selection and transformation, as well as credit assignment for whether a selected region contributed to correct reasoning, as learning hurdles.Source: §1 · pp. 1–2
- VLM-R3 significantly outperforms existing state-of-the-art models on multimodal reasoning benchmarks, with reported validation across MME, ScienceQA, and MathVista.The introduction also characterizes the results as setting a new benchmark for fine-grained, visually grounded inference.Source: §1 · p. 2
- VLM-R3 combines VLIR cold-start fine-tuning with R-GRPO to learn when and where to look, transform localized evidence, and integrate it into evolving reasoning.The framework is intended to interleave visual and textual reasoning rather than rely only on an initial image grounding.Source: §1 · p. 2
- VLIR provides explicit annotations for region localization, cropping instructions, and semantic enhancement cues within multi-step reasoning narratives.The introduction presents VLIR as a dataset tailored to training and evaluating interleaved visual-textual CoT with region-level interactions.Source: §1 · p. 2
2 Related Work
The related work reviews the development of structured reasoning in language models and its extension to multimodal settings. It notes that many multimodal approaches adapt text-based reasoning without adequately addressing visual information processing.
2.1 Large Language Model Reasoning
- LLM reasoning has expanded from Chain-of-Thought prompting to structured variants and reinforcement-learning approaches, including GRPO.The section describes these approaches as ways to break problems into intermediate steps or develop reasoning capabilities.Source: §2.1 · p. 3
2.2 Multi-modal Large Language Model Reasoning
- Multimodal reasoning work has explored CoT structures and training data, but many methods inherit text-only paradigms and inadequately address visual processing in visually intensive tasks.The section cites text-only reasoning data, staged reasoning, and verification-enhanced approaches as examples of this research direction.Source: §2.2 · p. 3
3 Method
The method combines VLIR supervision, interactive crop-based inference, and R-GRPO to train VLM-R3 for interleaved visual grounding. It links region selection with ongoing textual reasoning.
3.1 Visuo-Lingual Interleaved Rationale (VLIR) Dataset
- VLIR provides interleaved textual reasoning, visual-grounding actions, and corresponding cropped evidence to support iterative visual retrieval and cropping.It is designed to address prior approaches' limited linguistic interleaving, predefined grounding actions, and typically single visual interaction.Source: §3.1 · p. 3
3.1.1 Data Construction
- VLIR draws on document and text understanding, general visual question answering, chart interpretation, and spatial-relation benchmarks.The authors use prompted MLLMs with rejection sampling for some sources and GPT-4o with source metadata for challenging OCR or tabular cases.Source: §3.1.1 · pp. 3–4
3.1.2 Data Filtering
- Rationale filtering rejects crops without a complete, recognizable semantic unit and reasoning paths judged illogical, padded, or redundant.A smaller VLM checks cropped regions, while a text-only LLM evaluates reasoning coherence and relevance.Source: §3.1.2 · p. 4
3.2 Interactive Inference Pipeline
- The inference pipeline repeatedly crops and zooms selected image regions, appends them as visual context, and resumes generation until a final answer is produced.Further text or crop commands can follow each injected crop, enabling multi-step adaptive grounding.Source: §3.2 · p. 5
3.3 Region-Conditioned Reinforcement Policy Optimization (R-GRPO)
- R-GRPO adapts GRPO to question-and-visual-state policies, optimizing generated text and bounding-box actions while rewarding correct answers, valid regions, and reasoning.Its objective weights generated sequences by normalized group-relative reward and includes a KL-divergence term.Source: §3.3 · pp. 5–6
4 Experiments
VLM-R3 outperforms its Qwen2.5-VL 7B base model across benchmarks, with the largest reported gains on tasks requiring precise visual reasoning. Ablations and grounding analyses indicate that interleaved visual evidence, VLIR fine-tuning, R-GRPO, and accurate region grounding all contribute to the reported performance.
4.1 Experiment Setting
- Evaluation spans six capability benchmarks plus HallucinationBench; VLIR fine-tuning uses 11,810 samples from five source datasets.Baselines include open- and closed-source models, both with and without dedicated reasoning modules.Source: §4.1 · p. 6, §4.2 · pp. 6–7
4.3 Main Result
- Against its Qwen2.5-VL 7B base model, VLM-R3 improves by 2.2% on MathVista (70.4% vs. 68.2%), 5.1% on MathVision (30.2% vs. 25.1%), and 14.33% on ScienceQA (87.90% vs. 73.57%).The comparison covers mathematics, general commonsense, science, document understanding, and hallucination assessment.Source: §4.3 · p. 7, Table 1 · p. 7
4.4.1 Effectiveness of Interleaved Chain-of-Thought
- Removing region images while retaining bounding boxes lowers performance across all four ablation benchmarks, especially ScienceQA (-12.5%) and MMMU (-2.8%).The authors note that text-only descriptions miss visual patterns and spatial relationships in tasks such as scientific diagrams.Source: §4.4.1 · p. 8, Table 2 · p. 8
4.4.2 Effectiveness of Finetuning on VLIR
- Skipping VLIR fine-tuning degrades all four ablation benchmarks, particularly ScienceQA (-15.7%) and MMMU (-5.2%), and impairs bounding-box formatting instruction following.The paper links these formatting deficiencies to the performance deterioration.Source: §4.4.2 · p. 8, Table 2 · p. 8
4.4.3 Effectiveness of R-GRPO
- Omitting R-GRPO reduces performance across benchmarks, with the largest reported decreases in ScienceQA (-3.28%) and MathVista (-0.7%).The authors describe reinforcement learning as improving region-selection and reasoning policies beyond VLIR-based imitation learning.Source: §4.4.3 · p. 8, Table 2 · p. 8
4.5.1 Impact of Region Grounding Accuracy on the Reasoning Chain
- Performance rises with grounding accuracy from 40% to 90% on ScienceQA, MathVista, and MMMU; the reported increase is greatest for ScienceQA.The study varies accuracy by randomly replacing or perturbing bounding boxes; MathVista rises from 47.9% to 70.4%, and MMMU from 43.8% to 62.2%.Source: §4.5.1 · pp. 8–9, Figure 4 · p. 9
4.5.2 Why is the Interleaved Reasoning Chain with Visual Region Localization Effective?
- Attention to the image diminishes during general textual reasoning, whereas VLM-R3 maintains attention to cropped regions and identifies the blue object as an umbrella.The example focuses on the area behind a person for closer examination.Source: §4.5.2 · pp. 9–10, Figure 5 · p. 9
4.6 Demonstrations for VLM-R³
- Qualitative demonstrations illustrate natural-image detail capture, document-text extraction, tracking multiple visual cues, and interleaved text-image reasoning.These are presented as examples of VLM-R3 capabilities.Source: §4.6 · p. 10, Figure 6 · p. 10
5 Conclusion
The conclusion presents VLM-R3 as a framework for dynamic visual reasoning that combines VLIR with R-GRPO. It reports strong benchmark results, particularly for fine-grained spatial reasoning and visual evidence integration.
5 Conclusion
- VLM-R3 combines VLIR and R-GRPO for dynamic visual reasoning, with reported state-of-the-art results across multiple benchmarks, particularly on fine-grained spatial reasoning and visual evidence integration.The conclusion describes adaptive focusing on relevant regions during multi-step inference as a direction for future visually grounded reasoning systems.Source: §5 · p. 10
A.1.1 Model Hyperparameter Settings
The appendix specifies the base model and image-pixel range used in experiments, alongside qualitative examples of document and multi-cue reasoning.
A.1.1 Model Hyperparameter Settings
- Experiments use Qwen2.5VL 7B with dynamic-resolution images constrained to 3,136–1,605,632 pixels.The pixel range is unified because bounding-box values depend on the number of input pixels.Source: §A.1.1 · pp. 10–11
- Figure 7 demonstrates extracting text details from a document image.Source: Figure 7 · p. 11
- Figures 8–9 illustrate iterative tracking of multiple visual cues and complex interleaved text-image CoT reasoning.Source: Figure 8 · p. 11, Figure 9 · p. 12
A.1.2 Zoom Scaling Rule
The zoom rule sets a crop’s scaling factor according to the selected bounding box’s area relative to the original image.
A.1.2 Zoom Scaling Rule
- The zoom factor is determined dynamically from the selected bounding box’s area ratio to the original image.The ratio is Abbox/Aorig.Source: §A.1.2 · p. 12
- Smaller selected regions receive stronger scaling, up to 2.0, while larger regions are scaled less or not at all.Source: §A.1.2 · p. 12
- Intermediate region sizes use a linear interpolation of the scaling factor.Source: §A.1.2 · p. 12
A.2 Training Setting for Supervised Fine-tuning Stage
The supervised fine-tuning stage used the complete VLIR dataset and trained only the LLM component.
A.2 Training Setting for Supervised Fine-tuning Stage
- Supervised fine-tuning used the complete VLIR dataset for 3 epochs.The vision encoder and MLP projector were frozen.Source: §A.2 · p. 12
- Training used a batch size of 2, gradient accumulation of 8, and a learning rate of 2 × 10−7.Experiments used four NVIDIA A100 GPUs with 80GB memory each and DeepSpeed[41].Source: §A.2 · p. 12
- Only the LLM component was trained; the vision encoder and MLP projector were frozen.Source: §A.2 · p. 12
A.3 Training Setting for R-GRPO Stage
The R-GRPO stage sampled training examples from five datasets and used rule-based reinforcement learning with exact-match answer checking.
A.3 Training Setting for R-GRPO Stage
- Approximately 5,000 examples were sampled from TextVQA, GQA, VSR, DocVQA, and M3CoT for R-GRPO.Source: §A.3 · p. 12
- The formulation used M = 5 and β = 0.0, eliminating the KL divergence constraint.Source: §A.3 · p. 13
- R-GRPO training ran for 300 steps with a learning rate of 1×10−6, batch size 1 per device, and gradient accumulation of 16.It used six NVIDIA A100 GPUs with 80GB memory each and DeepSpeed[41].Source: §A.3 · p. 13
- The rule-based reinforcement learning approach judged final-answer correctness using exact match.Source: §A.3 · p. 13
- The vision encoder and MLP projector were frozen during R-GRPO, and only the LLM component was trained.Source: §A.3 · p. 13
B Prompt Templates for VLIR Dataset Construction and Filtering
The VLIR construction prompts elicit crop-grounded reasoning, while filtering prompts screen cropped regions and reasoning for specified quality criteria.
B.1 Data Construction Prompts
- The first construction prompt asks the model to find helpful image regions, crop and zoom at least once, and continue reasoning after each crop.It requests bounding boxes in JSON and separates reasoning and final answer with <think> and <answer> tags.Source: §B.1 · pp. 13–14
- A second construction prompt supplies a bounding-box crop and asks for concise, fluent step-by-step reasoning that preserves the crop operation.It instructs the model to write as though it independently identified when to crop, although the box is provided to ensure the reasoning reaches the correct answer.Source: §B.1 · p. 14
- The example reasons over a dog’s tag by zooming in, then answers that it contains no black numbers or letters.Source: §B.1 · p. 14
B.2 Data Filtering Prompts
- The first filtering prompt accepts a crop only if it contains a recognizable object, text block, or specific diagram part.The model is asked to return yes or no directly.Source: §B.2 · p. 14
- The reasoning-quality prompt checks whether the logic is coherent, progresses toward the answer, and avoids false or redundant content.It returns no if any criterion is unmet.Source: §B.2 · p. 15