VGR: Visual Grounded Reasoning

Jiacong Wang; Zijian Kang; Haochen Wang; Haiyong Jiang; Jiawen Li; Bohong Wu; Ya Wang; Jiao Ran; Xiao Liang; Chao Feng; Jun Xiao

Source-linked AI summary

VGR: Visual Grounded Reasoning

Jiacong Wang, Zijian Kang, Haochen Wang, Haiyong Jiang, Jiawen Li, Bohong Wu, Ya Wang, Jiao Ran, Xiao Liang, Chao Feng, Jun Xiao

arXiv:2506.11991v3Jun 13, 2025cs.CVcs.AIcs.CL

TL;DR

Multimodal reasoning often relies on language space and suffers language bias in perception-heavy tasks. VGR adds demand-driven visual-region replay and trains VGR-SFT with grounded reasoning data, improving multiple benchmarks while using 0.3× visual tokens. The method is evaluated primarily on LLaVA-based architectures, with stronger encoders, language models, and reinforcement learning left for future work.

Problem
Language bias in language-centered multimodal reasoning produces systematic performance drops on perception-heavy tasks.
Method
VGR detects question-relevant regions, replays their visual tokens during reasoning, and trains on VGR-SFT data combining visual grounding with language deduction.
Results
VGR outperforms the baseline across multiple benchmarks, including +6.4 on MMStar and +14.1 on ChartQA, while using 0.3× visual tokens.
Takeaways & Limitations
Selective visual attention and grounded reasoning data support fine-grained multimodal comprehension with lower visual-token usage.
Takeaways & Limitations
VGR is currently implemented on the LLaVA architecture; stronger visual encoders, language models, and reinforcement learning remain future directions.

Abstract

from arXiv · show

In the field of multimodal chain-of-thought (CoT) reasoning, existing approaches predominantly rely on reasoning on pure language space, which inherently suffers from language bias and is largely confined to math or science domains. This narrow focus limits their ability to handle complex visual reasoning tasks that demand comprehensive understanding of image details. To address these limitations, this paper introduces VGR, a novel reasoning multimodal large language model (MLLM) with enhanced fine-grained visual perception capabilities. Unlike traditional MLLMs that answer the question or reasoning solely on the language space, our VGR first detects relevant regions that may help to solve problems, and then provides precise answers based on replayed image regions. To achieve this, we conduct a large-scale SFT dataset called VGR -SFT that contains reasoning data with mixed vision grounding and language deduction. The inference pipeline of VGR allows the model to choose bounding boxes for visual reference and a replay stage is introduced to integrates the corresponding regions into the reasoning process, enhancing multimodel comprehension. Experiments on the LLaVA-NeXT-7B baseline show that VGR achieves superior performance on multi-modal benchmarks requiring comprehensive image detail understanding. Compared to the baseline, VGR uses only 30\% of the image token count while delivering scores of +4.1 on MMStar, +7.1 on AI2D, and a +12.9 improvement on ChartQA.

1 INTRODUCTION

VGR addresses language bias in multimodal reasoning by adding targeted visual analysis and replayed visual memory to the reasoning process. Its dataset and framework improve benchmark performance while using fewer image tokens.

Language bias causes systematic performance drops in perception-heavy multimodal tasks.
VGR extends text-only chain-of-thought with targeted visual analysis of question-relevant regions.
The visual memory replay module retrieves high-resolution visual tokens when the model emits a replay signal during reasoning.
VGR-SFT trains models to attend autonomously to arbitrary visual regions rather than relying on text-only reasoning or rigid interactions.
+6.4 on MMStar and +14.1 on ChartQA are reported over the baseline while using 0.3× visual tokens.
VGR is introduced as a visual reasoning framework that dynamically attends to visual content and outperforms LLaVA-NeXT while using 0.3× image tokens.

2 RELATED WORKS

Related work extends language-model reasoning into multimodal systems, but perception-heavy tasks expose language bias. Existing approaches emphasize mathematical and scientific reasoning, visual grounding, or supervised high-resolution grounding.

Flamingo, BLIP-2, and LLaVA established major architectures for cross-modal understanding and scalable instruction tuning.
Prior multimodal reasoning studies report performance degradation on perception-heavy tasks from accumulated language bias.
Vision-R1 uses image captioning and DeepSeek-R1 to create cold-initialization data, while VLM-R1 and Visual-RFT apply GRPO to visual grounding.

3 VISUAL GROUNDED REASONING

VGR builds a visual memory from high-resolution image features and replays selected regions during reasoning. Its control signal, pooling strategy, and detection training support selective visual grounding with reduced token use.

3 VISUAL GROUNDED REASONING: VGR introduces visual memory replay so the model can attend to arbitrary image regions during reasoning.
3 VISUAL GROUNDED REASONING: AnyRes partitions resized images into patches whose encoded features are concatenated into a unified visual memory.
3 VISUAL GROUNDED REASONING: VGR expands AnyRes crop capacity from 4 to 16 patches and uses 2 × 2 and 4 × 4 pooling to balance resolution and computation.
3 VISUAL GROUNDED REASONING: 70% lower token usage accompanies VGR’s reduction to 144 snapshot tokens and at most 720 high-resolution crop tokens, while supported resolutions expand 5×.
3 VISUAL GROUNDED REASONING: The replay signal <sot>[x1, y1, x2, y2]<eot> specifies a region whose visual tokens are retrieved on demand.
3 VISUAL GROUNDED REASONING: During inference, valid coordinates are mapped to a feature patch, down-sampled with 2 × 2 pooling, and appended as a token sequence.
3 VISUAL GROUNDED REASONING: Training appends retrieved image tokens after the replay signal, supervises signal and text tokens, and excludes image tokens from the loss.
3 VISUAL GROUNDED REASONING: The detection objective combines L1 and GIoU losses, with a small MLP mapping the replay-token hidden state to a four-dimensional box.

4 VISUAL REASONING DATA CURATION

VGR-SFT is constructed through cold-start annotation, rejection-based verification, annotator-model expansion, and refinement. The resulting data combines reasoning with explicit visual-region references.

4 VISUAL REASONING DATA CURATION: The three-stage pipeline generates cold-start data, trains an annotation model for broader coverage, and refines accepted examples.
4 VISUAL REASONING DATA CURATION: Training examples combine questions, reasoning chains, final answers, and references to image regions relevant to those answers.
4 VISUAL REASONING DATA CURATION: Reject sampling verifies answer formatting and bounding-box JSON before correctness checks discard or rewrite incorrect examples.
4 VISUAL REASONING DATA CURATION: A smaller 14B annotator model addresses cold-start annotation speed and rejection-rate limitations while expanding data across domains.
4 VISUAL REASONING DATA CURATION: Refinement revises accepted reasoning chains to match the predefined template and remove ambiguous or redundant content.

5 EXPERIMENTS

Experiments evaluate VGR against vision-language baselines, backbones, reasoning-data formulations, replay strategies, and token-scaling choices. Across these studies, visual grounding and replay improve fine-grained reasoning while reducing visual-token costs.

Comparison with Existing Methods: VGR outperforms strong vision-language alternatives on benchmarks requiring fine-grained high-resolution image comprehension, using 0.3× the visual tokens of LLaVA-NeXT.The comparison includes MMStar, ChartQA, DocVQA, TextVQA, InfoQA, AI2D, RealWorldQA, and POPE.
Ablations on Backbones: VGR’s complete architecture is evaluated against varied visual encoders and language-model backbones under the same dataset and fairness constraints.The baseline configuration uses Vicuna-7B-v1.5 with CLIP-ViT-L/14@336, matching LLaVA-NeXT-7B.
Ablations on Data Formulations: Removing either visual memory or reasoning consistently degrades performance, showing that grounding and reasoning make complementary contributions.The data-formulation ablation compares variants without memory and without reasoning against the complete system.
Ablations on Public CoT Data: Region-of-interest reasoning data improves overall performance, whereas directly adopting vanilla multimodal CoT datasets can underperform the baseline.The comparison contrasts VGR’s explicitly grounded reasoning data with LLaVA-CoT and MMPR.
Ablations on Replay: Dynamic visual memory replay provides a critical improvement by inserting features from selected regions into the language-model input.Merely predicting regions without replaying their image features yields limited gains.
Ablations on Data and Replay Strategies: Condensed rewritten reasoning data produces stronger grounded reasoning than raw annotated data, whose longer contexts make the model more error-prone.The study also examines detection loss and pooling choices; visual memory achieves comparable replay performance with fewer image tokens and lower training and inference costs.
Test-Time Token Scaling: Scaling replay tokens at test time further helps, especially on OCR-related tasks.Testing uses a larger cropping scheme while keeping the pooling strategy unchanged.

6 CONCLUSION

VGR combines selective visual feature replay with a grounded reasoning dataset to improve multimodal comprehension. Experiments report improvements across multiple benchmarks, while the authors identify stronger backbones and reinforcement learning as future directions.

Conclusion: VGR selectively attends to informative visual regions during reasoning through feature replay and the VGR-SFT dataset, targeting fine-grained comprehension of small regions in high-resolution inputs.The dataset integrates visual information into dense reasoning tasks.
Conclusion: VGR is currently implemented on the LLaVA architecture, and the authors suggest exploring stronger visual encoders, LLMs, and reinforcement learning.These are presented as directions for future research.

ETHICS STATEMENT

The study emphasizes responsible data use by relying on publicly available, established computer-vision datasets under their licenses and intended academic purpose.

Ethics Statement: All datasets are publicly available and established in computer vision, and their use follows the provided licenses and intended academic purpose.Benchmarking was conducted on LLaVA data.

REPRODUCIBILITY STATEMENT

The authors support reproducibility by documenting implementation details and planning to release the research artifacts publicly upon acceptance.

Reproducibility Statement: Appendix materials provide training procedures and hyperparameters, with source code, datasets, and trained checkpoints planned for public release upon acceptance.The statement frames these releases as support for reproducibility.

A MORE ABLATION EXPERIMENTS ANALYSIS OF VGR IN THE MAIN TEXT

Ablations show that VGR’s complete configuration performs best, while reasoning data, grounding, and visual memory replay each contribute beyond the LLaVA-NeXT-7B baseline. Scaling VGR-SFT improves benchmark performance, and feature retrieval substantially reduces inference time versus ZoomEye.

The complete VGR configuration achieves the optimal results in Table 2.
Removing either grounding or reasoning still surpasses the LLaVA-NeXT-7B baseline, supporting the contribution of both components.
Ablations show that removing detection loss, visual memory replay, or reasoning data still exceeds the baseline, validating the remaining components.
VGR achieves 7.5s and 9.2s average inference times on V* Bench and HR-Bench 8K, versus ZoomEye’s 48.46s and 55.52s.The comparison attributes the efficiency advantage to VGR’s feature retrieval-based design over recursive search.

B.1 ABLATIONS ON DIFFERENT POOLING STRATEGIES.

The supplementary analyses examine pooling, replay accuracy, grounding, generalization, and inference cost. VGR preserves visual detail with reduced tokens, but longer reasoning outputs and visual replay increase latency.

Replay accuracy: Incorrect visual features substantially degrade performance, with further drops when both text and visual replay features are incorrect.This supports the importance of accurate replay predictions for reasoning.
Grounding performance: VGR training consistently improves referring-expression comprehension on RefCOCO, RefCOCOg, and RefCOCO+ under both accuracy and IoU metrics.
Generalization and scope: VGR is evaluated primarily against LLaVA-NeXT-7B because reproducing data-closed InternVL3 and Qwen2.5-VL models from scratch is infeasible.The authors nevertheless report substantial improvements on comparable InternViT and Qwen2.5 LLM baselines.
Inference cost: VGR’s inference latency increases because it performs visual feature replay and generates longer reasoning outputs.The reported inference behavior and cost are summarized across tasks in Table 15.
Data construction: The data pipeline combines cold-start generation, annotation, reject sampling, rewriting, and VGR-SFT construction to produce structured visual reasoning data.The training prompt requires bounding-box references in [x1,x2,y1,y2] format and a final answer marker.

D.2 VISUALIZATION OF DATA AND VGR

The data-curation visualization illustrates progressive refinement from source examples to training data, while VGR visualizations show region localization followed by grounded reasoning and answers.

Data visualization: The curation pipeline improves data quality step by step: annotation increases reasoning complexity and efficiency, while rewriting makes training data more concise.
VGR visualization: On MMStar and ChartQA, VGR locates target regions, reasons over their contents, and produces accurate answers.

E FURTHER COMPARISON AND INSIGHTS FROM RELATED WORKS WITH VGR.

VGR differs from related approaches through autonomous visual-memory replay, an open end-to-end training pipeline, and efficient precomputed visual features. These design choices support flexible visual reasoning without repeated crop processing.

Native free-form visual reasoning: VGR dynamically retrieves and reuses visual memory during reasoning rather than relying on Visual CoT’s predefined multiturn interaction format.
Open solution: VGR provides an open end-to-end pipeline covering data construction, training, and joint detection and autoregressive supervision.
Efficient visual memory: VGR precomputes visual tokens once and uses multilevel pooling, saving approximately 70% of tokens compared with baseline methods.
Comparison with related work: Unlike CogCom’s predefined retrieval operations, VGR learns memory retrieval during reasoning and can analyze, reflect, and plan from retrieved information.
Comparison with related work: Compared with Chain-of-Focus, VGR learns from scratch and adds detection-loss supervision to improve grounding and retrieval quality.