VGR: Visual Grounded Reasoning

1 INTRODUCTION

• VGR counters language bias by extending text-only chain-of-thought with targeted visual analysis of question-relevant regions during reasoning.It can selectively retrieve visual memory on demand to improve multimodal reasoning accuracy and interpretability.Source: §1 · pp. 1–2
• A model-generated replay signal retrieves corresponding visual tokens from memory, which is built from high-resolution crops and pooled for efficiency.The paper describes the replay module as self-driven and says the overall design is more efficient than conventional MLLMs.Source: §1 · p. 2
• VGR-SFT teaches models to attend autonomously to arbitrary visual regions, using model-generated grounding rather than manual annotations or rigid interactions.The dataset is built from a cold-start set, refined through rejection sampling, and expanded with a custom-trained annotation model.Source: §1 · p. 2
• With 0.3× visual tokens, VGR improves over the baseline by +6.4 on MMStar and +14.1 on ChartQA.The contribution summary identifies LLaVA-NeXT as the baseline and says the performance gap grows as image-token quantity increases.Source: §1 · p. 2

Multimodal Large Language Models

• LLaVA’s modular vision-encoder and language-model framework, combined through a linear projection and instruction tuning, built on foundational cross-attention models such as Flamingo and BLIP-2.Later models extend efficiency, scalability, and task complexity; Qwen2.5-VL uses dynamic resolution, while InternVL3 emphasizes large-scale native multimodal pretraining.Source: §2 · p. 2

Reasoning MLLMs

• CoT prompting can degrade perception-heavy tasks through accumulated language bias, while reasoning MLLMs mainly target difficult math and science problems with image inputs.Vision-R1 uses generated image captions to create cold-initialization data, and VLM-R1 and Visual-RFT apply GRPO to open-ended visual grounding, where RL consistently outperforms SFT.Source: §2 · pp. 2–3
• Zoomeye’s tree-based zooming and Chain-of-Spot’s interactive spot search use supervised CoT to improve high-resolution visual grounding and perception-intensive reasoning.These approaches report gains on HR-Bench and V* Bench; VGR offers an alternative that trains autonomous selection of task-relevant regions and contextual reasoning from grounded observations.Source: §2 · p. 3

• VGR builds a visual memory by encoding AnyRes image patches into language-space tokens and concatenating patch features for later retrieval.Images are resized to dimensions divisible by 336, partitioned into non-overlapping 336 × 336 patches, and encoded by the vision encoder and adapter.Source: §3 · p. 3
• VGR expands AnyRes crops from a maximum of 4 to 16 patches, then compresses snapshot features with 2 × 2 pooling and high-resolution crop tokens with 4 × 4 pooling.This expand-then-compress strategy balances fine-grained retrieval features with training and inference efficiency.Source: §3 · p. 4
• VGR uses 144 snapshot tokens and at most 720 high-resolution crop tokens versus the baseline’s maximum 2,880, reducing token usage by 70% while supporting 5× higher resolutions.The design aims to preserve fine-grained visual detail for retrieval while lowering computational overhead.Source: §3 · p. 4
• At inference, VGR detects a replay signal containing region coordinates, extracts the corresponding feature patch, downsamples it with 2 × 2 pooling, and appends its tokens to the LLM input.The grounding area notation is <sot>[x1, y1, x2, y2]<eot>, with coordinates marking the region’s top-left and bottom-right corners.Source: §3 · p. 4
• Training applies cross-entropy to text and replay-signal tokens, excludes image tokens from the loss, and adds an auxiliary box-detection loss combining ℓ1 and GIoU.A small MLP maps the hidden state of <eot> to a 4-dimensional box; the detection loss supports continuous coordinate regression.Source: §3 · p. 4

4 VISUAL REASONING DATA CURATION

• The three-stage pipeline generates cold-start data with a large MLLM, applies reject sampling and refinement, then trains an annotator to scale data across domains.The pipeline overview is shown in Figure 2.Source: §4 · p. 4, Figure 2 · p. 5

4.1 COLD-START WITH INSTRUCTION MODEL

• A 72B MLLM (Bai et al., 2025) generates reasoning and answers while identifying relevant image regions, referencing them, and encoding their boxes and labels in JSON.The annotated regions become replayed areas during training.Source: §4.1 · p. 5

4.2 REJECT SAMPLING

• Reject sampling checks answer and JSON parseability, verifies answer correctness, and validates that replay boxes align with their semantic labels.Incorrect answers are discarded or rewritten, and replay boxes are intentionally expanded to preserve contextual information.Source: §4.2 · p. 5

4.3 DATA SCALING WITH ANNOTATION MODEL

• The 14B annotation model raised the data pass rate from 14% to 40% and accelerated annotation by 3.2×, enabling lower-cost data scaling.It was trained on cold-start data that passed rejection and augmented with Open-R1 distilled reasoning data.Source: §4.3 · p. 6

4.3.1 TRAINING DATA

• A final MLLM refinement revises reasoning for robustness and removes ambiguous or redundant content; VGR-SFT uses LLaVA’s official training data to align with the baseline’s data comparison.The refined examples follow a predefined template and are used to train the final reasoning model.Source: §4.3.1 · p. 6

5.1 EXPERIMENT SETTINGS

• Experiments use LLaVA-NeXT with CLIP-ViT-L/14@336 and Vicuna backbones, training on LLaVA-558K plus LLaVA-NeXT-770K and 158K self-constructed examples derived from original SFT data.For Vicuna-7B, the pretraining and fine-tuning learning rates are 1e-5 and 2e-5, respectively; the ViT learning rate is one-tenth of the base rate.Source: §5.1 · p. 6

5.2 COMPARISON WITH EXISTING METHODS

• With 0.3 × the visual tokens of original LLaVA-NeXT, VGR consistently outperforms strong alternatives, especially on benchmarks requiring fine-grained high-resolution image comprehension.The paper reports that the performance gap grows as image-token counts increase, suggesting region-focused processing is more effective than simply increasing visual tokens.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 ABLATION STUDIES

• VGR yields consistent improvements across varied visual encoders and language-model backbones, including on V* Bench and HR-Bench 8K, which require visual grounding.The experiments keep the dataset fixed and change only the LLM and ViT backbones; the Vicuna-7B and CLIP-ViT-L/14@336 configuration matches the LLaVA-NeXT-7B baseline.Source: §5.3 · p. 7, Table 3 · p. 7
• Removing either visual memory grounding or reasoning degrades performance, indicating that both components contribute and complement one another.The ablation compares variants without memory and without reasoning against the full data formulation.Source: §5.3 · p. 7, Table 4 · p. 8
• Region-of-interest reasoning data improves overall performance, while directly adopting LLaVA-CoT or MMPR data performs worse than the baseline.The paper attributes the contrast to language bias in complex reasoning data and the value of integrating relevant visual features.Source: §5.3 · p. 8, Table 5 · p. 8
• Detection loss supports precise coordinate regression, while omitting dynamic visual memory replay limits gains by leaving predicted regions’ image features out of the reasoning input.The detection-loss ablation motivates continuous localization for normalized coordinates; replaying boundary-region features lets the model use fine-grained visual detail.Source: §5.3 · p. 8, Table 6 · p. 9
• Condensing annotated reasoning into shorter rewritten data reduces confusion and supports stronger grounded reasoning, whereas raw annotated data creates longer contexts prone to mistakes in general question answering.Source: §5.3 · p. 8, Table 6 · p. 9
• VGR uses 2×2 pooling for base and replay features and 4×4 for local crops; test-time increases in crop tokens further help, especially on OCR tasks.Visual-memory replay achieves comparable performance to recropping boxes while reducing training and inference costs through fewer image tokens; test-time scaling uses a maximum of 64 crops, though most fine-tuning images do not reach that resolution.Source: §5.3 · pp. 8–9, Table 7 · p. 9, §5.4 · p. 9

Conclusion

• VGR uses selective feature replay and the VGR-SFT dataset to support visual reasoning and improve fine-grained comprehension, especially for small regions in high-resolution inputs.The authors report considerable improvements across multiple benchmarks.Source: §6 · p. 9

Discussion

• VGR is currently implemented on the LLaVA architecture (Liu et al., 2023b); future work could explore stronger visual encoders and LLMs or integrate reinforcement learning.The authors suggest RL could enable a more generalized and diverse reasoning process.Source: §6 · p. 9

• The authors state that all datasets are publicly available and established in computer vision, and that LLaVA data use follows its licenses and intended academic purpose.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The appendix provides model and experiment implementation details, including training procedures and hyperparameters.Source: REPRODUCIBILITY STATEMENT · p. 10
• Upon acceptance, the authors say they will publicly release the source code, datasets, and trained model checkpoints.Source: REPRODUCIBILITY STATEMENT · p. 10

A MORE ABLATION EXPERIMENTS ANALYSIS OF VGR IN THE MAIN TEXT

• Removing grounding cues, reasoning, detection loss, visual memory replay, or reasoning data still beats LLaVA-NeXT-7B, while full VGR achieves optimal results.The ablations support the contributions of reasoning data, grounding boxes, detection loss, and visual memory replay.Source: A MORE ABLATION EXPERIMENTS ANALYSIS… · p. 14
• Training follows LLaVA-NeXT settings for one epoch, with learning rates of 1e-5 for pre-training and 2e-5 for Vicuna-7B fine-tuning.The ViT learning rate is one-tenth of the base rate; warmup ratio is 0.03, with a cosine scheduler and zero weight decay.Source: A MORE ABLATION EXPERIMENTS ANALYSIS… · p. 14

A.1 PERFORMANCE COMPARISON OF VGR-SFT.

• With 25K training data, VGR using Qwen2.5-VL-72B-annotated cold-start data and 14B-annotated data achieves similar performance.The authors present this as evidence for scaling with a smaller annotator model.Source: §A.1 · p. 14, Table 9 · p. 14
• 9x more efficient: annotator-based models were 3.2x faster at inference and 2.9x higher in accuracy, raising pass rate from 14% to 40% versus the Qwen2.5-VL-72B cold-start baseline.The 14B annotator model enabled scaling data with limited resources.Source: Table 9 · p. 14, §A.1 · p. 14
• VGR-SFT trained on 158K data clearly outperforms models trained on 25K data across multiple benchmarks.Cold-start data scaling is constrained by slow annotation and a high reject rate.Source: Table 9 · p. 14, §A.1 · p. 14

A.2 COMPARISON WITH ZOOMEYE.

• ZoomEye uses recursive hierarchical patch splitting and node exploration, whereas VGR selectively retrieves relevant visual tokens from a preconstructed feature pool.The paper characterizes ZoomEye’s repeated splitting and traversal as computationally costly, especially for high-resolution images requiring deep recursion.Source: §A.2 · p. 15
• On V* Bench and HR-Bench 8K, ZoomEye averages 48.46s and 55.52s per question, respectively, while VGR averages 7.5s and 9.2s, demonstrating faster inference.Source: §A.2 · p. 15

B.1 ABLATIONS ON DIFFERENT POOLING STRATEGIES.

• Different pooling strides are important: the method uses 2×2 for base and replayed image features and 4×4 for local image features; Table 11 reports original-data results above the horizontal line.Source: Table 11 · p. 15

B.2 HOW INACCURATE BOUNDING BOX REPLAY IMPACTS REASONING.

• Incorrect replayed visual features substantially degrade reasoning, with further drops when both box text and visual features are incorrect.The first setting pairs a randomly cropped image tensor with the correct box token; the second replaces both with a random box. The results underscore the importance of accurate replay predictions.Source: §B.2 · pp. 15–16, Table 12 · p. 16

B.3 THE RELATIONSHIP BETWEEN REPLAY AND GROUNDING

• VGR consistently improves RefCOCO, RefCOCOg, and RefCOCO+ referring-expression comprehension over LLaVA-Next on acc@0.5 and IoU.The evaluation reports both the standard IoU>0.5 accuracy and the IoU metric.Source: Table 13 · p. 16, §B.3 · p. 16

B.4 CONDUCT VGR ON OTHER BACKBONES.

• VGR-style post-training improves InternVL3-8B on DocVQA, ChartQA, and TextVQA, indicating transfer beyond the original architecture.This was an analysis-oriented post-training probe using LLaVA-Next-770K data, not the full multi-stage VGR pipeline.Source: Table 14 · p. 16, §B.4 · p. 16

B.5 INFERENCE BEHAVIOR AND COST WITH VGR.

• VGR inference takes longer because it generates longer reasoning outputs and replays visual features, unlike the baseline’s short-answer generation.Inference times were measured on H20 GPUs using Lmms-eval; the paper notes that frameworks such as vLLM and sglang may improve efficiency.Source: Table 15 · p. 17, §B.5 · p. 17
• 97.5%–98.9% of predicted bounding boxes are valid across downstream benchmarks, and absolute pixel coordinates perform slightly worse than normalized coordinates.The reported coordinate comparison uses an absolute-pixel variant against the original normalized (0–1) setting.Source: §B.5 · p. 17, Table 16 · p. 17

D REASONING DATA PIPELINE

• Reject sampling uses two MLLM verification steps, then filters noisy data by parsing scores and applying a threshold of 3.The implementation uses Doubao1.5-VL with its default settings; the model checks open-ended-answer correctness and grounding areas, and is limited to well-defined tasks because it is costly and closed-source.Source: §D.1 · pp. 17–18

C THE CASE OF VGR

• Figure 4 presents three formulations of VGR-SFT data that are processed separately and then used to train VGR.The paper also points to a case study illustrating how VGR’s design contributes to visual chain-of-thought reasoning.Source: C THE CASE OF VGR · p. 17, Figure 4 · p. 18

D.2 VISUALIZATION OF DATA AND VGR

• The data pipeline improves examples step by step: annotation increases reasoning complexity and efficiency, while rewriting makes training data more concise and easier to learn.The annotation model generalizes from text-only to visual reasoning after training on cold-start data and complex reasoning data from DeepSeek-R1Face (2025); Guo et al. (2025a), but still makes mistakes.Source: §D.2 · p. 19
• Reject sampling expands bounding boxes, aligns them with ground truth, and enriches context; rewriting removes duplicate boxes, reformats the document, and clarifies explanations.The passage says short, clean data performs better than long data in Table 6c and is especially valuable for smaller models such as Vicuna, which supports only 4096 tokens.Source: §D.2 · p. 19
• VGR accurately locates target regions in MMStar and ChartQA responses, reasons from their contents, and provides accurate answers.The visualizations show responses from VGR trained with the authors’ data.Source: §D.2 · p. 19

Native Free-form Visual Reasoning

• VGR supports flexible visual re-reading at any reasoning step by dynamically replaying visual memory, unlike Visual CoT’s predefined multi-turn, one-time crop and static path (Shao et al., 2024a).VGR embeds grounding in the model, avoiding manual cropping and re-input of relevant visual features and text.Source: E FURTHER COMPARISON AND INSIGHTS FROM… · p. 20

Full Open Solution with Data and Training

• VGR offers an open-source data-and-training pipeline with joint detection and autoregressive losses, enabling visual reasoning from a VLM without inherent visual capabilities, unlike SketchPad and Refocus.Its end-to-end data construction pipeline provides a followable framework for data iteration.Source: E FURTHER COMPARISON AND INSIGHTS FROM… · p. 20

Efficient Visual Memory Architecture

• VGR precomputes visual tokens once and uses multi-level pooling to allocate computation to critical regions, saving approximately 70% of tokens versus baseline methods while achieving superior performance.Unlike tool-based methods including Visual CoT, SketchPad, and Refocus, and the grounding-then-crop approaches of CogCom and Chain-of-Focus, VGR avoids repeatedly processing crops or modified views.Source: E FURTHER COMPARISON AND INSIGHTS FROM… · p. 20

Comparison with CogCom

• Unlike CogCom (Qi et al., 2024), which retrieves information with predefined operations but does not reason over it, VGR learns retrieval during reasoning and can analyze, reflect, and plan follow-up operations.The paper characterizes this as more flexible for complex quests.Source: E FURTHER COMPARISON AND INSIGHTS FROM… · p. 20

Comparison with Chain-of-Focus

• Unlike Chain-of-Focus (Zhang et al., 2025), VGR can learn from scratch with an LLM lacking vision prior and uses scalable reasoning data plus detection-loss supervision to improve grounding and retrieval.The paper notes that many VGR backbones were not well-trained on CoT, reasoning, or grounding.Source: E FURTHER COMPARISON AND INSIGHTS FROM… · p. 20