PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking with Visual Grounding

Junkai Zhang, Yihe Deng, Kai-Wei Chang, Wei Wang

arXiv:2606.16122v1cs.AI

TL;DR

Vision-language reasoning traces often leave the image regions supporting their claims implicit, making them difficult to verify and supervise. This paper trains models to interleave reasoning with point or box groundings, improving counting and spatial reasoning while enabling 4B models to match or sometimes exceed 27B models on spatial benchmarks.

  • Problem

    Natural-language visual reasoning traces may identify claims without specifying their supporting image regions, limiting their verifiability and supervision.

  • Method

    The paper trains visually grounded thinking using a SAM3-based synthesis pipeline and grounding rewards for intermediate point or box references.

  • Results

    Visually grounded thinking substantially improves counting and spatial reasoning, with 4B models matching or sometimes exceeding the same family’s 27B model on spatial benchmarks.

  • Takeaways & Limitations

    The results support tying intermediate visual thoughts to the image regions that make their claims true.

  • Takeaways & Limitations

    Point grounding provides coarser feedback than box grounding, which may make its reward harder to optimize and may explain inconsistent accuracy gains.

Abstract

from arXiv · show

Visual thinking should not only sound right; it should show its evidence. While recent vision-language models (VLMs) can produce natural-language reasoning traces, these traces often leave the supporting image regions implicit, making them hard to verify and difficult to supervise. We introduce visually grounded thinking, a reasoning process in which models interleave natural-language thoughts with explicit point or box groundings of the visual evidence used at each step. This lets the model express intermediate reasoning in language while grounding key objects in the image regions they refer to. To train this behavior, we construct a scalable synthesis pipeline that distills correct visual reasoning traces, extracts the visual objects required by the traces, grounds them with a SAM3-based agent, and derives aligned point and box supervision from the resulting masks. We further propose grounding-aware reinforcement learning, which combines answer correctness rewards with dense grounding rewards that score whether generated object references match the correct image evidence. Across two counting benchmarks and four spatial reasoning benchmarks, adding visually grounded thinking to Gemma3-4B-IT consistently improves performance over the original model and the non-grounded thinking baseline. On spatial reasoning, the visually grounded thinking 4B models match, and in some cases surpass, Gemma3-27B-IT from the same model family. Our analysis shows that point grounding is well suited to counting, while box grounding benefits most from explicit grounding rewards on spatial tasks. Overall, our results show that VLMs think better when their intermediate thoughts are tied to the image regions that make them true.

1 Introduction

The paper introduces visually grounded thinking, which interleaves natural-language reasoning with point or box coordinates that identify the image evidence supporting each step. A SAM3-based synthesis pipeline and grounding-aware reinforcement learning train this behavior, improving counting and spatial reasoning.

  • Motivation: Pure language reasoning can describe visual evidence without identifying the corresponding image regions, making visual reasoning difficult to verify and supervise.Human visual reasoning instead links thoughts to concrete regions that guide attention and task-specific information extraction.
  • Method: Visually grounded thinking interleaves natural-language reasoning with coordinate-based point or bounding-box tags for important referenced objects.Language describes the thought, while coordinates specify the visual evidence supporting each reasoning step.
  • Training Data: A SAM3-based synthesis pipeline generates coordinate-annotated traces and high-fidelity object-mask supervision for both supervised fine-tuning and reinforcement learning.The pipeline addresses rollout variation by enabling grounding-level supervision even when models rename, reorder, skip, or add visual references.
  • Results: Visually grounded thinking improves counting and spatial reasoning, with point grounding especially effective for counting and grounding rewards particularly beneficial for box grounding on spatial tasks.On spatial benchmarks, 4B visually grounded models reach performance comparable to, and sometimes higher than, the same-family 27B model.
  • Training: Grounding-aware reinforcement learning jointly rewards answer correctness and whether intermediate visual references match the correct image evidence in point and box modes.Models are first cold-started on synthesized visually grounded traces before reinforcement learning explicitly supervises grounding quality.

2 Related Work

Related work progresses from using grounding to identify answer-relevant image regions toward interleaving visual coordinates with reasoning traces. This work follows that shift and adds an explicit reward for grounding quality during thinking.

  • Region Selection: Visual CoT uses intermediate bounding boxes to highlight key regions, while UV-CoT learns from preferences over model-generated regions to reduce human box annotations.These methods primarily use grounding to locate image regions needed for answering questions.
  • Grounded Reasoning: GCoT, Xia et al., and Argus represent later efforts that more tightly couple visual grounding with reasoning traces.
  • Grounding During Thinking: GRIT and ViGoRL use reinforcement learning to interleave natural language with visual coordinates, while VGR uses predicted regions for visual replay during inference.The present work extends this direction with an explicit grounding reward that directly scores visual grounding during thinking.

3 Data Synthesis Pipeline

The pipeline distills correct visual reasoning traces, extracts and grounds the objects they require with an iterative SAM3-based agent, and converts validated masks into aligned point- and box-grounded supervision. It produces 19,909 SFT traces with 107,613 grounding annotations over 72,381 distinct objects.

  • Overview: The pipeline uses held-out counting and spatial-reasoning datasets to synthesize correct reasoning traces with explicit visual-object grounding.Sources include TallyQA, Pixmo-Count, VSR, MultihopSpatial, and SpatialMQA.
  • Distilling visual thinking from VLMs: Examples are retained when Qwen3-VL-Plus or a second-pass Qwen3.5-Plus response matches the ground-truth answer.The second pass is applied to examples answered incorrectly initially.
  • Extracting groundable objects: An LLM extracts answer objects, alternatives, spatial anchors, counted instances, and relation endpoints, identifying each with a name and disambiguating context.Contexts distinguish visually or semantically similar instances using scene cues.
  • Agentic visual grounding: An iterative SAM3-centered agent proposes candidate masks, verifies them through image views, revises prompts when needed, selects valid masks, and removes unresolved objects.The agent cannot directly write coordinates; selected masks provide the shared supervision signal for both grounding modes.
  • Writing box and point supervision: RLE masks become normalized [0,1000] bounding boxes or interior points farthest from mask boundaries, then fill aligned object tags in box and point supervision.Coordinates are inserted from SAM3 outputs rather than generated by the annotation model, and malformed or substantially altered annotations are filtered.
  • Dataset Statistics: The synthetic pipeline produces 19,909 reasoning traces for SFT, containing 107,613 grounding annotations over 72,381 distinct grounded objects.These statistics summarize the resulting synthetic dataset.

4 Reinforcement Learning with Grounding Reward

The grounding reward matches generated grounding objects to ground-truth objects through contextual routing, then scores box regions with IoU or point-mask assignments with F1. These dense grounding scores are normalized and combined with sparse answer, format, and truncation rewards for GRPO training.

  • Grounding objects routing: A lightweight Qwen3.5-4B router matches generated grounding objects to ground-truth objects using names, nearby disambiguating context, and the image.The router receives each ground-truth object and the full list of generated grounding objects before returning the matched candidate.
  • Box grounding quality: Box grounding quality is the mean IoU between unions of generated and ground-truth boxes across supervised targets.Unmatched targets receive IoUi = 0, and each target contributes equally regardless of how many boxes it contains.
  • Point grounding quality: Point grounding quality uses one-to-one point-to-mask assignments, credits only points inside distinct target masks, and averages per-object F1 across supervised targets.Unmatched targets receive F1i = 0, while perfect point grounding scores 1.0.
  • Remarks: Point grounding is coarser than box grounding because F1 changes abruptly at mask boundaries, whereas box IoU changes smoothly with overlap.Moving a point within the same mask does not change the score, making the point reward potentially harder to optimize.
  • Remarks: Unmatched rollout grounding objects neither increase nor decrease grounding quality because the synthesized objects do not enumerate every reasonable visual cue.The method avoids penalizing additional evidence that may help answer the question.
  • Final reward: The final reward combines normalized dense grounding and sparse answer-correctness, thinking-format, grounding-format, and truncation terms for GRPO advantage estimation.The experiments use wans = 1.0, wground = 0.5, and wthink = wgfmt = 0.1.

5 Experiments

Experiments evaluate controlled Gemma3-4B-IT variants with non-grounded, box-grounded, and point-grounded thinking on counting and spatial reasoning benchmarks. Visually grounded thinking improves over the base and non-grounded baselines, with point grounding favored for counting and box rewards especially beneficial for spatial tasks.

  • Training: Models are trained from Gemma3-4B-IT using SFT followed by controlled variants with non-grounded, box-grounded, or point-grounded thinking.Training uses verl, SGLang, and FSDP2 on synthetic data.
  • Evaluation: Evaluation covers two counting benchmarks—TallyBench and CountQA—and four spatial benchmarks: VSR-zeroshot, EmbSpatial, SpatialMQA, and MultihopSpatial.Inference uses VLMEvalKit, SGLang, temperature 1.0, and four passes to reduce stochastic variance.
  • Main results: Visually grounded thinking substantially improves over Gemma3-4B-IT and can match or outperform Gemma3-27B-IT on spatial reasoning.The best 4B grounded models fall between Gemma3-12B-IT and Gemma3-27B-IT on VSR-zeroshot and EmbSpatial, and surpass Gemma3-27B-IT on SpatialMQA and MultihopSpatial.
  • Main results: Visually grounded variants strongly outperform non-grounded thinking, whose RL responses undergo length collapse and reduced exploration, while grounded rollouts remain more stable.Interleaved grounding tags and grounding-format rewards are hypothesized to provide additional local structure.
  • Grounding rewards: Grounding rewards improve box-mode average accuracy on all six benchmarks, with modest counting gains and more visible spatial-reasoning gains.Bounding boxes encode object identity and extent, making the reward especially useful when answers depend on fine-grained geometry.
  • Grounding interfaces: Point grounding consistently outperforms box grounding on counting, whereas the interfaces are broadly tied on spatial reasoning; point rewards yield less consistent downstream gains.Counting mainly requires instance-level localization, while boxes provide richer geometric cues for spatial relations without always improving accuracy.

6 Conclusion

The paper presents visually grounded thinking as a training recipe that interleaves natural-language reasoning with point or box groundings of supporting image regions. Its SAM3-based synthesis pipeline and grounding reward train VLMs to optimize answer correctness alongside accurate visual-object grounding.

  • Conclusion: Visually grounded thinking interleaves natural-language thoughts with point or box groundings of the image regions supporting each reasoning step.This makes the visual evidence behind intermediate reasoning explicit.
  • Conclusion: A scalable SAM3-based synthesis pipeline combined with reinforcement-learning grounding rewards trains VLMs to optimize answer correctness and accurate visual-object grounding.The training recipe jointly targets both outcome correctness and evidence grounding.

A Data Synthesis Details · A.1 Models Used in Each Pipeline Stage · A.2 Data Synthesis Prompt Details

The data synthesis pipeline uses several specialized vision-language models across its stages, while full prompt details are deferred to the source code because the prompts are lengthy.

  • A.1 Models Used in Each Pipeline Stage: Reasoning traces are distilled from Qwen3-VL-
  • A.1 Models Used in Each Pipeline Stage: Qwen3-VL-Plus and Qwen3.5-Plus are identified as models used for reasoning-trace distillation.
  • A.1 Models Used in Each Pipeline Stage: DeepSeek-V4-Flash extracts groundable objects in Stage 3 and annotates reasoning traces in Stage 6.
  • A.1 Models Used in Each Pipeline Stage: Qwen3.5-Flash powers the SAM3-based grounding-agent system in Stage 4.
  • A.1 Models Used in Each Pipeline Stage: Objects that fail to ground are retried sequentially with Qwen3.6-Plus and Gemini-3-Flash.
  • A.2 Data Synthesis Prompt Details: The prompts used in the data synthesis pipeline are lengthy.
  • A.2 Data Synthesis Prompt Details: Readers are referred to the source code for the full details of the data synthesis prompts.

A.3 Source Dataset Filtering

The source datasets were filtered for valid, compositionally informative counting and spatial examples, yielding 24,645 examples across five benchmarks.

  • A.3 Source Dataset Filtering: TallyQA retained AMT complex-counting data and imported VQA examples only with counts ≥4, compositional cues, and duplicate-image removal.Examples with missing or failed image downloads, invalid answer formats, or unparseable multiple-choice labels were skipped across all sources.
  • A.3 Source Dataset Filtering: VSR and MultihopSpatial training data were used after converting VSR captions into yes/no questions and removing the original Mu… bounding-box instruction.The supplied passage truncates the final dataset-specific instruction after “Mu…”.
  • A.3 Source Dataset Filtering: 24,645 source examples remained after filtering TallyQA, VSR, MultihopSpatial, SpatialMQA, and PixMo-Count.The totals were 7,197 TallyQA, 3,489 VSR, 6,791 MultihopSpatial, 4,316 SpatialMQA, and 2,852 PixMo-Count examples.

A.4 Final Data Composition by Source Dataset … B.1 SFT and RL Training Settings

The final dataset’s source composition is reported in Figure 6, while its grounding density is quantified across RL data and SFT traces. Training configurations for SFT and RL are presented in Tables 3 and 4.

  • A.4 Final Data Composition by Source Dataset: Figure 6 reports the final dataset composition by source dataset.
  • A.4 Final Data Composition by Source Dataset: The post-pipeline source-dataset distribution is shown in Figure 6.
  • A.5 Grounding Density Distribution: 19,909 paired rows comprise the final dataset.
  • A.5 Grounding Density Distribution: 72,381 grounded objects occur in the RL data, alongside 107,613 <obj> ... </obj> annotations in SFT traces.
  • A.5 Grounding Density Distribution: The dataset averages 3.64 grounded objects per row and 5.41 grounding annotations per row.
  • B.1 SFT and RL Training Settings: The higher SFT annotation density reflects repeated references to the same grounded objects during reasoning.
  • B Training and Evaluation Details: Training configurations for SFT and RL are presented in Tables 3 and 4.

B.2 Evaluation Settings · B.3 System Prompts

The evaluation setup is summarized in Table 5, while system prompts define answer formatting and distinct non-grounded, box-grounded, and point-grounded reasoning protocols. Grounded protocols normalize coordinates to [0, 1000] and require critical visual objects to be referenced with boxes or points inside the image objects.

  • B.3 System Prompts: The instruct-model prompt asks the assistant to provide a helpful response and place its final answer inside \boxed{}.
  • B.3 System Prompts: The non-grounded prompt requires step-by-step image reasoning in <think>...</think> tags followed by a boxed final answer.
  • B.3 System Prompts: The box-grounded prompt describes a visual reasoning assistant with precise spatial grounding ability.
  • B.2 Evaluation Settings: Evaluation configurations for visual-spatial and counting benchmarks are presented in Table 5.
  • B.3 System Prompts: Box coordinates use [0, 1000] normalization, with [0, 0] at the top-left and [1000, 1000] at the bottom-right.
  • B.3 System Prompts: Box grounding uses <obj> tags containing a descriptive phrase and tightly enclosing box coordinates, with one box per referenced instance.
  • B.3 System Prompts: The box prompt restricts grounding to objects critical for justification, permits multiple boxes for shared phrases, and omits tags for non-groundable questions.
  • B.3 System Prompts: The point-grounded prompt normalizes point coordinates to [0, 1000] and requires each point to lie inside its corresponding object.

B.4 Experiment Cost

The paper’s training and evaluation require approximately 400 H200 GPU hours.

  • B.4 Experiment Cost: 400 H200 GPU hours are required for training and evaluation.
Loading 2606.16122v1…