PaperlayerView exact revision on arXiv

Source-linked AI summary

GRIT: Teaching MLLMs to Think with Images

Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, Xin Eric Wang

arXiv:2505.15879v2cs.CVcs.AIcs.CL

TL;DR

Existing open-source visual reasoning models usually produce natural-language-only chains, leaving visual evidence insufficiently integrated. GRIT interleaves language with bounding boxes and trains this behavior with GRPO-GR, achieving grounded reasoning from only 20 triplets and unifying grounding with reasoning.

  • Problem

    Open-source vision reasoning models lack explicit visual information in their reasoning chains, while annotated grounded-reasoning data is scarce.

  • Method

    GRIT trains MLLMs to interleave natural-language reasoning with bounding-box coordinates using GRPO-GR rewards for answer accuracy and grounded-output format.

  • Results

    20 image–question–answer triplets suffice to train models that produce accurate, coherent grounded reasoning and unify grounding with reasoning abilities.

  • Takeaways & Limitations

    Grounded reasoning can be learned from existing triplets without dense reasoning-chain or explicit bounding-box annotations.

Abstract

from arXiv · show

Recent studies have demonstrated the efficacy of using Reinforcement Learning (RL) in building reasoning models that articulate chains of thoughts prior to producing final answers. However, despite ongoing advances that aim at enabling reasoning for vision-language tasks, existing open-source visual reasoning models typically generate reasoning content with pure natural language, lacking explicit integration of visual information. This limits their ability to produce clearly articulated and visually grounded reasoning chains. To this end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method for training MLLMs to think with images. GRIT introduces a grounded reasoning paradigm, in which models generate reasoning chains that interleave natural language and explicit bounding box coordinates. These coordinates point to regions of the input image that the model consults during its reasoning process. Additionally, GRIT is equipped with a reinforcement learning approach, GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused on the final answer accuracy and format of the grounded reasoning output, which eliminates the need for data with reasoning chain annotations or explicit bounding box labels. As a result, GRIT achieves exceptional data efficiency, requiring as few as 20 image-question-answer triplets from existing datasets. Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to produce coherent and visually grounded reasoning chains, showing a successful unification of reasoning and grounding abilities.

1 Introduction

GRIT addresses the lack of visually grounded reasoning in open-source vision models by interleaving language with image-region coordinates and training this behavior with reinforcement learning. Using only 20 image–question–answer triplets, it unifies grounding and reasoning while preserving broad task versatility.

  • Motivation: Open-source vision reasoning models generate natural-language-only chains, while visual reasoning annotations are scarce and reasoning paths are rarely unique.These limitations make visually grounded reasoning difficult to train and evaluate.
  • GRIT: GRIT interleaves natural-language reasoning with bounding boxes that identify the image regions consulted during reasoning.The resulting chain can be visualized as text interleaved with highlighted image regions.
  • GRPO-GR: GRPO-GR uses reinforcement learning rewards for reasoning and grounding format alongside answer accuracy, without requiring dense reasoning chains or box annotations.Training uses only image–question–answer triplets.
  • Results: 20 image–question–answer triplets train Qwen 2.5-VL and InternVL 3 on VSR and TallyQA data.The models retain broad versatility across visual question answering and grounding-heavy referring expression comprehension.
  • Results: GRIT-trained models unify grounding and reasoning abilities to produce accurate and coherent grounded reasoning.The evaluation covers multiple VQA and referring-expression-comprehension benchmarks.

2 Related Work

Prior vision-language reasoning methods use reinforcement learning, prompting, or auxiliary tools, but GRIT directly integrates visual grounding into the reasoning chain. This yields a grounded form of chain-of-thought reasoning with images.

  • Reinforcement-learning methods: Recent reinforcement-learning methods extend language-model reasoning to visual question answering but often treat visual grounding and text separately.Examples target diagram, mathematical, or symbolic reasoning tasks.
  • Prompted reasoning: Earlier vision-language chain-of-thought methods prompted visual cues or used auxiliary modules such as scene graphs.These approaches rely on prompting or external components rather than the GRIT paradigm.
  • GRIT: GRIT integrates visual information directly into the thought process, enabling models to reason with images rather than only about them.This grounded chain-of-thought form connects visual references to the reasoning sequence.

3 GRIT: Grounded Reasoning with Images and Text

GRIT represents reasoning as an interleaving of text and optional bounding boxes, then trains this behavior with GRPO-GR. The algorithm combines answer correctness with grounded-format rewards through group-relative policy optimization.

  • Grounded reasoning paradigm: GRIT generates a reasoning chain and final answer, allowing each reasoning step to emit either natural-language text or an optional bounding-box coordinate.Generated coordinates are intended to inform subsequent reasoning steps.
  • GRPO-GR training: GRPO-GR samples groups of completions and optimizes the policy using rewards that combine answer correctness with grounded-reasoning format.Group-normalized advantages guide the policy update.
  • Reward design: The composite task reward contains grounded-reasoning-format, optional grounded-target-counting, and GPT-aided answer-accuracy components.The counting reward applies to counting-related training examples.
  • Reward design: The grounded-reasoning-format reward checks special-token structure and syntactically valid bounding boxes within the reasoning chain.The special-token signal rewards correctly ordered think and rethink pairs, while the box signal rewards at least one valid box.
  • Reward design: The answer-accuracy reward combines a GPT-4o binary correctness judgment with sentence-level BLEU-1 similarity.Its formula is rans = sGPT + 0.1 sBLEU.
  • Optimization: The optimization objective balances maximizing group-relative reward with proximity to a reference policy through a KL-divergence penalty and trust-region constraint.This design promotes stable learning.

4 Experiments

GRIT is evaluated across grounding and reasoning tasks using multiple baselines and testing datasets. Despite training on only 20 triplets, it improves answer accuracy, grounding localization, cross-modal coherence, and subsequent visual attention, while data scaling exposes generalizability challenges.

  • 4.1 Setup: The evaluation covers six public datasets spanning spatial reasoning, counting, compositional questions, diverse visual tasks, mathematical reasoning, and open-vocabulary grounding.The testing sets are derived from VSR, TallyQA, GQA, MME, MathVista-mini, and OVDEval.
  • 4.1 Setup: GRIT-trained models are compared with Direct Query, Chain-of-Thought, one-shot ICL, and few-shot SFT baselines using answer accuracy and grounding IoU.The metrics separately assess natural-language answer correctness and bounding-box localization accuracy.
  • 4.2 Research question 1: How accurate is the grounded reasoning from both grounding and reasoning perspectives?: Despite training on only 20 samples, GRIT improves answer accuracy on VSR and TallyQA and generalizes effectively to out-of-domain testing data.The reported results indicate strong reasoning ability beyond the datasets seen during training.
  • 4.2 Research question 1: How accurate is the grounded reasoning from both grounding and reasoning perspectives?: GRIT-trained models outperform baselines in grounding IoU, including more accurate detection than zero-shot MLLMs on OVDEval.These results support a more successful integration of grounding and reasoning abilities.
  • 4.3 Research question 2: How generated bounding boxes interact with natural-language contents during grounded reasoning?: GRIT-trained models produce flexible reasoning strategies that ground relevant regions when needed and avoid erroneous grounding for nonexistent entities.Qualitative examples include answering before reflecting, grounding first and then reasoning, and declining to ground unsupported entities.
  • 4.3 Research question 2: How generated bounding boxes interact with natural-language contents during grounded reasoning?: GRIT-trained models show higher image-region–thought correlation than zero-shot ICL and few-shot SFT, although they remain below human-written reasoning chains.The comparison uses the Vision-Language Reasoning Cross-Modal Correlation metric and a 20-chain human baseline.
  • 4.3 Research question 2: How generated bounding boxes interact with natural-language contents during grounded reasoning?: Removing bounding boxes from pre-rethink segments provides a comparison condition for measuring their effect on subsequent reasoning attention.The analysis compares average visual-token attention during original and no-bounding-box rethink generation across 100 GQA samples and the first 50 generated tokens.

5 Conclusion

GRIT trains MLLMs to generate visually grounded reasoning by interleaving language with bounding boxes, using GRPO-GR without dense reasoning or box annotations. Experiments show accurate and coherent grounded reasoning with minimal data, while the study remains focused on smaller MLLMs and identifies data variety and pre-training as routes to better generalizability.

  • 5 Conclusion: GRIT interleaves natural language with bounding boxes and trains grounded reasoning with GRPO-GR, requiring neither dense reasoning chains nor explicit box annotations.The method is designed to instill complex grounded reasoning from minimal data.
  • 5 Conclusion: Experiments show that GRIT trains pre-trained MLLMs to produce accurate and coherent grounded reasoning with minimal data.The conclusion characterizes the method as extremely data efficient.
  • 5 Conclusion: The study focuses on smaller MLLMs to validate GRIT and its efficiency rather than explore peak performance through full-scale scaling.The authors identify data variety and model pre-training as directions for improving generalizability.

A Details for Testing Data

The evaluation uses curated data from six public benchmarks spanning visual reasoning, counting, compositional questions, mathematical reasoning, and object grounding. Most sets require short language answers, while OVDEval explicitly requires grounding as the answer.

  • Six public benchmarks provide testing data across spatial verification, counting, compositional spatial questions, diverse visual tasks, mathematical reasoning, and open-vocabulary grounding.
  • VSR evaluates spatial relation verification using manually filtered Visual CoT instances with unambiguous answers.
  • TallyQA evaluates counting with questions sampled uniformly across target object counts from 0 to 9.
  • GQA tests compositional object-spatial questions, while MME contributes counting, position, and existence subsets.
  • MathVista evaluates mathematical reasoning in visual contexts, whereas OVDEval tests open-vocabulary grounding from language queries to visual-feature coordinates.
  • VSR, TallyQA, and GQA include answer text and critical-region box annotations, but OVDEval explicitly requires grounding rather than treating it as optional.

B Details for Training data

GRIT demonstrates data efficiency with a 20-example training set combining spatial-relation and counting tasks. Ablation evidence shows that removing counting data and its reward harms out-of-domain answer accuracy.

  • 20 unique image-query-answer triplets form the small GRIT training dataset, sourced from existing VSR and TallyQA datasets.
  • The training set contains 10 VSR examples for localization and spatial relations plus 10 TallyQA examples for grounded counting.
  • The TallyQA examples uniformly cover object counts from 0 to 4 and support multi-step visual analysis linking grounding with logical deliberation.
  • Removing counting data and the grounded-target-counting reward decreases answer accuracy on out-of-domain data.

D Prompts

The experiments append a grounded-reasoning prompt to GRIT training and inference, while separate prompt formats support GPT-based answer scoring, rewards, and cross-modal correlation evaluation.

  • Figure 7 provides the prompt suffix appended during GRIT training and inference and for the Chain-of-Thought baseline.
  • Figure 8 defines the prompt format used for GPT-as-judge answer accuracy scores and GPT-aided answer-accuracy rewards.
  • Figure 9 specifies the Vision-Language Reasoning Cross-Modal Correlation prompt, comparing generated grounded-reasoning boxes with randomly sampled boxes using GPT-4o.

E More Examples

The examples illustrate grounded reasoning that uses bounding boxes, revisits ambiguity or an initial mistake, and produces correct final answers.

  • In the cow example, the model grounds both animals, considers ambiguity, and selects the cow as the correct answer.
  • The examples combine <think>, <rethink>, and <answer> stages with explicit box coordinates in the reasoning output.
  • The tennis example shows the model correcting an initially wrong one-person answer after recognizing two people with generated bounding boxes.
Loading 2505.15879v2…