PaperlayerView exact revision on arXiv

Source-linked AI summary

Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization

Sai Srinivas Kancheti, Aditya Kanade, Rohit Sinha, Vineeth N Balasubramanian, Tanuja Ganu

arXiv:2604.08476v1cs.CVcs.AI

TL;DR

RLVR can raise visual reasoning accuracy while leaving CoT traces inconsistent with answers or poorly grounded in images. The paper introduces FGRPO, which constrains consistency and grounding during GRPO optimization, and reports improved reasoning quality alongside higher answer accuracy across seven spatial benchmarks.

  • Problem

    RLVR-trained multimodal reasoning models can produce accurate answers with CoT traces that are logically inconsistent or visually ungrounded, making accuracy alone an incomplete measure of reasoning ability.

  • Method

    FGRPO defines verifiable consistency and grounding rewards and enforces minimum thresholds for them during GRPO using Lagrangian dual ascent.

  • Results

    Across seven spatial reasoning benchmarks, FGRPO improves reasoning quality and final answer accuracy over standard GRPO.

  • Takeaways & Limitations

    Faithful reasoning and accurate answers are complementary objectives in multimodal reinforcement learning.

Abstract

from arXiv · show

Multimodal reasoning models (MRMs) trained with reinforcement learning with verifiable rewards (RLVR) show improved accuracy on visual reasoning benchmarks. However, we observe that accuracy gains often come at the cost of reasoning quality: generated Chain-of-Thought (CoT) traces are frequently inconsistent with the final answer and poorly grounded in the visual evidence. We systematically study this phenomenon across seven challenging real-world spatial reasoning benchmarks and find that it affects contemporary MRMs such as ViGoRL-Spatial, TreeVGR as well as our own models trained with standard Group Relative Policy Optimization (GRPO). We characterize CoT reasoning quality along two complementary axes: "logical consistency" (does the CoT entail the final answer?) and "visual grounding" (does each reasoning step accurately describe objects, attributes, and spatial relationships in the image?). To address this, we propose Faithful GRPO (FGRPO), a variant of GRPO that enforces consistency and grounding as constraints via Lagrangian dual ascent. FGRPO incorporates batch-level consistency and grounding constraints into the advantage computation within a group, adaptively adjusting the relative importance of constraints during optimization. We evaluate FGRPO on Qwen2.5-VL-7B and 3B backbones across seven spatial datasets. Our results show that FGRPO substantially improves reasoning quality, reducing the inconsistency rate from 24.5% to 1.7% and improving visual grounding scores by +13%. It also improves final answer accuracy over simple GRPO, demonstrating that faithful reasoning enables better answers.

1 Introduction

RLVR-trained multimodal reasoning models can improve answer accuracy while producing logically inconsistent or visually ungrounded CoT traces. FGRPO addresses these failures by constraining consistency and grounding during optimization, improving reasoning quality and final accuracy.

  • Motivation: Accuracy gains can mask reasoning failures, including traces that contradict the final answer or describe visual content inaccurately.The paper distinguishes logical inconsistency from visual ungroundedness, and notes that either can occur without the other.
  • Method: FGRPO defines verifiable consistency and grounding rewards, targeting whether reasoning entails the answer and whether visual descriptions match image evidence.Consistency uses an LLM judge, while grounding combines per-sentence semantic scoring with spatial bounding-box matching.
  • Method: FGRPO constrains task accuracy with minimum consistency and grounding thresholds using Lagrangian dual ascent.Adaptive Lagrange multipliers increase pressure on violated constraints and decrease it on satisfied ones, avoiding manual reward-weight tuning.
  • Evaluation: FGRPO is evaluated on two Qwen2.5-VL backbones across seven spatial reasoning datasets and improves both reasoning quality and final answer accuracy over standard GRPO.The evaluation scope and reported improvements are stated across the paper’s benchmarks and model scales.
  • Contributions: The paper characterizes reasoning degradation in RLVR-trained multimodal reasoning models along logical consistency and visual grounding axes.These axes are reported for contemporary MRMs and models trained with standard GRPO.

2 Related Work

Prior work applies RLVR and constrained policy optimization to multimodal reasoning and related safety or alignment settings. FGRPO differs by using Lagrangian dual ascent to enforce reasoning-quality thresholds in multimodal RL.

  • RLVR for Multimodal Reasoning: RLVR multimodal methods extend the SFT-plus-GRPO paradigm with techniques for visual grounding, self-reflection, structured visual representations, and progressive thinking control.TreeVGR uses localization and reasoning rewards, while ViGoRL grounds reasoning steps with image coordinates.
  • Constrained Policy Optimization: Constrained policy optimization provides a framework for optimizing a primary objective subject to auxiliary constraints, often solved with Lagrangian relaxation.Prior applications include safe exploration, LLM alignment, and vision-language-action safety constraints.
  • Constrained Policy Optimization: FGRPO uses Lagrangian dual ascent to enforce constraint thresholds rather than fixed or conditional weights, applying this strategy to consistency and visual grounding in multimodal RL.This distinguishes its constrained optimization approach from related multi-objective GRPO methods.

3 Methodology

FGRPO extends multimodal GRPO by measuring consistency and visual grounding, then enforcing these reasoning-quality signals as adaptive constraints during advantage-based policy optimization.

  • Reasoning-quality rewards: FGRPO evaluates reasoning quality through consistency and complementary semantic and spatial grounding rewards.Consistency assesses whether the reasoning entails the final answer, semantic grounding checks visual claims, and spatial grounding matches predicted boxes to image regions.
  • Reasoning-quality rewards: Consistency is a binary LLM-judge reward that tests whether the reasoning conclusion matches the final answer, while ignoring visual correctness.The reward is masked to correct-answer samples.
  • Reasoning-quality rewards: Semantic grounding scores non-trivial reasoning sentences with a VLM judge for accurate objects, attributes, spatial relations, and bounding-box content.Trivial meta-reasoning, planning, and hedging sentences are excluded, and the reward is masked to correct-answer samples.
  • Reasoning-quality rewards: Spatial grounding uses Hungarian matching and Complete IoU to compare generated bounding boxes with ground-truth regions.This reward is masked to samples with bounding-box annotations, including VGR and TreeVGR.
  • Empirical motivation: Task-reward training reaches 65.2% average accuracy but still shows a 26.1% inconsistency rate and a 72.7% mean semantic grounding score across seven spatial benchmarks.The diagnostic results indicate that answer accuracy does not fully capture reasoning quality.
  • Constrained optimization: FGRPO maximizes task reward subject to minimum consistency and grounding thresholds through Lagrangian dual ascent.Multipliers increase pressure on violated constraints and decrease it on satisfied constraints, avoiding manual reward-weight tuning.
  • Constrained optimization: Within GRPO, task and constraint rewards are independently group-normalized into advantages before being combined with adaptive Lagrange multipliers.Batch-level constraint statistics update the multipliers, and spatial grounding is applied only where bounding-box annotations are available.

4 Experiments

FGRPO is evaluated across seven spatial reasoning benchmarks and two Qwen2.5-VL backbone scales, with comparisons to GRPO baselines and reward-design alternatives. It improves accuracy while substantially improving semantic grounding and reducing inconsistency.

  • Experimental Setup: FGRPO is evaluated on seven spatial reasoning benchmarks using Qwen2.5-VL-3B and Qwen2.5-VL-7B backbones.The study compares unconstrained GRPO baselines and reward-design alternatives, with common bbox-grounded CoT initialization and reported accuracy, inconsistency, and semantic-grounding metrics.
  • Main Results: 67.16% average accuracy: FGRPO improves over GRPO-T’s 65.17% average and shows the same pattern at the 3B scale.The gain is reported across the majority of benchmarks and is not specific to the 7B backbone.
  • Reasoning Quality: 86.0% semantic grounding versus 72.7% for GRPO-T: FGRPO is higher across all seven benchmarks, with largest gains on MindCube and OmniSpatial.The reported gains are +22.8 percentage points on MindCube and +21.1 percentage points on OmniSpatial.
  • Reasoning Quality: 1.7% inconsistency: FGRPO reduces the average rate from 26.1% for GRPO-T and reaches near-zero inconsistency on six of seven benchmarks.MindCube and OmniSpatial are the most difficult datasets for baselines, with GRPO-T inconsistency rates of 57.1% and 44.0%, respectively.
  • Accuracy vs Consistency Tradeoff: 67.2% accuracy with 1.7% inconsistency: FGRPO maintains low inconsistency during training while GRPO-T rises from 7.4% to 26.1%.GRPO-T accuracy increases from 57.7% to 65.1%, whereas FGRPO pushes accuracy to 67.2% from the same warmstart checkpoint.
  • Reward Design and Multiplier Strategy: Decoupled constraint advantages outperform additive and multiplicative reward combinations, while adaptive dual ascent yields the best reported accuracy.Adding grounding constraints lifts accuracy to 67.16 (+2.0 over GRPO-T), and adaptive multipliers improve over fixed multipliers by reallocating optimization pressure as constraints are satisfied.

5 Conclusion

FGRPO enforces logical consistency and visual grounding as hard constraints during multimodal policy optimization. Across seven spatial benchmarks, it improves reasoning quality while also increasing answer accuracy over standard GRPO.

  • FGRPO enforces logical consistency and visual grounding as hard constraints during multimodal reasoning policy optimization.
  • 26.1% to 1.7%: FGRPO reduces the inconsistency rate across seven spatial reasoning benchmarks.
  • +13 percentage points: FGRPO improves semantic grounding across seven spatial reasoning benchmarks.
  • +2%: FGRPO raises answer accuracy over standard GRPO while improving reasoning quality.
  • The paper argues that multimodal RL training should treat reasoning quality as a first-class objective alongside accuracy.

A Training and Data Curation Details

The appendix details how semantic grounding rewards evaluate individual reasoning steps against image evidence. Sentences are decomposed, filtered, judged by a VLM, and supported by human-reliability validation.

  • The semantic grounding reward evaluates whether each reasoning step accurately reflects image content.
  • Semantic grounding reward: Reasoning traces are decomposed into individual sentences, which are classified as visual or trivial before scoring.
  • VLM-as-Judge Scoring: Each non-trivial sentence is judged by a VLM using the image, question, and preceding reasoning context.
  • VLM-as-Judge Scoring: The judge labels visual claims CORRECT, INCORRECT, or SKIP; SKIP sentences are excluded from reward computation.
  • Judge reliability is validated against human annotations in Appendix B.3.

A.2 Two-Stage Training Pipeline

The training pipeline follows two stages: supervised fine-tuning on curated Chain-of-Thought data, followed by RLVR with GRPO. The appendix specifies the implementation setup and additional FGRPO constraint configuration.

  • Stage one: SFT: Stage one performs supervised fine-tuning on curated Chain-of-Thought data to establish baseline spatial reasoning capability.
  • Implementation: SFT uses a frozen vision encoder while training the LLM backbone and visual projector.
  • Stage two: RLVR with GRPO: Stage two fine-tunes the SFT checkpoint with RLVR and GRPO to learn reasoning behaviors beyond the training distribution.
  • Implementation: RL training uses groups of G = 5 rollouts per prompt and a rollout batch size of 128.
  • Data flow: The pipeline curates bounding-box-grounded CoT data with MCTS and a strong visual teacher before SFT and RL training.
  • Implementation: FGRPO training additionally specifies constraint-specific hyperparameters in Table 6.

A.3 Training Data Curation

Training data are curated from diverse spatial-question datasets for both SFT and RL. SFT uses teacher-distilled MCTS traces, while RL data are filtered toward intermediate difficulty and include bounding-box supervision.

  • SFT and RL datasets are curated from three seed sources spanning diverse real-world spatial questions and images.
  • Seed datasets: SAT contributes 150K samples, VGR 90K, and VisCoT 363K samples as seed datasets.
  • Seed datasets: The seed datasets draw from COCO, GQA, OpenImages, and Flickr30k and cover diverse spatial questions.
  • CoT data curation for SFT: For SFT, 1.5K SAT, 1.5K VGR, and 3K VisCoT samples are selected, totaling 6K seed samples for MCTS trace generation.
  • CoT data curation for SFT: MCTS reasoning chains are distilled from Qwen2.5-VL-72B-Instruct, a strong visual teacher.
  • RL data curation: RL data selection prioritizes intermediate-difficulty samples because very easy or very hard samples contribute less effectively to GRPO training.

A.4 MCTS Procedure

The procedure formulates spatial reasoning as tree search, using PUCT-guided exploration and teacher-generated reasoning actions. Search trajectories are then linearized into Direct and Corrected Chains for SFT data.

  • Search formulation: Spatial reasoning is represented as a decision tree rooted at the multimodal input, with each action generating the next reasoning step.Bounding-box grounded actions pair text with coordinates and progressively zoom from broad regions to specific subregions.
  • Tree traversal: PUCT recursively selects child nodes using estimated value, prior probability, and visit count.The procedure promotes exploration with cpuct = 2.0.
  • Search phases: At expanded leaves, the teacher samples k = 3 reasoning steps, performs n = 2 high-temperature rollouts, and propagates exact-match terminal rewards backward.Expansion occurs while the tree depth remains below dmax.
  • Data synthesis: Post-search linearization extracts Direct Chains from optimal correct paths and Corrected Chains containing an incorrect branch, backtracking cue, and corrected path.These trajectories are used to synthesize SFT training data.

B.1 Evaluation Datasets and Protocol

Evaluation covers seven curated spatial datasets and approximately 6.3K questions, using standardized inference and answer-scoring procedures. Constraint evaluations separately assess consistency and semantic grounding with judge-based protocols.

  • Datasets: The benchmark suite contains seven curated datasets totaling approximately 6.3K challenging visual spatial reasoning questions.Table 8 summarizes the evaluation datasets, including circular evaluation indicators.
  • Inference protocol: Models are evaluated with greedy pass@1 decoding under fixed vLLM, hardware, context, token, and precision settings.Reasoning is generated inside <think> tags before answers inside <answer> tags.
  • Answer scoring: A lightweight non-reasoning judge extracts answers from free-form generations and compares them with ground truth, accommodating minor formatting differences.This avoids penalizing equivalent answer forms or reasoning text inside answer tags.
  • Judge validation: κ = 0.997 measures agreement between the lightweight judge and GPT-5 re-scoring, supporting near-perfect scoring agreement.The local judge scores all 6.3K test samples.
  • Constraint evaluation: Consistency evaluation asks whether the final answer follows from the reasoning trace without using the image, while semantic grounding checks visual claims against the image.The consistency judge outputs YES or NO; the semantic grounding judge evaluates each reasoning sentence with image and context.

D Training Dynamics

Training dynamics track batch-level consistency and grounding signals alongside their adaptive Lagrange multipliers. The evaluation prompts provide the image, question, accumulated reasoning context, and latest sentence for semantic grounding judgments.

  • Constraint trajectories: Figure 8 tracks batch-level consistency, spatial grounding, and semantic grounding signals across training steps.The top row shows the mean constraint signals C, G, and S.
  • Multiplier dynamics: Corresponding multipliers λC, λG, and λS adapt through dual ascent during training.The bottom row visualizes these multiplier trajectories.
  • Grounding evaluation: The semantic grounding judge receives an image, question, accumulated reasoning context, and latest sentence to evaluate.Its prompt frames the judge as a visual grounding and spatial verification evaluator.

E Qualitative Examples

Qualitative comparisons show FGRPO producing visually grounded and logically consistent traces where GRPO-Task often contradicts its answers or misreads spatial evidence. Examples span perspective, signage, depth, counting, direction, and egocentric reasoning.

  • Overall comparison: Across Figures 9–17, both models answer correctly, but GRPO-Task often produces unfaithful or inconsistent reasoning while FGRPO produces grounded, consistent traces.Reasoning steps are color-coded by grounding and consistency status.
  • Perspective estimation: FGRPO correctly identifies a low vantage point where GRPO-Task claims the image was taken from above.The contrast uses the upward-looking perspective of a wine glass.
  • Navigation and signage: FGRPO correctly interprets entering signs, terminal access, and road direction while GRPO-Task contradicts its own conclusions.These examples cover Figures 10, 12, and 15.
  • Depth and distance: FGRPO correctly reasons about relative depth and distance using scene layout or bounding-box annotations.GRPO-Task instead selects conclusions that conflict with its reasoning in Figures 11 and 14.
  • Counting and egocentric reasoning: FGRPO accurately counts three giraffes and handles egocentric perspective changes consistently.GRPO-Task reports two giraffes and misreasons about turning toward a door.
  • Evaluation-set behavior: The eval-set responses are described as exhibiting both spatial groundedness and consistency.This summary is shown in Figure 17.
Loading 2604.08476v1…