PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Drift

Pengzhan Sun, Shiu-hong Kao, Shijie Li, Yongyi Su, Junbin Xiao, Arjun Reddy Akula, Angela Yao

arXiv:2609.39183v1cs.CV

TL;DR

In visual intention grounding, IoU-based RL can reward a correct box even when the reasoning trace identifies a different object. Rita adds reference-answer probability rewards and filters for informative easy-to-medium training samples; it consistently improves grounding and reasoning on EgoIntention and RefEgo-Int.

  • Problem

    IoU-only grounding rewards evaluate the final box but cannot distinguish a consistent trace from reasoning that identifies another target, creating thinking drift.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    Rita combines reference-answer probability rewards for trace support and trace–box consistency with rollout-based filtering for informative easy-to-medium samples.

    Source: §1 · p. 2

  • Results

    P@0.5 improves over SFT Qwen2.5-VL by +4.8 (3B) and +2.1 (7B) on EgoIntention, while Rita consistently improves grounding and intention reasoning on EgoIntention and RefEgo-Int.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The authors present Rita as a step toward more thought–answer consistent vision-language models.

    Source: §5 · p. 10

    Computing the thinking reward requires a secondary forward pass with reference-answer tokens, increasing training compute and memory use versus standard RL pipelines.

    Source: F LIMITATIONS AND FUTURE WORK · p. 17

Abstract

from arXiv · show

This paper studies \textbf{thinking--answer consistency} in vision-language models. We focus on Visual Intention Grounding, where a model infers a target object based on a human intention query and predicts a bounding box. We reveal that previous IoU-based reinforcement learning (RL) frameworks suffer from ``thinking drift'', where the model produces a correct bounding box, despite having an incorrect reasoning process pointing to a different target object. Thus, we propose \textbf{Rita} (\textit{ReInforcing Thinking--Answer consistency}) as a novel RL paradigm to tame the drift. Specifically, Rita introduces two reasoning-label-free RL rewards, constructed from the conditional probability of reference answers: a \textbf{thinking reward} and a \textbf{consistency reward}. It also adopts a difficulty-aware \textbf{data filtering} strategy that selects informative easy-to-medium samples for RL using rollout error rate and reward variance. Extensive experiments on EgoIntention and the new RefEgo-Int benchmarks show that Rita performs consistently superior to the supervised finetuning approaches and vanilla RL-finetuned frameworks.

1 INTRODUCTION

Visual Intention Grounding asks a model to infer and localize an object that fulfills a goal described without naming it. The paper identifies reasoning–answer mismatches in IoU-based RL and proposes Rita’s probability-based rewards and difficulty-aware filtering to address them.

1 INTRODUCTION

  • Visual Intention Grounding requires connecting a goal to objects and their image context because an intention query may not name its target.The paper evaluates Rita on EgoIntention and the new RefEgo-Int benchmark using Qwen2.5-VL-3B and 7B-Instruct.Source: §1 · pp. 1–2
  • IoU-only RL can produce a correct box while reasoning identifies another object, and may reinforce the inconsistent high-reward rollout.In UniVG-R1, 15.8% of newly corrected boxes had reasoning targeting a different object, 2.4× the pre-RL drift rate.Source: §1 · p. 2
  • Rita uses reference-answer conditional probabilities to reward traces that support the answer and couples trace quality to whether the predicted box overlaps the target.The consistency reward addresses the risk that a high thinking-reward score could compensate for a poor predicted box, or vice versa.Source: §1 · p. 2
  • Rollout-based filtering selects easy-to-medium samples with informative reward variation, enabling RL fine-tuning on 5% of the original SFT training set.Medium-difficulty samples show the greatest reward variation; samples with no successful rollout provide little contrast for policy updates.Source: §1 · p. 2
  • Rita raises P@0.5 over SFT Qwen2.5-VL by +4.8 (3B) and +2.1 (7B) on EgoIntention, and by +10.2 and +7.5 zero-shot on RefEgo-Int.It also raises grounded reasoning accuracy over IoU-only RL and outperforms UniVG-R1 in P@0.5 without distilled reasoning traces.Source: §1 · p. 2

2 RELATED WORK

Related work establishes think-then-answer RL for VLMs and explores outcome-only rewards, trace supervision, reference-likelihood rewards, and data filtering. Rita differs in using reference likelihood to score trace support in grounding and selecting informative easy-to-medium samples.

RL for Vision Language Models

  • Perception-focused VLM work applies RL against final perceptual metrics, while trace-supervision approaches require human rationales or distilled chains of thought.The paper notes that such reasoning data can be costly to collect and that distilled traces may inherit limited teacher-model reasoning patterns.Source: §2 · p. 3

Label-free RL Rewards

  • Reference-likelihood rewards have been used to learn correct answers on unverifiable QA tasks, whereas Rita uses reference likelihood to assess whether a grounding trace supports the correct target.Rita additionally ties the trace score to the model’s own prediction, unlike reference-likelihood rewards that discard that prediction.Source: §2 · p. 3

Data Filtering in RL

  • Prior difficulty-aware RL methods prioritize hard-but-solvable or unsolved examples, while Rita favors informative easy-to-medium grounding samples.The paper motivates avoiding noisy, uninformative gradients from persistent failures such as near-zero IoU rewards.Source: §2 · p. 3

3 METHOD

Rita extends IoU-and-format reinforcement learning for visual intention grounding with reference-answer likelihood rewards that evaluate reasoning-trace support and couple it to the predicted box, alongside difficulty- and variance-aware sample selection.

3.1 PRELIMINARY: RL FOR VISUAL INTENTION GROUNDING

  • IoU-only RL evaluates the final box but not whether the reasoning trace identifies its target, allowing correct boxes to accompany drifted reasoning.The passage reports that this mismatch increased in the authors’ UniVG-R1 re-implementation.Source: §3.1 · p. 4

3.2 RITA: OUTCOME-CONDITIONED PROBABILITY-BASED REWARDS

  • The thinking reward preserves a sampled trace, replaces its answer with the reference, and scores the length-normalized likelihood of that answer conditioned on the trace.This makes reward differences reflect how strongly different traces support the fixed annotated answer.Source: §3.2 · p. 5
  • The consistency reward multiplies trace support by the predicted box’s IoU minus margin δ, rewarding strong agreement and penalizing a reference-supporting trace paired with a missed box.The paper uses δ = 0.1 so the penalty applies to boxes with little or no target overlap.Source: §3.2 · p. 5
  • Rita combines spatial-accuracy, format, trace-support, and trace–answer consistency rewards without reasoning labels, normalizing rewards within each rollout group using GRPO.The IoU-only baseline can give equal reward to the same predicted box even when reasoning traces identify different targets.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 RITA: DATA FILTERING

  • Rita’s filtering emphasizes medium-difficulty samples with variable rewards, adds hard and easy samples for exploration and stability, and discards cases with no correct rollout.Samples are bucketed by the number of correct IoU-based rollouts; larger reward spread indicates greater separation between good and bad rollouts.Source: §3.3 · p. 6

4 EXPERIMENTS

Rita improves grounding and reasoning results on EgoIntention and generalizes zero-shot to RefEgo-Int. Experiments also compare reward components, filtering strategies, and paired reasoning-grounding outputs.

4.1 IMPLEMENTATION DETAILS

  • Experiments use egocentric VIG benchmarks, Qwen2.5-VL 3B/7B models, and report grounding, reasoning, and grounded-reasoning metrics.The setup describes GRPO and LoRA for RL, full finetuning for SFT, and a UniVG-R1 reproduction using teacher-distilled traces.Source: §4.1 · pp. 6–7

4.2 QUANTITATIVE RESULTS

  • Rita-3B improves P@0.5 by 3.1 on Context and 6.3 on Uncommon; Rita-7B gains 1.7 and 2.5, respectively.Rita-3B also exceeds UniVG-R1 by 2.0/2.8 P@0.5 on Context/Uncommon without teacher-distilled reasoning traces; UniVG-R1 has higher mIoU.Source: §4.2 · p. 7, Table 1 · p. 7
  • Adding Rthink and Rcons gives the best reasoning results across both splits, while Rcons alone slightly degrades performance.With both rewards, Accthink reaches 86.6%/75.0% and Accalign 63.7%/56.5% on Context/Uncommon; Rcons complements Rthink but does not help alone.Source: §4.2 · pp. 7–8, Table 2 · p. 7

4.3 DATA FILTERING

  • Rita’s filtering strategy emphasizes informative easy-to-medium samples and excludes all-wrong cases, using rollout errors and reward variance to guide selection.The paper compares this strategy with random, unsolved, near-threshold, and normal-distribution sampling; the filtering figure tracks P@0.5 across training steps.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 QUALITATIVE RESULTS

  • In paired examples, Rita’s reasoning describes the correctly localized target where the IoU-reward baseline names a different object.Examples include tissue paper called a sponge, a tray called a bucket, and scissors called a knife; in the final example, Rita localizes the stool while the baseline predicts a ladder incorrectly.Source: §4.4 · p. 8, Figure 4 · p. 9

4.5 ZERO-SHOT EVALUATION

  • Rita improves zero-shot P@0.5 on RefEgo-Int by 10.2% for 3B and 7.5% for 7B over SFT baselines.At step 500, the 3B model leads the baseline by 1.0% in intention reasoning accuracy, while the 7B model reaches 37.6% peak reasoning accuracy.Source: §4.5 · p. 9, Figure 5 · p. 9

5 CONCLUSION

The conclusion presents Rita as a response to mismatches between reasoning traces and correct boxes in visual intention grounding. It combines probability-based rewards with difficulty- and informativeness-aware filtering, and reports consistent grounding and reasoning improvements.

5 CONCLUSION

  • Rita addresses cases where a correct bounding box is paired with reasoning that identifies a different object.Its rewards reinforce reasoning traces based on their contribution to predicting the correct answer.Source: §5 · p. 9
  • Rita’s filtering strategy focuses GRPO training on informative easy-to-medium samples.The conclusion describes the filtering as difficulty- and informativeness-aware.Source: §5 · pp. 9–10
  • Experiments on EgoIntention and RefEgo-Int show consistent improvements in grounding accuracy and intention reasoning over strong Qwen2.5-VL baselines.The paper reports state-of-the-art performance within the evaluated setting.Source: §5 · p. 10

AI USAGE STATEMENT

The authors report using AI tools to assist with drafting and polishing the paper’s wording.

AI USAGE STATEMENT

  • AI tools assisted with drafting and polishing sections for wording, grammar, and clarity.The authors state that they take responsibility for the final content; teacher-generated baseline traces are described in Section 4.1 and Appendix C.Source: AI USAGE STATEMENT · p. 10

A REFEGO-INT DATASET CONSTRUCTION

RefEgo-Int is constructed as an image-based counterpart to RefEgo, retaining its challenge distribution while adding intention sentences. Its construction samples frames by clip and verifies intention and box annotations.

A REFEGO-INT DATASET CONSTRUCTION

  • RefEgo-Int contains 1,317 images and adds a human intention sentence to each sample from RefEgo.It is an image-based counterpart intended to preserve RefEgo’s challenge distribution.Source: A REFEGO-INT DATASET CONSTRUCTION · p. 14
  • The construction selects one annotated frame per clip, stratifying across eight bins of camera motion, referred-object count, and target visibility.When multiple frames qualify, the earliest is preferred; the resulting image dataset mirrors the video set’s challenge mix while retaining a single frame per clip.Source: A REFEGO-INT DATASET CONSTRUCTION · p. 14
  • Annotators add intention sentences and use VGG Image Annotator to check them and collect or correct bounding boxes.Figure 6 shows the VIA interface used in this annotation workflow.Source: A REFEGO-INT DATASET CONSTRUCTION · p. 14, Figure 6 · p. 14

B STATISTICAL CHARACTERISTICS OF SAMPLE DIFFICULTY BINS

Rollout statistics support selecting easy-to-medium samples for grounding RL: medium samples offer the strongest contrast, while very hard samples mostly produce failures.

B STATISTICAL CHARACTERISTICS OF SAMPLE DIFFICULTY BINS

C ANALYSIS OF THINKING DRIFT UNDER IOU-ONLY RL

Under IoU-only RL, grounding accuracy rises while thinking drift also increases: some newly correct boxes point to the wrong target category.

C ANALYSIS OF THINKING DRIFT UNDER IOU-ONLY RL

  • A sample counts as drift when its box is correct (IoU ≥0.5) but its predicted object category differs from ground truth.The predicted category is used as a proxy for the reasoning target; the <think> text itself is not judged.Source: C ANALYSIS OF THINKING DRIFT UNDER… · p. 15, Table 5 · p. 16
  • The comparison uses 10,000 Context and Uncommon test samples, contrasting a chain-of-thought cold start with the model after 500 RL steps.The UniVG-R1 re-implementation conditions its teacher on the ground-truth box and category and generates one trace per sample.Source: C ANALYSIS OF THINKING DRIFT UNDER… · p. 15
  • IoU-only RL raises P@0.5 by 1.4 points, while overall drift rises from 4.01% to 4.98% of samples.Of 539 newly corrected boxes, 85 (15.8%) have a wrong category, 2.4× the pre-RL drift rate among correct boxes (6.54%); the paired increase is significant (exact McNemar test, p = 3.7×10−7).Source: C ANALYSIS OF THINKING DRIFT UNDER… · p. 15, Table 5 · p. 16

D ADDITIONAL ABLATION STUDY

The ablations examine the consistency-reward margin, reference-answer overwrite, and prompt setup, finding benefits from the default margin and reference-answer scoring.

D.1 IOU MARGIN OF THE CONSISTENCY REWARD

  • At step 500, δ = 0.1 beats δ = 0.5 on all three Context metrics (+0.5 P@0.5, +1.0 Accthink, +1.1 Accalign) and both Uncommon reasoning metrics (+0.5 each), with equal P@0.5.At steps 100 and 300, δ = 0.5 leads on most metrics; δ = 0.1 penalizes boxes that largely miss the target rather than partially overlapping boxes.Source: §D.1 · p. 16, Table 6 · p. 16

D.2 NECESSITY OF REFERENCE-ANSWER OVERWRITE

  • At step 500, reference-answer overwrite raises Accthink by 1.4 points over Base RFT, while scoring the sampled answer lowers it by 0.6 points.The reference answer ties the reward to whether the trace supports the correct target; the sampled answer can reward confident incorrect predictions.Source: §D.2 · pp. 16–17, Table 7 · p. 16

E PROMPT TEMPLATE

  • Training and evaluation use VLM-R1’s referring-expression prompt unchanged, inserting the intention sentence verbatim and requesting a reasoning trace followed by a JSON box answer.The prompt does not request an object label; the answer label is inherited from supervised finetuning and is used to score Accthink and Accalign.Source: E PROMPT TEMPLATE · p. 17

F LIMITATIONS AND FUTURE WORK

Rita’s reference-answer reprocessing increases training cost and memory use; future work proposes making its reward explicitly verify visual grounding.

F LIMITATIONS AND FUTURE WORK

  • Computing Rthink requires a secondary forward pass that overwrites predicted answers with reference-answer tokens, increasing training cost and memory use.This is additional overhead compared with standard RL pipelines.Source: F LIMITATIONS AND FUTURE WORK · p. 17
  • Future work proposes conditioning the reward on attended or masked image features derived from reasoning content to check use of scene cues.The current rewards implicitly consider images through joint embeddings but do not ensure the trace uses specific visual cues.Source: F LIMITATIONS AND FUTURE WORK · p. 17

Related papers

Loading 2609.39183v1…