PaperlayerView exact revision on arXiv

Source-linked AI summary

VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning

Qunzhong Wang, Jie Liu, Jiajun Liang, Yilei Jiang, Yuanxing Zhang, Yaozhi Zheng, Xintao Wang, Pengfei Wan, Xiangyu Yue, Jiaheng Liu

arXiv:2510.10518v4cs.CV

TL;DR

Video reward models are constrained by visual context budgets and static visual evidence during reasoning. VR-THINKER adds active visual reasoning and a staged training pipeline, and the authors report improved preference judgments and interpretable reward signals. Its longer inference chains, however, increase latency and computational cost.

  • Problem

    Video reward models face context limits that can exclude fine-grained frame details, while visual evidence is typically fixed in the initial prompt during reasoning.

    Source: §1 · p. 1

  • Method

    VR-THINKER equips reward models with frame-selection operations and configurable visual memory, trained through Cold Start, verified-trace rejection sampling, and GRPO.

    Source: §1 · p. 2

  • Results

    VR-THINKER achieves state-of-the-art performance across both evaluation protocols, with average improvement of up to 11.4%.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    In principle, VR-THINKER can evaluate long-video reward tasks without a preset upper bound on frames while preserving fidelity.

    Source: §1 · p. 2

    Longer inference chains increase latency and computational cost.

    Source: F Limitations · p. 24

Abstract

from arXiv · show

Recent advancements in multimodal reward models (RMs) have substantially improved post-training for visual generative models. However, current RMs face inherent limitations: (1) visual inputs consume large context budgets, forcing fewer frames and causing loss of fine-grained details; and (2) all visual information is packed into the initial prompt, exacerbating hallucination and forgetting during chain-of-thought reasoning. To overcome these issues, we introduce VideoReward Thinker (VR-Thinker), a thinking-with-image framework that equips the RM with visual reasoning operations (e.g., select frame) and a configurable visual memory window. This allows the RM to actively acquire and update visual evidence within context limits, improving reasoning fidelity and reliability. We activate visual reasoning via a reinforcement fine-tuning pipeline: (i) Cold Start with curated visual chain-of-thought data to distill basic reasoning skills and operation formatting; (ii) select samples whose per-dimension and overall judgments are all correct, then conduct Rejection sampling Fine-Tuning on these high-quality traces to further enhance reasoning; and (iii) apply Group Relative Policy Optimization (GRPO) to strengthen reasoning. Our approach delivers state-of-the-art accuracy among open-source models on video preference benchmarks, especially for longer videos: a 7B VR-Thinker achieves 80.5% on VideoGen Reward, 82.3% on GenAI-Bench, and 75.6% on MJ-Bench-Video. These results validate the effectiveness and promise of thinking-with-image multimodal reward modeling.

1 Introduction

The introduction argues that video reward models face context-budget and visual-forgetting limitations, then presents VR-THINKER’s visual reasoning framework and staged training as a way to improve preference judgments and their interpretability.

1 Introduction

  • Video reward models face large visual-input context costs and cannot revisit or update visual evidence during text-only reasoning.Earlier reward models also tended either to provide black-box scalar scores or brief justifications with limited logical structure and analysis.Source: §1 · p. 1
  • VR-THINKER uses frame selection and a configurable visual memory window to acquire and update visual evidence while keeping memory footprint stable.In principle, this lets it process videos without a preset upper bound on frame count.Source: §1 · p. 2
  • Training proceeds through Cold Start, rejection-sampling fine-tuning on traces with correct judgments, and GRPO to strengthen visual and textual reasoning.Cold Start uses curated visual chain-of-thought data to teach basic reasoning and operation formatting.Source: §1 · p. 2
  • VR-THINKER’s visual reasoning improves preference-judgment accuracy and reliability, with increased usability and fidelity.The paper identifies it as the first multimodal reward model capable of visual reasoning.Source: §1 · p. 2

2 Related Work

Related work describes shortcomings of existing video reward models, situates Thinking-with-Image as an operable visual workspace, and contrasts it with query-driven video retrieval.

2 Related Work

  • Existing VLM-based reward models can subsample video frames under context limits and reason without revisiting or updating visual evidence.Prior approaches also often lack rigorous logical structure and deep analysis.Source: §2 · p. 3
  • Thinking-with-Image treats vision as a dynamic workspace used during intermediate reasoning, through intrinsic imagination or active exploration.Active exploration retrieves visual information through tool invocation or programmatic manipulation.Source: §2 · p. 3
  • Video RAG retrieves segments relevant to an external text query, whereas Thinking-with-Image enables autonomous, reasoning-driven visual exploration.Video RAG uses predefined similarity metrics to determine relevance.Source: §2 · p. 3

3 VR-Thinker

VR-Thinker combines iterative visual evidence retrieval and windowed memory with a multi-stage training pipeline. Its training develops multimodal reasoning through curated traces, verified samples, and reinforcement learning.

3.1 Thinking-with-Image-Based Framework

  • The model retrieves additional video evidence as needed and retains tool outcomes only within a preset window, while preserving the full reasoning prefix.Windowed memory keeps total token use relatively stable as reasoning proceeds.Source: §3.1 · pp. 3–4

3.2 Multi-Stage Reward Model Training

  • Training proceeds through Cold Start, Rejection Sampling Fine-Tuning, and Exploratory Reinforcement Learning to develop integrated multimodal reasoning.The stages elicit initial skills, consolidate reasoning, then reinforce multimodal capability.Source: §3.2 · p. 5

3.2.1 Cold Start & Rejection Fine-Tuning

  • Cold Start uses format-conforming visual chain-of-thought data to teach tool-calling structure and bootstrap visual reasoning.The generated traces are filtered for format compliance and correct per-dimension and overall judgments before supervised fine-tuning.Source: §3.2.1 · p. 5
  • Rejection Sampling Fine-Tuning trains on model-generated traces filtered for correctness and format to consolidate reasoning and increase high-quality samples.Multiple samples are drawn per input, and tool-execution outcome tokens are masked from the supervised loss.Source: §3.2.1 · p. 5

3.2.2 Exploratory Reinforcement Fine-Tuning

  • GRPO ranks multiple sampled reasoning traces with rule-based rewards for response format and factual accuracy, including per-dimension judgments and overall preference.The accuracy reward assumes that satisfying correctness rules indicates a high-quality reasoning trace.Source: §3.2.2 · pp. 5–6
  • CoT Gain Reward scores accuracy improvements in updated answers across reasoning segments.The reward is intended to encourage acquiring more visual evidence and updating conclusions with greater accuracy and factual alignment.Source: §3.2.2 · p. 6
  • Because textual reasoning can yield decent answers despite weak visual reasoning, exploratory incentives encourage a minimum proportion of multimodal reasoning.The constrained objective is expressed with an auxiliary reward that activates when a query's multimodal-reasoning proportion is below its lower bound.Source: §3.2.2 · p. 6

4 Experiments

VR-THINKER outperforms a range of reward models across both preference-accuracy protocols, while ablations and hard-subset analyses examine visual reasoning, training stages, reward settings, and challenging inputs.

4.2 Main Results

  • Across tau and diff protocols, VR-THINKER achieves state-of-the-art performance, with an average improvement of up to 11.4%.It surpasses classic classifier-based, generative-based, and reasoning-style models; Table 1 reports preference accuracy on the evaluation dataset.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Ablation Studies

  • Randomly retrieving video information instead of following visual-reasoning-driven tool invocations causes a clear performance drop.Figure 3 summarizes this visual-reasoning ablation.Source: §4.3 · p. 7, Figure 3 · p. 8
  • GRPO contributes the largest performance improvement, while Cold Start and Rejection sampling Fine-Tuning provide reasoning foundations that further boost post-GRPO performance.The Rejection sampling Fine-Tuning gains are especially pronounced, which the authors suggest may reflect its increased likelihood of high-quality reasoning segments.Source: §4.3 · p. 7, Figure 3 · p. 8
  • Removing either the CoT gain reward or exploratory incentive reduces performance, with the larger drop occurring when the CoT gain reward is removed.The authors associate the CoT gain reward with encouraging multimodal reasoning attempts.Source: §4.3 · pp. 7–8
  • The adopted 50/50 mix of overall and per-dimension accuracy rewards performs better than using either reward alone.Figure 3 summarizes the comparison among the mixed scheme and the two single-reward settings.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 Further Analysis

  • Figure 4 tracks GenAI-Bench accuracy, average tool invocations, and reasoning-segment length during GRPO training across experimental settings.The settings include variants without exploratory reward or one of the accuracy-reward components.Source: §4.4 · pp. 8–9, Figure 4 · p. 9
  • VR-THINKER shows a smaller accuracy drop than baseline models across hard subsets selected by video length or prompt length.The analysis selects the top 10% by each criterion; Table 2 reports preference accuracy on the long-video and complex-prompt subsets.Source: §4.4 · p. 9, Table 2 · p. 9

5 Conclusion

The paper presents VR-THINKER as a multimodal reward model with visual reasoning, designed to alleviate context-length constraints and visual-information forgetting. Its experiments report improved preference accuracy and interpretability of reward signals.

5 Conclusion

  • VR-THINKER is presented as the first multimodal reward model capable of visual reasoning, and the paper reports improved preference accuracy and interpretability of reward signals.The authors state that its framework alleviates context-length constraints and mitigates forgetting of visual information.Source: §5 · p. 9

Appendix

  • The appendix is organized to provide mathematical details, experimental details, additional results, prompt templates, and further discussion of failure modes and terminology.It also identifies appendices on limitations and LLM usage.Source: Appendix · p. 13

A.1 Mathematical details of the training pipeline

The appendix details supervised and reinforcement fine-tuning, including methods for improving sampling efficiency and constraining multimodal reasoning incentives. It also reports implementation settings and the reward-model baseline categories.

A.1 Mathematical details of the training pipeline

  • SFT masks tool-execution outcome tokens from the loss, while the GRPO objective uses clipping for stability and a KL penalty to constrain policy updates.The policy ratio compares new and old policies; the section defines the KL penalty as a constraint on the model update range.Source: §A.1 · pp. 13–14
  • GRPO uses a rule-based reward to nudge higher-quality reasoning; Dynamic Sampling filters batches with accuracy 0 or 1 and resamples until every batch yields nonzero gradients, improving training efficiency.Source: §A.1 · pp. 13–14
  • At observed accuracy 0.7, estimated invalid samples fall from 15% with N=3 to 0.375% with N=81.Invalid samples can be correct by chance without finding key information, making them unhelpful or harmful when rewarded in RL.Source: §A.1 · p. 14

A.2 Derivation of the GRPO Exploratory Incentive

  • The Exploratory Incentive encodes a minimum multimodal-reasoning ratio as a constraint rather than directly rewarding a higher CoT ratio, to avoid reward hacking.The formulation uses a fixed η instead of dynamically adjusting a Lagrange multiplier.Source: §A.2 · pp. 14–15

B.1 Training Details.

  • GRPO uses 64 queries and 8 responses per query, with 512 samples per batch; cold-start and rejection-sampling fine-tuning use 8 A800 GPUs, versus 32 for GRPO.The reported defaults include α=0.5, k=0.2, η=0.5, ω=0.2, and window width p=1.Source: §B.1 · p. 15

B.2 Compared Baselines.

  • The baselines span classifier-based, generative-based, and reasoning-based reward models; VR-THINKER adds multimodal reasoning to the reasoning-based category.The paper characterizes this addition as addressing processed-frame limits and forgetting associated with purely textual reasoning.Source: §B.2 · pp. 15–16

B.3 Datasets and usage settings

The section describes the fine-grained training and evaluation datasets, then reports hyperparameter, rejection-sampling data-volume, and subset comparisons. The supplementary results indicate weaker gains on the residual subset and greater robustness to duplicated frames.

B.3 Datasets and usage settings

  • Training uses datasets with per-dimension preference labels, while evaluation uses overall preference accuracy on GenAI-Bench, VideoGen-RewardBench, and MJ-Bench-Video.The training datasets are VideoGen-Reward, MJ-Bench-Video train, and Text2Video-Human Preferences; evaluation retains the benchmark prompt and response format.Source: §B.3 · p. 16

C.1 Comparison of different hyperparameter choices

  • α=0.5 and k=0.2 perform best in the reported searches; larger k values do not further improve performance and may invite reward hacking.α balances overall and per-dimension accuracy rewards; k controls the CoT gain reward.Source: Figure 5 · p. 17, §C.1 · p. 17

C.2 Comparison of reject fine-tuning data volume

  • Rejection-sampling fine-tuning data improves post-GRPO performance at smaller scales, but using 40k examples degrades it, potentially by reducing output entropy.The authors describe the positive relationship at smaller data volumes as a correlation.Source: Figure 5 · p. 17, §C.2 · p. 17

C.3 Evaluation on remained and duplicated set

  • Improvements on the Residual subset are less pronounced than on the Longer video and Complex prompt subsets; with duplicated frames, VR-THINKER has a smaller performance drop than other models.The redundancy test duplicates randomly positioned frames to double the total frame count.Source: §C.3 · p. 18

C.4 Inference metrics comparison with baseline models

The section compares inference accuracy and system efficiency, assesses out-of-distribution results and cold-start sources, and supplies prompt and dimension definitions. It reports stronger system efficiency against a reasoning baseline and similar final accuracy across cold-start sources.

C.4 Inference metrics comparison with baseline models

  • VR-THINKER supports 2.25× higher concurrency (9 vs. 4) and 2.4× greater throughput (162 vs. 67 tok/s) than Unified-Reward-think(32), while achieving superior performance.The authors report throughput comparable to 8-frame baselines; naively expanding input frames can cause OOM and diminishing or negative accuracy returns.Source: §C.4 · p. 18

C.5 Out-Of-Distribution Evaluation

  • Excluding MJ-Bench-Video training data has negligible influence on GenAI-Bench and VideoGen-RewardBench, with a slight decrease on the MJ-Bench-Video test set.The authors note that MJ-Bench-Video accounts for approximately 4% of total training data and that its train and test subsets may overlap distributionally.Source: §C.5 · p. 19

C.6 Effect of Cold-Start Data Source

  • After the full pipeline, cold-start datasets generated by GPT-4o, Qwen3-VL-8B, or InternVL3.5-8B yield nearly identical accuracy across the three benchmarks.The authors say the differences diminish after RFT and become negligible after GRPO.Source: §C.6 · p. 20

D Prompts templates

  • The prompt asks for iterative frame analysis, dynamic frame selection, and judgments for each dimension plus an overall preference.The format specifies Snapshot summaries after new visual information, Think reasoning, and Answer or Recommend Answer outputs.Source: D Prompts templates · pp. 20–21

VideoGen-Reward

  • VideoGen-Reward defines Text Alignment as video-prompt alignment, Visual Quality as aesthetics, and Motion Quality as motion coherence.Source: VideoGen-Reward · p. 21

Text2Video-Human

  • Text2Video-Human defines Alignment as video-prompt alignment, Preference as intrinsic aesthetics, and Coherence as temporal coherence.Source: Text2Video-Human · p. 21

MJ-Bench-Video

  • MJ-Bench-Video defines Alignment as video-prompt alignment, Fineness as visual-content fineness, and Coherence & Consistency as temporal coherence and consistency.Source: MJ-Bench-Video · p. 21

E.1 Qualitative Analysis of Failure Modes

VR-THINKER is generally reliable on standard preference pairs but can reason overly cautiously in out-of-distribution or ambiguous cases. The paper frames its visual reasoning operations as deliberate evidence-seeking and evidence-integration processes, not merely retrieval.

E.1 Qualitative Analysis of Failure Modes

  • VR-THINKER performs well on caption-aligned video pairs with discernible quality differences, while deviations from that regime can trigger overly cautious reasoning.The authors report that these cases mainly occur in out-of-distribution or ambiguous scenarios; typical preference data remain accurate and reliable.Source: §E.1 · pp. 22, 24
  • Overly cautious reasoning adds frame-selection and multimodal-reasoning rounds even when the preference is apparent, increasing latency and redundant tool calls.The authors say this behavior rarely harms final accuracy.Source: §E.1 · p. 22
  • One-sided caption alignment can prompt searches for caption-relevant evidence absent from one video, while near-identical pairs can prompt redundant searches for nonexistent differences.The examples involve additional zooming or frame retrieval, including hesitation to give the neutral judgment for near-identical videos.Source: §E.1 · p. 23
  • Targeted fine-tuning is proposed to make VR-THINKER more decisive once visual evidence is sufficient, potentially improving efficiency and robustness.The authors describe overly cautious reasoning as a tendency to avoid premature conclusions.Source: §E.1 · p. 24

E.2 Discussion on Terminology of Visual Reasoning Operation

  • A visual reasoning operation includes reasoning that determines whether retrieval is needed and which frames to select; the tool call executes that plan.The model analyzes visual context, identifies gaps, and forms a hypothesis about where decisive evidence may be found.Source: §E.2 · p. 24
  • The visual memory window also requires active reasoning to summarize and compress new visual evidence while integrating it with the existing understanding.The model selects salient information to retain in the <snapshot> tag as frames are added.Source: §E.2 · p. 24
  • The authors prefer “visual reasoning operation” because the framework reasons about what and when to retrieve and how to integrate evidence, rather than merely retrieving frames.They distinguish this deliberate process from passive, externally driven retrieval.Source: §E.2 · p. 24

F Limitations

The authors identify inference overhead and limits in the current training pipeline as areas for improvement.

F Limitations

  • Multimodal reasoning introduces longer inference chains, increasing latency and computational cost.The authors also note that Reject Fine-Tuning and GRPO tend to amplify learned capabilities, and call for higher-quality supervised CoT data to achieve more substantial gains.Source: F Limitations · p. 24
Loading 2510.10518v4…