PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning

Yibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, Jiaqi Wang

arXiv:2505.03318v3cs.CV

TL;DR

Multimodal reward models often provide direct or shallow judgments, motivating more reliable reasoning for complex vision reward tasks. UnifiedReward-Think addresses this gap with staged CoT distillation, rejection sampling, and GRPO reinforcement fine-tuning, and experiments show superior performance across vision reward tasks, including without explicit CoT outputs.

  • Problem

    Existing multimodal reward models are generally restricted to direct responses or shallow reasoning, while large-scale CoT reward data is scarce for supervised training.

  • Method

    UnifiedReward-Think uses a three-stage exploration-driven pipeline combining GPT-4o-distilled cold start, rejection sampling, and GRPO-based reinforcement fine-tuning.

  • Results

    UnifiedReward-Think consistently outperforms existing baselines across all vision reward tasks, including when producing direct rewards without explicit CoT traces.

  • Takeaways & Limitations

    Long CoT reasoning improves reward-signal accuracy and robustness, while internalized reasoning also strengthens direct reward outputs without explicit reasoning traces.

  • Takeaways & Limitations

    Long-form CoT reasoning increases inference time, and reinforcement learning can amplify only capabilities already acquired during supervised fine-tuning.

Abstract

from arXiv · show

Recent advances in multimodal Reward Models (RMs) have shown significant promise in delivering reward signals to align vision models with human preferences. However, current RMs are generally restricted to providing direct responses or engaging in shallow reasoning processes with limited depth, often leading to inaccurate reward signals. We posit that incorporating explicit long chains of thought (CoT) into the reward reasoning process can significantly strengthen their reliability and robustness. Furthermore, we believe that once RMs internalize CoT reasoning, their direct response accuracy can also be improved through implicit reasoning capabilities. To this end, this paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks. Specifically, we adopt an exploration-driven reinforcement fine-tuning approach to elicit and incentivize the model's latent complex reasoning ability: (1) We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o, which is then used for the model's cold start to learn the format and structure of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge and generalization capabilities, we prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks. During this phase, correct reasoning outputs are retained for rejection sampling to refine the model (3) while incorrect predicted samples are finally used for Group Relative Policy Optimization (GRPO) based reinforcement fine-tuning, enabling the model to explore diverse reasoning paths and optimize for correct and robust solutions. Extensive experiments across various vision reward tasks demonstrate the superiority of our model.

1 Introduction

The paper addresses limitations in multimodal reward models by introducing explicit long-chain reasoning and a staged reinforcement fine-tuning strategy. UnifiedReward-Think uses CoT reasoning for visual understanding and generation, improving both explicit reward reliability and implicit direct-response accuracy.

  • Existing multimodal reward models typically assign scores, rank outputs, or use shallow reasoning, limiting reward reliability in complex scenarios.
  • Explicit long-chain CoT is proposed to enhance reward-model reliability and robustness, while internalized CoT may improve direct reward accuracy without visible reasoning traces.
  • UnifiedReward-Think is a unified multimodal CoT reward model for multi-dimensional, step-by-step reasoning across visual understanding and generation tasks.
  • The training pipeline distills GPT-4o reasoning for cold start, elicits CoT on unified preference data, and uses verified samples for rejection sampling and GRPO fine-tuning.
  • Experiments show that long CoT improves reward-signal accuracy and reliability, while internalized reasoning enables direct outputs to outperform existing baselines across vision reward tasks.

2 Related work

Prior multimodal reward models support preference alignment but generally provide direct or shallow judgments. This work applies exploration-driven GRPO reinforcement fine-tuning to integrate long CoT reasoning into multimodal reward modeling.

  • Multimodal reward models learn human-judgment-based reward functions to align vision understanding and generation models with preferences.
  • Existing methods are largely limited to direct responses or shallow reasoning, which can produce inaccurate or unreliable rewards in complex scenarios.
  • Reinforcement learning has been used for language-model reasoning and several vision tasks, but its application to multimodal reward models remains under exploration.
  • The proposed pipeline combines cold start, rejection sampling, and GRPO to elicit multi-dimensional, step-by-step reasoning across diverse vision tasks.

3 Method

UNIFIEDREWARD-THINK uses exploration-driven reinforcement fine-tuning to elicit long-chain multimodal reward reasoning across visual understanding and generation tasks. Its pipeline combines cold-start distillation, rejection sampling, and GRPO with verifiable format and accuracy rewards.

  • Overview: UNIFIEDREWARD-THINK targets reliable multimodal reward decisions by introducing multi-dimensional, step-by-step long-chain CoT reasoning across visual understanding and generation.The approach addresses the challenge of adding CoT reward reasoning when large-scale CoT reward data is scarce.
  • Cold start: The cold-start stage distills GPT-4o reasoning from a small image-generation preference set and retains trajectories whose final answers match ground truth.This initializes the model to follow a structured CoT reasoning format.
  • Rejection sampling: Rejection sampling applies the initialized model to large-scale unified preference data, retaining correctly reasoned samples to reinforce accurate reasoning patterns across vision tasks.The process concentrates training on accurate reasoning patterns and supports generalization across visual domains.
  • GRPO: GRPO uses challenging incorrectly reasoned samples for reinforcement fine-tuning, enabling trial-and-error exploration guided by verifiable rewards.The broader GRPO procedure samples multiple responses, evaluates them with format and accuracy rewards, and compares their relative quality within each group.
  • Verifiable rewards: Format reward assigns Rfmt=1 when both ⟨think⟩ and ⟨answer⟩ tags are correctly present, while accuracy reward assigns 1 when the answer exactly matches ground truth.The overall verifiable reward is R = Rfmt + Racc, encouraging structured and correct responses.
  • GRPO: GRPO normalizes within-group rewards into relative advantages, clips policy ratios to [1 −δ, 1 +δ], and adds KL regularization against a reference model for stable optimization.These components guide the policy toward higher-quality, verifiably correct CoT paths.

4 Experiments

Experiments evaluate UnifiedReward-Think across image and video understanding and generation reward tasks, showing broad gains from multi-dimensional CoT reasoning and staged training.

  • Experimental setup: Evaluations cover image and video understanding on VLRewardBench and ShareGPTVideo, plus image and video generation on GenAI-Bench and VideoGen-RewardBench.Understanding evaluations use 5K test samples each on VLRewardBench and ShareGPTVideo.
  • Comparison results: The model consistently outperforms existing methods across evaluations, with substantial gains over UnifiedReward from multi-dimensional, multi-step reasoning.Image understanding shows particularly significant improvements; generation evaluation excludes tie scenarios because they were absent from training.
  • Ablation studies: After cold start, the model learns CoT format but still struggles with accurate reward prediction.Rejection sampling improves performance by retaining correctly reasoned samples for supervised fine-tuning.
  • Ablation studies: GRPO produces the largest ablation gains by focusing on mispredicted cases and exploring multiple reasoning paths toward more accurate solutions.The staged results indicate complementary roles for cold start, rejection sampling, and GRPO.
  • Ablation studies: GRPO without CoT yields only slight baseline improvements, indicating that final-answer optimization alone does not teach the underlying reasoning process.CoT-based GRPO instead explores multiple reasoning trajectories and supports deeper understanding and more robust generalization.

5 Limitations and future works

The method improves reward modeling with long-form CoT, but this increases inference time and remains bounded by the capabilities acquired during supervised fine-tuning.

  • Long-form CoT reasoning inevitably increases inference time during reasoning.The paper also reports that implicit reasoning can improve answer accuracy without explicitly generating CoT traces.
  • Because reinforcement learning cannot fundamentally extend capabilities beyond those acquired during SFT, scaling high-quality CoT supervision remains a promising direction.The paper frames reinforcement fine-tuning as amplifying latent long-CoT ability rather than creating wholly new capability.

6 Conclusion

The paper presents UNIFIEDREWARD-THINK as a unified multimodal CoT reward model and reports improved explicit and implicit reward reasoning across visual tasks.

  • UNIFIEDREWARD-THINK performs multi-dimensional, step-by-step reward reasoning for visual understanding and generation tasks.It is described as the first unified multimodal CoT reward model with this capability.
  • An exploration-driven reinforcement fine-tuning strategy uses cold start, rejection sampling, and GRPO to elicit latent complex reasoning.The three stages form the paper’s central training approach.
  • CoT reasoning improves reward-signal accuracy and robustness while also enabling superior performance without explicit CoT outputs.The conclusion links explicit reasoning improvements with strong implicit reasoning capabilities.

A Further methodological insights

The methodological analysis explains why limited image-generation CoT data can generalize, why rejection sampling precedes GRPO, and how multidimensional scoring aligns reasoning with final decisions.

  • Cold start only with image generation preference data: why it works: A small amount of high-quality image-generation CoT data teaches reasoning format and structure across visual reward tasks.The paper attributes this transfer partly to video tasks being interpretable as multi-image understanding problems.
  • Rejection sampling for unified reward generalization fine-tuning: why we need it: Rejection sampling retains correct CoT analyses before GRPO, avoiding inefficient optimization of already-mastered samples.The method leverages prior cross-task knowledge while reserving GRPO for cases where further exploration is useful.
  • Why we trust the CoT reasoning when the final answer is correct during GRPO: Multidimensional image scoring followed by aggregation aligns intermediate reasoning with the final reward decision.This structure makes a correct final answer serve as an implicit validation signal for the reasoning trajectory during GRPO.
  • Why learning can be generalized to different tasks: The same visual-content-plus-text input and reward-judgment output structure supports generalization across image and video understanding and generation tasks.The unified model is designed around this shared input-output structure despite differing downstream tasks.

B Robustness on different baselines

UNIFIEDREWARD-THINK is evaluated across different base-model backbones and different CoT distillation sources. The cited tables compare LLaVA-OneVision versus Qwen2.5-VL, and GPT-4o versus Qwen2.5-VL distillation.

  • Different backbones: Qwen2.5-VL-7b is used as an additional backbone to assess UNIFIEDREWARD-THINK robustness across base models.The Qwen2.5-VL–based model achieves consistent improvements.
  • Different backbones: Table 6 compares UNIFIEDREWARD-THINK trained on LLaVA-OneVision and Qwen2.5-VL.
  • CoT distillation sources: Table 7 compares CoT samples distilled from GPT-4o versus Qwen2.5-VL.

C Role of distilled CoT samples on cold-start stage

Cold-start CoT data primarily teaches the model the structure and format of CoT reasoning, while later rejection sampling and GRPO strengthen reward discrimination. Comparable performance after reinforcement is possible even with distilled samples from Qwen2.5VL-72b instead of GPT-4o.

  • Cold-start role: Cold-start CoT data teaches reasoning structure and format rather than directly improving reward decision quality.
  • Subsequent reinforcement: Rejection sampling and GRPO on large-scale unified multimodal preference data strengthen reward discrimination through correct reasoning trajectories.These stages iteratively reinforce correct reasoning trajectories.
  • Distillation source: Qwen2.5VL-72b distilled CoT samples can yield comparable performance to GPT-4o distillation after rejection sampling and GRPO.

D Details of our cold-start dataset: ImageGen-CoT-Reward-5K

The cold-start dataset consists of a small set of GPT-4o-distilled CoT reward reasoning samples for image generation. Samples are drawn from HPD, EvalMuse, and OIP to teach CoT format and structure.

  • Dataset construction: GPT-4o generates distilled CoT reward reasoning samples for image generation cold-start training.
  • Dataset construction: The data contains 10K randomly selected samples from the HPD, EvalMuse, and OIP datasets.
  • Reasoning format: The cold-start samples teach the model the format and structure of CoT reasoning.The reasoning format is designed to maintain consistency between the reasoning process and final decision.

E More experimental details

The paper evaluates unified visual reward modeling across image and video generation and understanding tasks using diverse baselines and benchmarks. Experimental details include the UnifiedReward architecture, baseline coverage, and benchmark-specific evaluation settings.

  • Reward model baselines: The baseline comparison covers strong reward models across image and video domains, including generation and understanding tasks.
  • Architecture: UnifiedReward provides the base architecture by unifying image and video generation and understanding reward tasks through multi-task learning.It is described as a solid baseline for holistic visual reward modeling.
  • Training pipeline: UNIFIEDREWARD-THINK adds explicit long CoT reasoning through cold start, rejection sampling, and GRPO training stages.The extension targets both visual understanding and generation tasks.
  • Evaluation benchmarks: VLRewardBench contains 1,250 curated image-understanding samples spanning vision-language queries, hallucination detection, and complex reasoning.Response orders are randomly shuffled during testing.
  • Evaluation benchmarks: GenAI-Bench evaluates alignment with human preferences across image and video generation tasks.The image and video generation subsets are used for generative reward evaluation.
  • Evaluation benchmarks: VideoGen-RewardBench contains 26.5k human-labeled video pairs, with Overall Quality scores used for benchmarking.Each pair is ranked across multiple criteria.

F Regarding performance on “Tau” metric

The “Tau” metric evaluates behavior on tie cases, but UnifiedReward-Think is intentionally designed for discriminating between better and worse samples rather than handling ties. Its training excludes tie cases, so it is not optimized for that evaluation scenario.

  • “Tau” measures reward-model behavior when two images or videos are equally preferred.
  • UnifiedReward-Think targets fine-grained discrimination between high-quality candidates with subtle quality differences.The multidimensional CoT process is introduced to improve sensitivity to these differences.
  • Tie cases are excluded from training, leaving the method unoptimized for scenarios evaluated by “Tau”.This design prioritizes clear judgments between competing outputs for vision-based reinforcement learning.
Loading 2505.03318v3…