PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Reward Model for Multimodal Understanding and Generation

Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, Jiaqi Wang

arXiv:2503.05236v2cs.CV

TL;DR

Reward models are often task-specific, and no unified model covers preference learning across image and video generation and understanding. UnifiedReward combines joint reward assessment with preference-data construction and DPO; experiments report synergistic improvement across visual domains.

  • Problem

    Existing reward models are often task-specific, and no unified model covers preference learning across image and video generation and understanding tasks.

    Source: §II · p. 2

  • Method

    UnifiedReward is trained on a unified preference dataset, then used with pair ranking and point sifting to curate data for DPO alignment.

    Source: §1 · p. 1, A. Overview · p. 2

  • Results

    Experiments report that jointly assessing image and video tasks yields synergistic performance improvements across visual domains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pipeline achieves reported improvements in both vision understanding and generation model alignment.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recent advances in human preference alignment have significantly improved multimodal generation and understanding. A key approach is to train reward models that provide supervision signals for preference optimization. However, existing reward models are often task-specific, limiting their adaptability across diverse visual applications. We also argue that a reward model that jointly learning to assess multiple vision tasks may foster a synergistic effect, where improved image understanding enhances image generation assessment, and refined image evaluation benefits video assessment through better frame analysis. To this end, this paper proposes UnifiedReward, the first unified reward model for multimodal understanding and generation assessment. It supports both pairwise ranking and pointwise scoring, providing effective reward signals for vision model preference alignment. Specifically, (1) we first train UnifiedReward on our constructed large-scale human preference dataset, which covers both image and video generation/understanding tasks. (2) Then, we leverage it to automatically construct high-quality pairwise preference data from vision models by progressively filtering their outputs through our two-stage strategy, i.e., pair ranking and point sifting. (3) Finally, we use these data to align vision models with human preferences via Direct Preference Optimization (DPO). Experimental results show that jointly learning to assess diverse visual tasks yields substantial mutual benefits. We further apply our pipeline to both vision understanding and generation, achieving consistent improvements across each domain.

I. INTRODUCTION · 1. Unified Reward Model Training · II. RELATED WORK

UnifiedReward addresses task-specific reward models with a model for image and video understanding and generation, trained on broad human preference data and used in a preference-alignment pipeline. The paper reports reciprocal gains from joint assessment across visual tasks and improvements in both understanding and generation.

I. INTRODUCTION

  • Human preference collection is resource-intensive, while existing reward models are often task-specific and lack a comprehensive preference dataset spanning visual tasks.The paper motivates learned reward models as a way to generate synthetic preferences from vision-model outputs, reducing the need for extensive human annotation.Source: §I · p. 1, §1 · p. 1

1. Unified Reward Model Training

  • UnifiedReward is presented as the first reward model for multimodal understanding and generation that supports both pairwise ranking and pointwise scoring.Its preference dataset spans image and video generation and understanding tasks.Source: §1 · p. 1
  • The pipeline uses UnifiedReward to filter VLM and diffusion-model outputs through pair ranking and point sifting, then aligns the models with human preferences using DPO.The process first trains the reward model on a large-scale human preference dataset covering image and video generation and understanding.Source: §1 · pp. 1–2
  • Joint assessment of multiple visual tasks yields reciprocal gains, and the alignment pipeline improves vision models in both understanding and generation.The authors report that cross-task learning improves performance across individual visual domains.Source: §1 · p. 2, §II · p. 2

II. RELATED WORK

  • Traditional FID and CLIP scores have limits in capturing human preferences; preference-tuned CLIP – and VLMs [21], [22] offer alternatives, with VLM reward models split between understanding [5], [6] and generation assessment [4], [7], [12], [23].Source: §II · p. 2
  • Prior preference-learning work addresses video understanding [24], [3], [9], [25], image understanding [25],, [27], image generation, [8], [12], and video generation [2], [4], [7], [12], [28]–[31].These approaches use human feedback, AI-generated feedback, or reward models to refine understanding systems or align diffusion models.Source: §II · p. 2
  • The related-work gap is that preference learning across image and video understanding and generation lacks a unified reward model, limiting alignment generalizability and efficiency.The paper investigates whether joint assessment can improve evaluation across these domains.Source: §II · p. 2

III. METHOD · A. Overview · 1) Unified Preference Dataset Construction:

UnifiedReward is trained on a large-scale human-preference dataset spanning image and video understanding and generation, then used to curate model preferences and align vision models with human preferences through DPO.

A. Overview

  • UnifiedReward supports pair ranking and point scoring across visual tasks, and the pipeline uses it to curate model outputs before DPO alignment.The pipeline constructs and trains the reward model, applies pair ranking and point sifting to VLM and diffusion-model outputs, then uses the curated datasets for DPO.Source: A. Overview · p. 2

1) Unified Preference Dataset Construction:

  • The first large-scale unified human-preference dataset contains approximately 236K samples across image and video understanding and generation, addressing the lack of comprehensive task coverage.Existing feedback datasets are typically task-specific [2], [4], [5].Source: 1) Unified Preference Dataset… · p. 3
  • Image-generation data combines EvalMuse [32] ratings and element labels, HPD [33] vote-count rankings, and OIP preference pairs.EvalMuse ratings are averaged for pointwise scores, and its highest- and lowest-rated images per prompt form ranking pairs.Source: 1) Unified Preference Dataset… · p. 3
  • For image understanding, LLava-Critic-113K [5] supplies 25K selected samples each for pairwise ranking and pointwise scoring.The source dataset contains 40K pointwise and 73K pairwise samples.Source: 1) Unified Preference Dataset… · p. 3
  • Video-generation assessment uses VideoDPO [2] for pairwise ranking and incorporates LiFT-HRA [4] and VideoFeedback [23] for pointwise scoring.VideoDPO provides 10K synthesized video pairs.Source: 1) Unified Preference Dataset… · p. 3
  • Video-understanding data from ShareGPTVideo-DPO [34] supports pairwise ranking from response pairs and pointwise scoring from individual response scores.The dataset contains 17K video-understanding DPO examples.Source: 1) Unified Preference Dataset… · p. 3
  • Pairwise answers are standardized as preference statements, while pointwise scoring retains diverse rating styles and ranges; prompts are adjusted to match evaluation criteria.Evaluation justifications are retained when available [4], [5].Source: 1) Unified Preference Dataset… · p. 3
  • Video-generation pairwise data is relatively limited, but the authors expect multitask learning to help mitigate this deficiency.The overall collection covers pairwise ranking and pointwise scoring across image and video understanding and generation.Source: 1) Unified Preference Dataset… · p. 3

2) Unified Preference Learning: … 3. Generation/Understanding

UnifiedReward adds assessment to a pretrained vision-language model with strong visual understanding, training it across multimodal generation and understanding tasks. The paper uses its evaluations to construct preference data and align vision models with human preferences through DPO.

2) Unified Preference Learning:

  • UnifiedReward is fine-tuned from a pretrained VLM with strong vision understanding, adding evaluation as a discriminative skill across diverse vision tasks.The approach leverages the model’s existing visual comprehension rather than learning evaluation from scratch.Source: 2) Unified Preference Learning · p. 3
  • For generation evaluation, the model receives vision tokens, an instruction, and a caption; for understanding, the caption is replaced by a question.Source: 2) Unified Preference Learning · p. 3

1. Unified Reward Model Training

  • Training covers pointwise scoring of a generated image or video and pairwise ranking of two generated outputs given a caption.Source: §1 · p. 4

2. Preference Data Construction

  • For understanding preference data, pointwise prompts review a response to an image or video question, while pairwise prompts compare two responses.Source: §2 · p. 4

3. Generation/Understanding

  • Preference data construction uses vision-model outputs, pairwise ranking to select chosen and rejected outputs, and pointwise filtering to refine those samples.Source: Figure 3 · p. 4
  • The constructed preference data is used to align multimodal understanding and generation models with human preferences via Direct Preference Optimization.The alignment pipeline follows preference data construction with model alignment.Source: Figure 3 · p. 4, §3 · p. 4
  • The model predicts a pointwise score or pairwise ranking from the instruction-specified criteria, and generates evaluation explanations when justifications are included in the training data.Training uses standard cross-entropy loss computed only on the model’s predicted answer.Source: §3 · p. 4

C. Preference Data Construction · D. Generation/Understanding Model Alignment

The sections describe a preference-data pipeline combining pairwise ranking with pointwise sifting, followed by DPO alignment of multimodal generation and understanding models using ranked preferences.

C. Preference Data Construction

  • The pipeline generates multiple candidates for each prompt, ranks them in N/2 pairs, and separates each pair into chosen and rejected outputs.Candidates come from a VLM for image/video-question inputs or a diffusion model for generation prompts.Source: C. Preference Data Construction · p. 4
  • Point sifting scores the chosen and rejected outputs, then forms the final preference pair from the most- and least-preferred outputs.The model assigns pointwise scores across both lists.Source: C. Preference Data Construction · p. 4
  • Combining pairwise ranking and pointwise scoring is intended to produce reliable preference signals that capture both relative comparisons and absolute quality.The paper presents this as a higher-quality alternative to approaches limited to one evaluation strategy.Source: C. Preference Data Construction · p. 4

D. Generation/Understanding Model Alignment

  • DPO uses the constructed ranked preference pairs to align multimodal generation and understanding models with human preferences without explicit reward modeling.The optimization acts directly on preferred and less-preferred outputs or responses.Source: D. Generation/Understanding Model… · p. 5
  • For diffusion-based image and video generation, DPO compares fine-tuned and reference-model noise predictions, reducing denoising error for preferred samples and increasing it for less-preferred ones.The paper applies this approach to diffusion models used for image and video synthesis.Source: D. Generation/Understanding Model… · p. 5
  • For multimodal understanding, DPO increases the VLM’s likelihood of preferred responses and decreases it for less-preferred responses to improve human-preference alignment and reasoning quality.The preference pairs consist of responses to inputs such as image/video-question pairs.Source: D. Generation/Understanding Model… · p. 5

IV. EXPERIMENTS · A. Implementation Details

UnifiedReward and its DPO applications use specified model, training, and data configurations, with reward assessment and aligned models evaluated on image and video benchmarks.

A. Implementation Details

  • UNIFIEDREWARD uses pre-trained LLaVA-OneVision 7B, trains on 8 H100 GPUs, and runs inference in ∼1s for direct answers or ∼3s with brief rationales.Training uses batch size 2, gradient accumulation 16, learning rate 2.5×10−6, and warm-up ratio 0.3; Qwen2.5-VL is also used to test robustness across baselines.Source: A. Implementation Details · p. 5
  • Understanding DPO applies UNIFIEDREWARD to LLaVA-OneVision 7B for image understanding and LLaVA-Video for video understanding.Both use batch size 1, gradient accumulation 16, learning rate 5 × 10−7, and βu = 0.1.Source: A. Implementation Details · p. 5
  • Generation DPO uses SDXL-Turbo for images and T2V-Turbo for video, with βg = 5000, candidate count N = 10, and 3 training epochs.Batch sizes are 32 for SDXL-Turbo and 16 for T2V-Turbo; the preference data contains 10K examples for video generation and 14k for other DPO tasks.Source: A. Implementation Details · p. 5
  • Reward assessment is evaluated on VLRewardBench for image understanding, ShareGPTVideo (1K test samples) for video understanding, and GenAI-Bench for image and video generation.Video generation assessment also uses VideoGen-RewardBench.Source: A. Implementation Details · p. 5
  • Image-understanding DPO is evaluated on LLaVABench, WildVision, LLaVABench-Wilder, LiveBench, MMHal, MMBench, MME, MathVista, DocVQA, and TextVQA using LMMs-Eval.Source: A. Implementation Details · p. 5
  • Video-understanding DPO uses gpt-3.5-turbo-1106 for MSRVTT, MSVD, and TGIF, and VLMEvalKit for LongVideoBench, MLVU, and VideoMME.Source: A. Implementation Details · p. 5
  • Image-generation evaluation uses captions from Partiprompt (1632 captions) and HPSv2 (3200 captions), while qualitative DPO comparisons show T2V-Turbo and SDXL-Turbo against their respective baseline datasets.The figure captions identify comparisons with VideoDPO for T2V-Turbo and Pick-a-Pic for SDXL-Turbo, without specifying a winner.Source: A. Implementation Details · p. 5, Figure 4 · p. 6, Figure 5 · p. 6

B. Reward Model Comparison Results … 1) Multi-task Assessment Learning:

UnifiedReward’s multi-task assessment and preference-data pipeline improve reward-model performance and vision-model alignment across image and video tasks. The reported gains include stronger benchmark results and better generation quality and semantic consistency.

B. Reward Model Comparison Results

  • UnifiedReward outperforms baselines on most image-understanding metrics, including macro accuracy, and performs relatively better than separate-task VisionReward models for image generation.The comparison includes LLaVA-Critic and two closed-source models for image understanding; VisionReward trains separate image- and video-generation models.Source: B. Reward Model Comparison Results · p. 6
  • Despite relatively limited video-generation assessment data, UnifiedReward excels across all metrics against the compared baselines.The paper attributes the gains to multi-task learning mitigating insufficient training data and enhancing video-generation assessment.Source: Table IV · p. 6, B. Reward Model Comparison Results · p. 6

C. DPO Comparison Results

  • UNIFIEDREWARD-based DPO significantly outperforms Houd-DPO and TPO on MSRVTT, MSVD, and TGIF, while remaining comparable on three other multi-choice datasets.The DPO data do not include the multi-choice question type, but this does not negatively affect performance.Source: C. DPO Comparison Results · pp. 6–7, Table V · p. 7
  • UNIFIEDREWARD-based DPO achieves a 3.4% improvement on LLaVABench and consistently outperforms LLaVA-Critic across image-understanding benchmarks.The comparison uses the same image-question pair source for OV-7B preference data.Source: C. DPO Comparison Results · p. 6, Table VI · p. 7
  • UNIFIEDREWARD-based DPO improves video generation quality and semantic consistency over VideoDPO using the same prompt source.The comparison uses the same prompt source for preference-data construction.Source: Table VII · p. 7, C. DPO Comparison Results · p. 7
  • Training on UNIFIEDREWARD-constructed preference data achieves better image-generation performance than directly training on the original Pick-a-Pic dataset.The preference data are constructed from Pick-a-Pic prompts.Source: Table VII · p. 7, C. DPO Comparison Results · p. 7

1) Multi-task Assessment Learning:

  • Jointly training image and video understanding improves overall accuracy by 5.3% and macro accuracy by 8.3% over training only on image understanding.The study evaluates single-task and cross-task configurations spanning image and video modalities and understanding and generation tasks; results for other tasks also consistently show multi-task effectiveness.Source: 1) Multi-task Assessment Learning · p. 7

2) Cross-Task Synergy Beyond Scaling Data: … 6) Data Construction Strategy Ablation:

UnifiedReward shows positive cross-task transfer beyond the effects of training budget, and its strengths persist across backbones and data compositions. Its preference signals and two-stage data-construction strategy also improve downstream alignment over the tested alternatives.

2) Cross-Task Synergy Beyond Scaling Data:

  • With update steps matched, UnifiedReward still leads every evaluation axis, showing gains from positive cross-task synergy rather than more data or longer optimization.The control compares native single-task models with step-matched models oversampled to the same total update steps as UnifiedReward.Source: 2) Cross-Task Synergy Beyond Scaling… · pp. 7–8, Table IX · p. 8
  • Single-domain training transfers positively to other modalities but performs best within its own; UnifiedReward remains strong across targets while reducing modality-specific overfitting.Understanding-centric training also improves generation evaluation.Source: Table IX · p. 8, 2) Cross-Task Synergy Beyond Scaling… · p. 8

3) Robustness on Different Baselines:

4) Impact of Imbalanced Training Data:

  • Increasing nonvideo-generation data while holding video-generation data fixed improves over-represented tasks but degrades video generation; rebalancing restores its performance.The other tasks remain largely stable after video-generation data is rebalanced, supporting balanced sampling to maintain mutual gains.Source: Table XI · p. 8, 4) Impact of Imbalanced Training Data · p. 8

5) Preference Signal Source Comparison:

  • For SDXL [39] DPO pairs, UnifiedReward signals outperform GPT-4o and an image-generation-only reward model across evaluation dimensions.The comparison holds the DPO backbone and training pipeline fixed; qualitative cases also show better prompt alignment, cleaner compositions, and fewer visible artifacts.Source: 5) Preference Signal Source Comparison · p. 8

6) Data Construction Strategy Ablation:

  • The full two-stage construction pipeline beats random selection, point-score-only, and pair-rank-only variants, with pair ranking and point scoring serving complementary roles.Pair ranking orders candidates relatively, while point scoring filters low-quality responses that can survive pairwise comparison, producing cleaner, more stable preference pairs and stronger downstream alignment.Source: 6) Data Construction Strategy Ablation · p. 8

7) Applied to Image Generation GRPO: · V. ETHICAL STATEMENT · VI. CONCLUSION

UNIFIEDREWARD improves FLUX.1-dev generation when used with GRPO, and the paper reports no ethical concerns. The conclusion presents a unified reward model and preference-alignment pipeline, with improvements across visual tasks.

7) Applied to Image Generation GRPO:

V. ETHICAL STATEMENT

  • The authors state that, to their knowledge, the study involves no data, methods, or applications raising ethical concerns and follows established ethical guidelines.They also affirm their commitment to ethical research practices and responsible innovation.Source: §V · p. 9

VI. CONCLUSION

  • The conclusion describes UNIFIEDREWARD as a unified model supporting pair ranking and point scoring, trained on a broad visual-task dataset and used to construct preference pairs through pair ranking and point sifting.The constructed data are then used to align vision models via direct preference optimization.Source: §VI · p. 9
  • The authors conclude that joint learning across visual tasks yields mutual benefits, with substantial improvements in image and video understanding and generation.Source: §VI · p. 9

APPENDIX A MORE IMPLEMENTATION DETAILS · A. Reward Model Baselines · B. Evaluation Benchmarks

Appendix A describes UnifiedReward’s model and training context, the reward-model baselines, and the benchmarks used for multimodal evaluation.

A. Reward Model Baselines

  • Image-generation baselines include PickScore v1 (Pick-a-Pic, CLIP-style plus InstructGPT reward objective), HPSv2 (CLIP fine-tuned on HPD v2), and ImageReward (137k expert comparisons).Source: A. Reward Model Baselines · p. 12
  • LLaVA-Critic is the image-understanding baseline, using an LLM-based critic trained on critic-instruction data and supporting pair ranking and point scoring.Source: A. Reward Model Baselines · p. 12
  • VideoScore and LiFT-Critic serve as video-generation quality baselines, trained respectively using VideoFeedback scores for 37.6K videos and approximately 20k LiFT-HRA annotations.Source: A. Reward Model Baselines · p. 12
  • VisionReward supplies separate image and video reward models, while VideoReward is a multidimensional video reward model trained on a human-labeled 182k-preference dataset.Source: A. Reward Model Baselines · p. 12
  • The authors describe UnifiedReward as the first unified reward model for multimodal understanding and generation assessment, while Figure 8 qualitatively compares original SDXL-Turbo with DPO trained on Pick-a-Pic and UnifiedReward.Source: A. Reward Model Baselines · p. 12, Figure 8 · p. 12

B. Evaluation Benchmarks

  • Evaluation uses VLRewardBench for image understanding, ShareGPTVideo (1K test samples) for video understanding, GenAI-Bench for image and video generation, and VideoGen-RewardBench for video generation.Source: B. Evaluation Benchmarks · p. 12

1) Multimodal Understanding: … APPENDIX C SOCIETAL IMPACTS

The supplied sections describe evaluation benchmarks and DPO baseline settings for multimodal understanding and generation, then provide further qualitative comparisons and discuss potential societal benefits of preference-aligned vision models.

1) Multimodal Understanding:

  • VLRewardBench evaluates image understanding across multimodal queries, hallucination detection, and reasoning, while ShareGPTVideo supplies video data used for reward-model training and evaluation.VLRewardBench contains 1,250 examples; from ShareGPTVideo's 17k preference examples, the work uses 16k for reward-model training and 1k for video-understanding evaluation.Source: 1) Multimodal Understanding · p. 12

APPENDIX B MORE QUALITATIVE COMPARISON

  • The appendix points to further qualitative DPO comparisons of original T2V-Turbo, VideoDPO-trained DPO, and UnifiedReward-trained DPO.These comparisons are shown in Figures 8 and 9.Source: Figure 9 · p. 13, Appendix B · p. 14

2) Multimodal Generation:

  • GenAI-Bench evaluates image-generation, image-editing, and video-generation rewards against human preferences; this work uses its image- and video-generation portions.The benchmark assesses MLLM judgments of AI-generated content by comparing them with human preferences.Source: 2) Multimodal Generation · p. 13
  • VideoGen-RewardBench contains 26.5k manually constructed video pairs rated on four quality dimensions, and this work uses Overall Quality for baseline reward comparison.The dimensions are Visual Quality, Motion Quality, Text Alignment, and Overall Quality; the authors also say they will release evaluation code.Source: 2) Multimodal Generation · p. 13

C. DPO Baselines

  • For DPO baselines, the study matches LLaVA-Critic's image-question data and settings, and applies LLaVA-Houd-DPO's ShareGPTVideo data to LLaVA-Video.The UnifiedReward approach samples 14k of ShareGPTVideo's 17k preference examples for LLaVA-Video DPO, keeping training settings identical for fairness.Source: C. DPO Baselines · p. 13
  • Because LLaVA-TPO's training data is unavailable, the study compares against its released checkpoint.LLaVA-TPO uses curated preference data at localized and comprehensive temporal-grounding granularities.Source: C. DPO Baselines · pp. 13–14
  • VideoDPO's released T2V-Turbo preference data and Pick-a-Pic preferences serve as DPO baselines, while UnifiedReward constructs data from video-caption pairs and 14k sampled captions, respectively.Pick-a-Pic contains around 851k preference pairs and 58.9k unique prompts after tied pairs are excluded; its data is used directly for SDXL-Turbo DPO in the baseline.Source: C. DPO Baselines · p. 14

APPENDIX C SOCIETAL IMPACTS

  • The authors say preference alignment could improve vision-model quality and reliability across digital media, entertainment, education, and accessibility, while enabling more controlled AI-assisted creativity.They also identify potential benefits for artists, designers, and content creators through more consistent and interpretable generative-model evaluation.Source: Appendix C · p. 14

Related papers

Loading 2503.05236v2…