PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Reward Model for Multimodal Understanding and Generation

Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, Jiaqi Wang

arXiv:2503.05236v2cs.CV

TL;DR

Task-specific reward models do not readily adapt across diverse image and video understanding and generation tasks, motivating a unified evaluator. The paper introduces UNIFIEDREWARD with pairwise and pointwise assessment, uses it to filter preference data, and applies DPO; joint learning and downstream alignment improve performance across visual domains.

  • Problem

    Existing reward models are typically task-specific, and no unified model covers both image and video generation and understanding preference learning.

  • Method

    UNIFIEDREWARD jointly assesses diverse visual tasks with pair ranking and point scoring, then constructs preference pairs through pair ranking and point sifting for DPO alignment.

  • Results

    Joint learning across diverse visual tasks yields reciprocal benefits, while the alignment pipeline improves vision understanding and generation performance across domains.

  • Takeaways & Limitations

    A unified reward model can provide preference signals for multimodal understanding and generation alignment within one cross-task pipeline.

Abstract

from arXiv · show

Recent advances in human preference alignment have significantly improved multimodal generation and understanding. A key approach is to train reward models that provide supervision signals for preference optimization. However, existing reward models are often task-specific, limiting their adaptability across diverse visual applications. We also argue that a reward model that jointly learning to assess multiple vision tasks may foster a synergistic effect, where improved image understanding enhances image generation assessment, and refined image evaluation benefits video assessment through better frame analysis. To this end, this paper proposes UnifiedReward, the first unified reward model for multimodal understanding and generation assessment. It supports both pairwise ranking and pointwise scoring, providing effective reward signals for vision model preference alignment. Specifically, (1) we first train UnifiedReward on our constructed large-scale human preference dataset, which covers both image and video generation/understanding tasks. (2) Then, we leverage it to automatically construct high-quality pairwise preference data from vision models by progressively filtering their outputs through our two-stage strategy, i.e., pair ranking and point sifting. (3) Finally, we use these data to align vision models with human preferences via Direct Preference Optimization (DPO). Experimental results show that jointly learning to assess diverse visual tasks yields substantial mutual benefits. We further apply our pipeline to both vision understanding and generation, achieving consistent improvements across each domain.

I. INTRODUCTION

UNIFIEDREWARD addresses the limited adaptability of task-specific reward models by jointly assessing diverse image and video understanding and generation tasks. The paper uses this unified evaluator to construct preference data and align vision models with human preferences.

  • Existing reward models are often tailored to specific tasks, limiting adaptability across diverse visual understanding and generative applications.
  • Joint learning across visual tasks is intended to create mutually reinforcing benefits, including improved image-generation assessment and video-frame evaluation.
  • The paper proposes UNIFIEDREWARD, a unified reward model that supports both pairwise ranking and pointwise scoring across multimodal understanding and generation.
  • The pipeline trains UNIFIEDREWARD on a broad human preference dataset, filters vision-model outputs through pair ranking and point sifting, then applies Direct Preference Optimization.
  • Experiments report improvements across vision understanding and generation domains, while the framework aims to expand reward-model adaptability across visual applications.

III. METHOD

The method constructs a unified human-preference dataset spanning image and video understanding and generation, trains UNIFIEDREWARD on it, and uses the model to support preference alignment. Its design combines pairwise and pointwise assessment to produce broader and more reliable preference signals.

  • A. Overview: UNIFIEDREWARD is designed as a unified reward model for preference alignment across diverse visual tasks, supporting pair ranking and point scoring.
  • A. Overview: The trained reward model ranks and scores outputs from vision and diffusion models, enabling construction of preference pairs for subsequent model alignment.
  • 1) Unified Preference Dataset Construction:: The unified dataset integrates human-preference resources covering image and video generation and understanding, including both pairwise rankings and pointwise scores.
  • 1) Unified Preference Dataset Construction:: The dataset includes a relatively limited amount of video-generation pairwise-ranking data compared with other tasks.

2) Unified Preference Learning:

UNIFIEDREWARD is trained as an evaluation model for multimodal generation and understanding, using task-specific prompts with either pointwise or pairwise outputs. Its predictions then support preference-data construction and downstream alignment.

  • 2) Unified Preference Learning:: The model reuses a pretrained vision-language model’s visual comprehension while jointly learning evaluation across diverse vision tasks.
  • 2) Unified Preference Learning:: For generation assessment, the model receives vision tokens, an instruction, and a caption; for understanding assessment, the caption is replaced by a question.
  • 2) Unified Preference Learning:: Pointwise scoring evaluates one generated image or video against its caption, whereas pairwise ranking compares two generated outputs for the same caption.
  • 2) Unified Preference Learning:: For understanding assessment, pointwise and pairwise prompts evaluate one or two responses to a question about an image or video.
  • 2) Unified Preference Learning:: Training uses cross-entropy loss on the model’s predicted answer, with optional explanation generation when justifications are available.
  • 3. Generation/Understanding: After training, UNIFIEDREWARD supports preference-data construction before generation and understanding models are aligned using those data.

C. Preference Data Construction

The preference-data pipeline combines pairwise ranking with pointwise scoring to filter generated outputs into reliable preference pairs, which are then used for DPO alignment of generation and understanding models.

  • Preference filtering: Existing approaches often use only pairwise rankings or pointwise scores, whereas UnifiedReward combines both evaluation strategies for preference-data construction.The combination captures relative comparisons and absolute quality assessments.
  • Preference filtering: The pipeline generates multiple candidate outputs from image/video-question pairs or generation prompts, then groups them into pairs for ranking.Ranked outputs are divided into chosen and rejected lists.
  • Preference filtering: Point sifting assigns scores to outputs in both lists, selecting the highest-scoring chosen output and lowest-scoring rejected output as the final preference pair.S(O) denotes the model’s pointwise score.
  • DPO alignment: The resulting preference pairs are used with DPO to align multimodal generation and understanding models directly with human preferences.DPO optimizes from ranked preference pairs without explicit reward modeling.
  • DPO alignment: For diffusion generation, DPO compares denoising errors between preferred and less-preferred samples, reducing error for preferred outputs and increasing it for less-preferred ones.The optimization uses differences between fine-tuned and pretrained reference diffusion models.
  • DPO alignment: For multimodal understanding, DPO increases the likelihood of preferred responses and decreases the likelihood of less-preferred responses.The objective uses response probabilities from the fine-tuned and reference models.

A. Implementation Details

The implementation builds UnifiedReward on pretrained vision-language models and applies DPO to image and video understanding and generation models using task-specific training configurations.

  • Reward model: UnifiedReward uses pretrained LLaVA-OneVision 7B as its base architecture and is additionally trained with Qwen2.5-VL to test robustness.Training uses 8 H100 GPUs, while inference takes approximately 1 second for direct answers and 3 seconds with brief rationales.
  • Understanding alignment: For multimodal understanding, DPO is applied to LLaVA-OneVision 7B and LLaVA-Video for image and video understanding, respectively.The understanding optimization uses βu = 0.1.
  • Generation alignment: For multimodal generation, SDXL-Turbo is used for image generation and T2V-Turbo for video generation.The generation optimization sets βg to 5000, with batch sizes of 32 and 16, respectively.
  • Generation alignment: The generation experiments construct 10K preference examples for video generation and 14K for the other DPO tasks, using N = 10 candidate outputs.All models are trained for 3 epochs.
  • Qualitative comparisons: The qualitative comparisons evaluate original models against DPO models trained with VideoDPO or UnifiedReward for T2V-Turbo, and Pick-a-Pic or UnifiedReward for SDXL-Turbo.These comparisons are presented for video and image generation, respectively.

B. Reward Model Comparison Results

UNIFIEDREWARD outperforms baselines across most reward-model evaluation metrics and supports strong assessment of image and video generation and understanding. Its gains extend to DPO alignment, where constructed preference data improve vision-model performance across domains.

  • Image Understanding: UNIFIEDREWARD outperforms baselines on most image-understanding assessment metrics, including macro accuracy.
  • Multimodal Generation: UNIFIEDREWARD excels across all reported image- and video-generation assessment metrics despite relatively limited video-generation training data.The comparison uses the generation metrics defined in Table IV, including TAU and DIFF accuracy variants.
  • Vision Understanding DPO: DPO with UNIFIEDREWARD consistently outperforms the LLaVA-Critic baseline across image-understanding benchmarks, improving LLaVABench by 3.4%.
  • Vision Understanding DPO: UNIFIEDREWARD-based DPO remains comparable on several video-understanding multiple-choice datasets and significantly outperforms baselines on MSRVTT, MSVD, and TGIF.The DPO data did not include the multiple-choice question type, yet performance remained comparable there.
  • Generation DPO: UNIFIEDREWARD-based DPO improves image-generation performance over direct training on Pick-a-Pic and enhances both quality and semantic consistency in video generation.Image generation is evaluated with image-reward metrics, while video generation is evaluated on VBench.
  • Multitask Learning: Jointly learning across image and video understanding and generation produces reciprocal benefits across individual reward-model tasks.The study varies training-data configurations across modalities and task types to examine this effect.

2) Cross-Task Synergy Beyond Scaling Data:

UNIFIEDREWARD’s advantage persists under update-budget matching, supporting positive cross-task synergy rather than gains explained solely by more data or longer optimization. Cross-domain transfer is broadly positive, while data imbalance can harm underrepresented tasks.

  • Budget-Matched Control: UNIFIEDREWARD achieves the best performance across all evaluation axes even when single-task models are oversampled to match its update budget.The control compares native single-task models with step-matched models.
  • Cross-Domain Transfer: Cross-task promotion occurs because understanding-centric training also improves generation evaluation, while single-domain models generalize best within their own modality.UNIFIEDREWARD remains consistently strong across all targets and reduces modality-specific weaknesses.
  • Backbone Robustness: Larger backbones preserve UNIFIEDREWARD’s improvement trend, indicating compatibility with stronger visual-language model priors.The comparison evaluates models trained on LLaVA-OneVision and Qwen2.5-VL.
  • Data Balance: Increasing nonvideo-generation data while fixing video-generation data improves over-represented tasks but consistently degrades video-generation performance.Rebalancing the video-generation data restores its performance, showing sensitivity to cross-task data imbalance.
  • Preference Signals: UNIFIEDREWARD outperforms GPT-4o and an image-generation-only reward model under matched DPO training budgets.The comparison replaces only the preference-signal source while keeping the DPO backbone and training pipeline fixed.
  • Preference Construction: The full pair-ranking and point-sifting pipeline outperforms random selection and either stage alone, because the two stages provide complementary filtering functions.Pair ranking orders candidates relatively, while point scoring removes low-quality responses that survive pairwise comparison.

7) Applied to Image Generation GRPO:

Applying UNIFIEDREWARD to GRPO on FLUX.1-dev yields stronger reward-guided generation than the vanilla baseline and alternative reward signals. Qualitative results associate this advantage with better prompt faithfulness, cleaner composition, and more coherent fine-grained detail.

  • GRPO Results: UNIFIEDREWARD consistently improves all reported evaluation views over the vanilla FLUX.1-dev baseline under GRPO.It remains strongest overall compared with alternative reward signals.
  • Qualitative Comparison: UNIFIEDREWARD-optimized FLUX samples show better prompt faithfulness, cleaner compositions, and more coherent fine-grained details than baseline and other reward variants.The qualitative comparison is presented in Figure 6.
  • Transfer Across Optimization Methods: The agreement between quantitative and qualitative comparisons supports using UNIFIEDREWARD as an optimization signal beyond DPO.The paper applies the reward model to group relative policy optimization on FLUX.1-dev.
  • Pipeline Context: The broader alignment pipeline combines unified reward assessment, two-stage preference filtering, and DPO across vision understanding and generation tasks.The conclusion describes the same pipeline and reports improvements in each applied domain.

APPENDIX A MORE IMPLEMENTATION DETAILS

The appendix documents reward-model baselines, evaluation benchmarks, and additional qualitative comparisons across image and video generation. It also establishes the paper’s novelty claim and describes the multimodal datasets used for assessment.

  • Reward Model Baselines: Baseline reward models include PickScore, HPSv2, ImageReward, and LLaVA-Critic, covering image generation and image understanding assessment.PickScore combines a CLIP-style model with a reward-model objective, while LLaVA-Critic supports pair ranking and point scoring.
  • Reward Model Baselines: The paper identifies UNIFIEDREWARD as the first unified reward model for multimodal understanding and generation assessment.
  • Qualitative Comparisons: Additional qualitative comparisons examine DPO on SDXL-Turbo and T2V-Turbo using UnifiedReward against original models and alternative preference datasets.Figures 8 and 9 compare SDXL-Turbo and T2V-Turbo variants, respectively.
  • Multimodal Understanding: Evaluation covers image and video understanding with VLRewardBench and ShareGPTVideo, and generation with GenAI-Bench and VideoGen-RewardBench.
  • Multimodal Understanding: VLRewardBench assesses general multimodal queries, visual hallucination detection, and complex reasoning tasks.The benchmark contains 1,250 high-quality examples.
  • Multimodal Understanding: ShareGPTVideo contains 900k captions and 17k preference examples; this work uses 16k for reward-model training and 1k for video-understanding evaluation.

2) Multimodal Generation:

The multimodal generation evaluation compares reward models using established image- and video-generation benchmarks, with baselines aligned through matched datasets and training settings. The paper also reports additional qualitative results and plans to release evaluation code for reproducibility.

  • Generation Reward Evaluation: GenAI-Bench evaluates multimodal generative-model rewards by comparing model judgments of AI-generated content with human preferences.The evaluation uses its image- and video-generation portions.
  • Generation Reward Evaluation: VideoGen-RewardBench contains 26.5k manually constructed video pairs evaluated for Visual Quality, Motion Quality, Text Alignment, and Overall Quality.The paper uses Overall Quality for baseline reward comparison.
  • Reproducibility and Qualitative Results: The evaluation code will be released to facilitate community reproduction, and additional qualitative results appear in Figures 8 and 9.These resources complement the benchmark-based comparisons.
  • DPO Baselines: The DPO comparisons match datasets and settings across methods, including LLaVA-RLHF image-question pairs, ShareGPTVideo preferences, released VideoDPO data, and sampled Pick-a-Pic captions.Pick-a-Pic provides around 851k preference pairs after excluding approximately 12% tied pairs, while the paper samples 14k captions for its own preference data.
  • Broader Applications: The unified reward model is presented as applicable across domains including digital media, entertainment, education, and accessibility.The stated motivation is to align generated content more closely with human preferences and improve vision-model quality and reliability.
Loading 2503.05236v2…