PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Personalized Reward Model for Vision Generation

Yibin Wang, Yuhang Zang, Feng Han, Jiazi Bu, Yujie Zhou, Cheng Jin, Jiaqi Wang

arXiv:2602.02380v2cs.CV

TL;DR

Existing vision-generation reward models often use a one-size-fits-all assessment that misses content-specific visual cues. UnifiedReward-Flex dynamically adapts hierarchical evaluation criteria to prompt intent and visual content, and it consistently outperforms strong baselines on image and video reward tasks while improving downstream generation quality.

  • Problem

    Existing reward models assume shared global preferences or fixed evaluation rubrics, limiting sensitivity to content-specific visual cues.

  • Method

    UnifiedReward-Flex interprets prompt intent and visual evidence, then dynamically composes hierarchical criteria with fine-grained sub-criteria and additional dimensions when needed.

  • Results

    UnifiedReward-Flex consistently outperforms strong reward-model baselines on image and video tasks and improves downstream image and video generation quality.

  • Takeaways & Limitations

    Context-adaptive reward assessment provides more robust, context-aware signals for preference-based optimization of vision generators.

  • Takeaways & Limitations

    UnifiedReward-Flex incurs higher computation costs because it performs context-personalized reasoning beyond fixed evaluation criteria.

Abstract

from arXiv · show

Recent advancements in multimodal reward models (RMs) have significantly propelled the development of visual generation. Existing frameworks typically adopt Bradley-Terry-style preference modeling or leverage generative VLMs as judges, and subsequently optimize visual generation models via reinforcement learning. However, current RMs suffer from inherent limitations: they often follow a one-size-fits-all paradigm that assumes a monolithic preference distribution or relies on fixed evaluation rubrics. As a result, they are insensitive to content-specific visual cues, leading to systematic misalignment with subjective and context-dependent human preferences. To this end, inspired by human assessment, we propose UnifiedReward-Flex, a unified personalized reward model for vision generation that couples reward modeling with flexible and context-adaptive reasoning. Specifically, given a prompt and the generated visual content, it first interprets the semantic intent and grounds on visual evidence, then dynamically constructs a hierarchical assessment by instantiating fine-grained criteria under both predefined and self-generated high-level dimensions. Our training pipeline follows a two-stage process: (1) we first distill structured, high-quality reasoning traces from advanced closed-source VLMs to bootstrap SFT, equipping the model with flexible and context-adaptive reasoning behaviors; (2) we then perform direct preference optimization (DPO) on carefully curated preference pairs to further strengthen reasoning fidelity and discriminative alignment. To validate the effectiveness, we integrate UnifiedReward-Flex into the GRPO framework for image and video synthesis, and extensive results demonstrate its superiority.

1 Introduction

Existing vision-generation reward models rely on fixed or globally shared assessment paradigms, limiting sensitivity to prompt- and content-specific preferences. UnifiedReward-Flex addresses this limitation with hierarchical, context-adaptive reasoning and consistently outperforms strong baselines on image and video reward tasks.

  • Related Work: Earlier approaches progressed from fixed discriminative scorers to Bradley–Terry pairwise preference models and VLM-as-a-judge evaluations for reinforcement learning.These approaches respectively assign scalar rewards, learn from relative comparisons, or use generative VLMs for context-dependent evaluation.
  • Contributions: UnifiedReward-Flex addresses the “one-size-fits-all” limitation by dynamically composing personalized evaluation hierarchies through content-aware reasoning.It is designed to provide more robust and context-aware reward signals for vision generation.
  • Motivation: Prompts with narrative intent require criteria such as storytelling consistency, subject relationships, and emotional tone, while motion-intensive videos require action dynamics and physical plausibility.These examples motivate adapting assessment to both prompt intent and visual content, mirroring human evaluation behavior.
  • Method: Given a prompt and generated visual content, the model interprets semantic intent, extracts salient visual evidence, and hierarchically instantiates fine-grained evaluation criteria.This couples reward modeling with context-adaptive reasoning to tailor assessment criteria.
  • Results: UnifiedReward-Flex consistently outperforms strong reward-model baselines on both image and video reward tasks.The reported results characterize its reward signals as more robust and context-aware.

A. Semantic Alignment & Accuracy

Image 2 achieves stronger semantic alignment than Image 1 by including both requested subjects and depicting their healing interaction, while both images equally capture the intended atmosphere.

  • Subject Completeness: Image 2 fulfills the core subject requirements with both the child and kirin (Score: 9), whereas Image 1 omits the child entirely (Score: 1).The category reasoning attributes the difference to Image 2 following the prompt’s subject instructions while Image 1 ignores the child.
  • Action Fidelity: Image 2 visualizes the requested healing interaction through the child reaching toward the creature (Score: 9), while Image 1 remains a static portrait without interaction (Score: 1).
  • Atmosphere: Both images balance the grimdark setting with luminous highlights and glow equally well, each receiving a score of 5.

C. Aesthetics & Artistry

Image #2 demonstrates stronger composition through dramatic scale contrast, while both images maintain high-fantasy anime style coherence. Image #1’s standard close-up framing is less dynamic.

  • Composition: Image #2 uses scale contrast between the massive kirin and small child to create a balanced, dramatic composition, scoring 7.Image #1’s standard close-up headshot lacks dynamic arrangement and scores 3.
  • Style Coherence: Both images fit the high-fantasy anime aesthetic well, with each receiving a style-coherence score of 5.Image #2 is described as resembling a high-end light novel cover.

B. Image Quality & Realism

Image 1 edges out Image 2 in technical rendering, especially 3D shading and hyper-detailed texture, while both images achieve equally strong color vibrancy and contrast. Image 1 also has more dynamic lighting, whereas Image 2’s lighting leaves peripheral details less distinct.

  • Image Quality & Realism: Image 1’s dynamic backlighting better accentuates the silhouette and horn transparency than Image 2’s centralized lighting.Image 1 scores 6 for lighting, while Image 2 scores 4 because peripheral details are less distinct.
  • Image Quality & Realism: Both images receive a color vibrancy and contrast score of 5, using distinct high-contrast palettes.Image 1 combines rich blues and magentas, while Image 2 contrasts a white creature, dark void, and warm fire or magic colors.
  • Image Quality & Realism: Image 1 edges out Image 2 in technical rendering, particularly for 3D shading and hyper-detailed texture.Image 1 scores 7 for crisp, hyper-realistic scales and fur, compared with Image 2’s score of 3 for softer, painterly texture.

D. Narrative & Interaction

UnifiedReward-Flex favors Image 2 because it semantically aligns with the prompt’s child-healing-kirin interaction, conveys trust, magic, and majestic scale, and outweighs Image 1’s superior texture rendering.

  • D. Narrative & Interaction: Image 2 evokes trust and magic through the child’s gentle gesture toward the beast, scoring 8 versus Image 1’s score of 2.
  • D. Narrative & Interaction: Image 2 conveys the kirin’s majestic size by contrasting it with the small child, scoring 7 versus Image 1’s score of 3.Image 1’s scale is indeterminate without a reference point.
  • D. Narrative & Interaction: Image 2 is the clear winner because it depicts the child healing and interacting with the wounded majestic kirin, unlike Image 1’s creature-only portrait.The comparison identifies semantic alignment as decisive, despite Image 1’s slightly superior texture rendering.

2 Related Work

Related work frames multimodal reward models as central to aligning vision generation with human preferences, while noting that fixed scoring functions struggle to adapt across prompts and contents. Vision-generation reinforcement learning has progressed from scalar reward supervision and reward-weighted regression toward policy-gradient methods inspired by PPO.

  • Multimodal Reward Models: Fixed discriminative reward scorers assign scalar values to generated samples but act as task-agnostic heuristics, limiting their ability to reflect diverse evaluation focuses across prompts and contents.These approaches are lightweight and easy to deploy, but their reward functions remain largely fixed.
  • Multimodal Reward Models: Reward modeling supports alignment between vision-generation models and human preferences, motivating multimodal reward models as a core component of visual-generation systems.
  • Reinforcement Learning for Vision Generation: Reinforcement learning for vision generation began with scalar reward fine-tuning [26] [27] and reward-weighted regression [3] [4] [28], followed by policy-gradient updates inspired by PPO [29].The passage describes reward-weighted regression as a more stable way to exploit reward feedback.

3 Method · 3.1 Overview · 3.2 Unified Personalized Reward Modeling

UnifiedReward-Flex addresses the context insensitivity of one-size-fits-all multimodal reward models with a personalized, hierarchical reasoning framework that adapts evaluation criteria to prompt intent and visual evidence. It is trained through reasoning distillation for SFT followed by reasoning-aware DPO preference alignment.

  • 3.1 Overview: UnifiedReward-Flex replaces monolithic scoring functions and fixed rubrics with a unified personalized reward-modeling framework designed to capture content-specific visual cues and prompt intent.Existing reward models can become systematically misaligned because they apply a single preference distribution or uniform criteria across inputs.
  • 3.2.1 Context-adaptive Reasoning Process Design: The model begins with three predefined high-level dimensions, instantiates prompt-specific sub-dimensions beneath them, and adds new dimensions when the context requires them.This hierarchy is intended to mimic human evaluation by combining stable anchors with fine-grained, evidence-dependent criteria.
  • 3.2.1 Context-adaptive Reasoning Process Design: For video assessment, UnifiedReward-Flex evaluates semantic adherence, motion-aware visual quality, and cinematographic coherence, while introducing Action Dynamics & Physics for temporal coherence and physical plausibility.The added dimension is motivated by prompts involving rich motion and physical interactions.
  • 3.2.1 Context-adaptive Reasoning Process Design: By tailoring the evaluation hierarchy to each generation context, the model provides richer reward supervision for preference optimization and targets both intent satisfaction and content-critical quality factors.The adaptive hierarchy is the mechanism connecting context-specific assessment to improved reward supervision.
  • 3.2.2 Stage I: Reasoning Distillation for SFT: Stage I distills structured, evidence-grounded reasoning traces from the closed-source VLM and uses supervised fine-tuning to initialize context-adaptive evaluations for paired visual comparisons.The traces include instantiated high-level dimensions, prompt-specific sub-dimensions, reasoning, and winner labels.
  • 3.2.3 Stage II: Reasoning-Aware Preference Alignment via DPO: Stage II applies DPO to align both the final preference decision and the quality of the adaptive reasoning trajectory under human-grounded supervision.When only one sampled evaluation is correct, it is preferred; when both are correct, a closed-source judge ranks their reasoning trajectories and humans verify the comparison.
  • 3.2.3 Stage II: Reasoning-Aware Preference Alignment via DPO: Reasoning-aware preference alignment improves discriminative power even among samples that are all correct, while reinforcing context-adaptive reasoning behavior.The DPO construction directly favors preferred structured evaluations rather than optimizing only their final winner labels.

3.3 Reinforcement Learning for Vision Generation

This section describes GRPO and Pref-GRPO for reinforcement learning in flow-matching vision generators, then integrates UnifiedReward-Flex into Pref-GRPO to provide personalized, multidimensional rewards. The resulting update combines fine-grained anchor preferences with holistic, context-adaptive evaluation while retaining GRPO’s training objective.

  • GRPO [37] applies terminal reward supervision to flow-matching generators [9] [10], using prompt-wise standardized group-relative advantages to favor higher-quality samples within each prompt group.Its clipped surrogate objective with KL regularization provides stable post-training with learned reward signals.
  • Pref-GRPO [6] replaces absolute rewards with pairwise preference judgments and assigns each candidate a normalized within-group win rate before applying standard GRPO optimization.The preference-derived rewards use the same group-relative advantage and clipped surrogate objective as GRPO.
  • UnifiedReward-Flex is integrated into Pref-GRPO to provide personalized, multidimensional rewards for vision-generation policy updates.The reward model evaluates candidate pairs across predefined anchor dimensions and also captures dynamic high-level dimensions through an overall win rate.
  • The combined GRPO advantage averages dimension-wise win-rate advantages with overall win-rate advantages, controlled by α.This lets fine-grained anchor preferences and holistic, context-adaptive evaluations jointly contribute to policy optimization.

4 Experiment

Experiments show that UnifiedReward-Flex improves preference evaluation and GRPO-based image and video generation across in-domain, out-of-domain, and multi-generator settings. Ablations further support DPO-based preference alignment, balanced reward aggregation, and scaling benefits.

  • Preference Evaluation: UnifiedReward-Flex achieves the best performance across all preference-evaluation benchmarks, improving over UnifiedReward-Think by +3.2 points on MMRB2 and +2.2 points on GenAI-Bench-Video.The comparison covers fixed scorers, Bradley–Terry preference models, and VLM-as-a-judge baselines.
  • Image Generation: On UniGenBench, UnifiedReward-Flex improves overall semantic consistency by +14.56 over the base model and +5.06 over UnifiedReward-Think, with gains on compositional and intent-aware dimensions.The reported improvements include challenging Compound and Logical Reasoning dimensions, while Table 3 evaluates out-of-domain generalization.
  • Video Generation: On VBench, UnifiedReward-Flex raises Dynamic Degree from 58.6 to 70.8 for Wan2.1-T2V-14B GRPO, improving dynamic quality and compositional semantics.The Dynamic Degree increase indicates more informative supervision for motion-intensive generations beyond static appearance.
  • Ablation Studies: DPO consistently improves preference discrimination, including when both sampled reasoning traces predict the correct final preference, by selecting higher-quality reasoning trajectories.This shows that DPO supervises how the decision is reached, not only the final decision.
  • Ablation Studies: UnifiedReward-Flex improves smoothly with model capacity, while reward performance is best when overall and dimension-wise win rates are balanced rather than dominated by either component.Smaller models remain competitive, whereas excessive reliance on either overall or dimension-wise rewards is suboptimal.
  • Cross-Generator Robustness: Across Wan2.1-T2V-14B and Wan2.2-T2V-A14B, UnifiedReward-Flex consistently improves Dynamic Degree, Motion Smoothness, and Human Action metrics.The results support robustness across video generators with diverse architectures and capabilities.

5 Conclusion

UnifiedReward-Flex addresses one-size-fits-all evaluation in visual generation by combining personalized hierarchical assessment with context-adaptive reasoning. Its two-stage training uses structured reasoning distillation followed by reasoning-aware DPO.

  • 5 Conclusion: UnifiedReward-Flex combines dynamic hierarchical assessment with context-adaptive reasoning to capture nuanced, subjective human preferences in visual generation.The approach is designed to overcome rigid evaluation rubrics and traditional one-size-fits-all evaluation.
  • 5 Conclusion: Its two-stage training pipeline consists of structured reasoning distillation and reasoning-aware Direct Preference Optimization (DPO).

Appendix

Appendix qualitative comparisons show UnifiedReward-Flex selecting images that better satisfy spatial, expression, and lighting constraints, while favoring videos with more realistic motion and fluid dynamics. The image and video judgments prioritize core visual quality when outputs trade off against individual prompt constraints.

  • Image Generation: UnifiedReward-Flex identifies Image 1 as the clear winner because it satisfies the prompt’s spatial, expression, and lighting constraints better than Image 2.Image 1 places the subject on the left, shows the requested smile, and provides a soft, even glow; Image 2 places the subject on the right with a neutral expression and shadowed face.
  • Video Generation: UnifiedReward-Flex judges Video 1 vastly superior for photorealism, fluid dynamics, and natural human motion despite violating the fixed-camera constraint.Video 1 renders pouring water with realistic physics, transparency, and interaction, whereas Video 2 shows artificial viscosity and remains visibly AI-generated.
Loading 2602.02380v2…