Source-linked AI summary
VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Dan Zhang, Ming Ding, Xiaohan Zhang, Xiaotao Gu, Shiyu Huang, Minlie Huang, Jie Tang, Yuxiao Dong
TL;DR
Visual reward models for image and video generation can be opaque, biased, and insufficient for fine-grained human preference alignment. VisionReward addresses this with hierarchical binary assessment, interpretable linear weighting, and consistent multi-dimensional optimization, reporting stronger preference prediction and generation alignment than existing reward models.
Problem
Existing visual reward models have limited interpretability, potential unexpected biases, and insufficient accuracy for video preference prediction and optimization.
Method
VisionReward combines hierarchical fine-grained binary visual assessment, linear-weighted preference learning, and multi-dimensional consistent preference optimization.
Results
VisionReward outperforms existing image and video reward models across benchmarks, including 17.2% higher preference-prediction accuracy than VideoScore and a 31.6% higher video pairwise win rate.
Takeaways & Limitations
VisionReward provides a fine-grained and interpretable reward model for visual generation and supports multi-dimensional preference optimization.
Takeaways & Limitations
Training videos are limited to 6 seconds at 4 fps, which may be insufficient for evaluating longer and more complex generated videos.
Abstract
from arXiv · showhide
Visual generative models have achieved remarkable progress in synthesizing photorealistic images and videos, yet aligning their outputs with human preferences across critical dimensions remains a persistent challenge. Though reinforcement learning from human feedback offers promise for preference alignment, existing reward models for visual generation face limitations, including black-box scoring without interpretability and potentially resultant unexpected biases. We present VisionReward, a general framework for learning human visual preferences in both image and video generation. Specifically, we employ a hierarchical visual assessment framework to capture fine-grained human preferences, and leverages linear weighting to enable interpretable preference learning. Furthermore, we propose a multi-dimensional consistent strategy when using VisionReward as a reward model during preference optimization for visual generation. Experiments show that VisionReward can significantly outperform existing image and video reward models on both machine metrics and human evaluation. Notably, VisionReward surpasses VideoScore by 17.2% in preference prediction accuracy, and text-to-video models with VisionReward achieve a 31.6% higher pairwise win rate compared to the same models using VideoScore. All code and datasets are provided at https://github.com/THUDM/VisionReward.
1 Introduction
VisionReward addresses interpretability, bias, and video-specific limitations in visual reward modeling through fine-grained assessment and multi-dimensional preference optimization. It reports stronger preference prediction and generation alignment than prior approaches.
- Motivation: VisionReward targets opaque, potentially biased visual reward models and the limited accuracy of existing video reward models.The framework is motivated by interpretability and unexpected-bias concerns, alongside the challenge of modeling temporal dependencies in video.
- Approach: VisionReward combines fine-grained visual assessment with interpretable preference learning based on hierarchical dimensions and linear weighting.Its assessment uses binary visual questions, while linear weighting makes each question’s contribution more interpretable.
- Approach: VisionReward uses multi-dimensional preference optimization that retains only sample pairs with consistent cross-dimensional preferences.This strategy is intended to mitigate unintended and unquantifiable biases during optimization.
- Results: 17.2% higher preference-prediction accuracy than VideoScore demonstrates VisionReward’s reported advantage on visual preference prediction.The contribution statement identifies this as an example of its performance across multiple benchmarks.
- Results: 31.6% higher pairwise win rate is reported for video models optimized with VisionReward than for the same models using VideoScore.The comparison concerns text-to-video generation models under preference optimization.
2 Related Work
Related work covers RLHF and preference learning for visual generation, while positioning VisionReward as a finer-grained, more interpretable alternative. Its framework organizes visual assessment hierarchically and extends preference optimization across multiple dimensions.
- Fine-Grained Assessment: VisionReward’s hierarchical design uses 5 image dimensions with 18 sub-dimensions and 61 binary questions, versus 9 video dimensions with 20 sub-dimensions and 64 questions.The figure presents the image and video assessment structures side by side.
- Preference Learning: VisionReward differs from prior visual preference methods by disentangling human preferences into fine-grained dimensions for more accurate and interpretable reward modeling.The related-work discussion contrasts this design with approaches that augment annotations or expand preference dimensions without the same stated disentangling goal.
- RLHF for Visual Generation: Existing RLHF approaches for visual generation require reward models, while Diffusion-DPO directly optimizes diffusion models from human-labeled preference pairs.The related work distinguishes gradient-based and policy-based RLHF from direct preference optimization.
- RLHF for Visual Generation: VisionReward addresses the bias concern associated with overall-preference DPO through multi-dimensional preference optimization.The cited passage characterizes ordinary DPO as potentially biased and presents VisionReward as an enhancement.
- VisionReward Framework: The framework combines fine-grained visual assessment, linear-weighted preference learning, and multi-dimensional preference optimization.These are the three stages identified in the overall framework.
3 VisionReward
VisionReward builds fine-grained visual preference assessments from binary questions and learns interpretable linear reward weights for image and video generation. It also uses multi-dimensional preference optimization to select consistently ordered pairs and address observed preference biases.
- Dataset and Annotation: 89.29% image and 89.33% video annotator consistency follows structured training, qualification, inspection, and example-based annotation guidance.The annotation document provides clear definitions and more than 10 examples for each judgment.
- VisionReward Training: VisionReward combines fine-grained visual assessment with interpretable linear preference learning to model human preferences across visual dimensions.The framework decomposes preferences into binary questions, predicts their outcomes, and combines them with learned weights.
- Multi-Dimensional Preference Optimization: MPO-enhanced DPO selects preference pairs only when one sample dominates the other in every dimension, unlike ordinary DPO’s total-reward selection.The dominance condition requires Ri(dimk) ≥ Rj(dimk) for every dimension dimk.
- Multi-Dimensional Preference Optimization: Figure 4 evaluates score deviations across 18 sub-dimensions for 10,000 human preference pairs and corresponding SDXL images after Diffusion-DPO.The figure uses average yes-proportions of checklist questions within each sub-dimension.
4 Experiments
Experiments evaluate VisionReward for preference prediction and visual-generation optimization across image and video settings. Results show strong preference accuracy, improved human evaluations, and benefits from multi-dimensional preference optimization.
- 4.1 VisionReward for Text-to-Vision Evaluation: VisionReward accurately predicts human preferences on multiple datasets, including longer videos where other reward models struggle.On 6-second MonetBench videos, VisionReward reaches 22.1% over random, compared with 12.5% over random for the best other method.
- 4.2 VisionReward for Text-to-Vision Optimization: VisionReward achieves the best human preference performance among DPO systems using different reward models or preference datasets.Five annotators evaluate each pair and select the better sample.
- 4.1 VisionReward for Text-to-Vision Evaluation: VisionReward’s preference-prediction accuracy improves as the number of binary questions increases while remaining strong after weight masking.The experiment tests the scalability of decomposing preferences into fine-grained questions.
- 4.2 VisionReward for Text-to-Video Optimization: Optimization with VisionReward improves CogVideoX-2B across VBench and Video-MonetBench, whereas VideoScore optimization tends to degrade performance.The comparison evaluates multi-dimensional optimization for text-to-video generation.
- 4.3 Ablation Study of MPO: MPO avoids degradation of individual factors and improves trade-offs, while outperforming DPO in both efficiency and effectiveness.The study reports preserved quality and avoidance of excessively slow dynamic changes.
5 Conclusion
VisionReward is presented as a fine-grained, multi-dimensional reward model that combines binary visual assessments with interpretable linear weighting. It supports accurate preference evaluation and multi-dimensional optimization for visual generation.
- 5 Conclusion: VisionReward combines binary assessments from a vision-language model with weighted linear summation to produce an accurate and interpretable reward model.The model is designed for visual generative optimization and supports a multi-dimensional strategy.
6 More Details of Annotation
The annotation pipeline organizes visual preferences across dimensions and records calibrated statistics for image and video data. It also uses masking and standardized annotation procedures for applicable sub-dimensions.
- 6.1 Annotation Taxonomy: The annotation taxonomy uses five dimensions for text-to-image generation and expands to nine dimensions for text-to-video generation.The dataset sources and annotation statistics are summarized in the accompanying tables.
- 6.2 Statistics of Annotation Result: Annotation statistics treat higher labels as better performance, use zero for neutrality, and separate absent elements into a “Not Contain” category.Video labels are adjusted by subtracting a sub-dimension-specific neutral value.
- 6.2 Statistics of Annotation Result: Most sub-dimension options are approximately normally distributed, while some factors require masks when the relevant content is absent.Hands, faces, and backgrounds can be unavailable and are handled separately in annotation statistics.
7 More Details and Results of VisionReward
VisionReward converts fine-grained visual judgments into preference scores through linear regression with iterative weight masking. Additional experiments examine judgment accuracy, consistency, scalability, and training-set size.
- 7.2 Visual Judgment Process: VisionReward represents each sub-dimension through progressive binary questions, balancing positive and negative examples for judgment learning.Samples at or above an option threshold are treated as positive examples for that question.
- 7.1 Visual Judgment Results: VisionReward’s visual judgment evaluation uses separate image and video quality-question sets covering multiple dimensions and question types.The benchmark includes 1,364 image cases and 1,308 video cases.
- 7.1 Visual Judgment Results: VisionReward achieves more than 97% consistency in seven of eight evaluated dimensions.Consistency measures agreement across multiple questions assessing the same factor.
- 7.2 Interpretable Preference Learning: The iterative regression algorithm learns preference weights from paired binary-response features and masks negative weights before convergence.It uses logistic prediction on feature differences and repeatedly updates weights until the convergence criterion is met.
- 7.3 Ablation Study: Preference accuracy increases monotonically with training-set size up to 4,000 samples and then stabilizes.This result is reported on HPDv2-test.
8 More Details and Results of MPO
The section details MPO data construction, evaluation, and preference-optimization results for visual generation. VisionReward improves multiple metrics and supports simultaneous gains across dimensions, with dimension-level dominance performing best among MPO strategies.
- Dataset & Training Settings: The training setup uses 63,165 prompts, generated image pairs, and consistent optimization parameters across experiments for fair comparison.
- VisionReward achieves leading results across multiple machine metrics and significant improvements across all four evaluated dimensions.
- MPO fine-tuning surpasses the original model across multiple aspects and outperforms other scoring methods for image and video generation.
- Training Curve: MPO enables simultaneous improvement across sub-dimensions, whereas DPO does not achieve the same concurrent enhancement.
- Ablation Study: Different Strategy for MPO: Dimension-level dominance yields the best VisionReward results among MPO strategies based on dimension, sub-dimension, or binary-question scores.
9 Details of Fine-Grained Questions
The section identifies the tables documenting the annotation taxonomy for images and videos.
- Tables 15–18 document the annotation taxonomy, with Tables 15–16 covering images and Tables 17–18 covering videos.
10 More Results of Fine-Grained Design
The section evaluates fine-grained judgment accuracy, dimensional correlations, benchmark construction, and VisionReward’s scope boundaries. It also documents limitations involving video duration, frame rate, and dependence on the base vision-language model.
- Weight and Accuracy of Checklist: VisionReward’s fine-grained questions are evaluated on separate image and video test sets using accuracy, Spearman correlation, and learned linear weights.
- Correlation of Sub Dimensions: A correlation heatmap examines relationships among text-to-video sub-dimensions after preference decoupling.
- Supported Video Frame Rate and Length: VisionReward training videos are limited to 6 seconds at 4 fps, which may be insufficient for longer and more complex next-generation video outputs.
- Leverage of Foundation Model: VisionReward performance depends on the capabilities of its base vision-language model, while explicit reasoning is identified as a future enhancement direction.
- Benchmark Construction: The benchmark design balances content and challenge categories across image and video generation, enabling multidimensional evaluation of visual reward models.