Source-linked AI summary
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, Yuxiao Dong
TL;DR
Text-to-image generators remain difficult to align with human preferences, while collecting and using human judgments at scale is costly. The paper introduces ImageReward, trained on expert comparisons through a systematic annotation pipeline, and ReFL, which directly tunes diffusion models using reward feedback. ImageReward aligns better with human rankings than existing scoring methods, and evaluations support ReFL's advantages over compared approaches.
Problem
Text-to-image models exhibit alignment, anatomical, aesthetic, toxicity, and bias problems, while expert human-feedback annotation is costly and difficult to scale.
Method
The paper builds ImageReward from a systematic expert-annotation pipeline and develops ReFL to directly optimize diffusion models using ImageReward feedback.
Results
ImageReward outperforms CLIP, Aesthetic, and BLIP in understanding human preference, with gains of 38.6%, 39.6%, and 31.6%, respectively.
Takeaways & Limitations
ImageReward is a promising automatic metric for text-to-image evaluation, while ImageReward and ReFL together better align generation with human values and preferences.
Takeaways & Limitations
Reward-weighted training can retain influence from non-preferred images because reward coefficients remain within the [0, 1] range, causing interference with real-user prompts.
Abstract
from arXiv · showhide
We present a comprehensive solution to learn and improve text-to-image models from human preference feedback. To begin with, we build ImageReward -- the first general-purpose text-to-image human preference reward model -- to effectively encode human preferences. Its training is based on our systematic annotation pipeline including rating and ranking, which collects 137k expert comparisons to date. In human evaluation, ImageReward outperforms existing scoring models and metrics, making it a promising automatic metric for evaluating text-to-image synthesis. On top of it, we propose Reward Feedback Learning (ReFL), a direct tuning algorithm to optimize diffusion models against a scorer. Both automatic and human evaluation support ReFL's advantages over compared methods. All code and datasets are provided at \url{https://github.com/THUDM/ImageReward}.
1 Introduction
Text-to-image models often fail to align generated images with human preferences, including text-image coherence, anatomical quality, aesthetics, and safety. The paper addresses these challenges with ImageReward and ReFL, supported by a systematic expert-annotation pipeline.
- Pre-training distributions can be noisy and differ from actual user-prompt distributions, complicating alignment with human preference.
- Text-to-image generators struggle with text-image alignment, body abnormalities, human aesthetics, and toxicity or bias.
- Human-feedback reward modeling requires costly expert comparison annotation and substantial effort to establish criteria, train annotators, and validate labels.
- ImageReward is introduced as the first general-purpose text-to-image human-preference reward model, trained and evaluated on 137k expert comparisons from real-user prompts and model outputs.
- ReFL directly tunes diffusion models against human-preference scorers, with automatic and human evaluations supporting advantages over data augmentation and loss reweighing.
2 ImageReward: Learning to Score and Evaluate Human Preferences
The paper constructs ImageReward from expert annotations and evaluates it as a human-preference metric for text-to-image models. It reports stronger human alignment and sample distinguishability than existing scoring approaches, while motivating zero-shot evaluation and selection.
- ImageReward construction: ImageReward combines expert annotation with reward-model training to encode preferences over text-image pairs.
- ImageReward construction: 10,000 diverse real-user prompts produced 177,304 candidate image pairs for labeling, with each prompt accompanied by 4 to 9 sampled images.
- ImageReward construction: The annotation pipeline consists of prompt annotation, text-image rating for alignment, fidelity, and harmlessness, image ranking, annotator training, and quality inspection.
- ImageReward construction: 136,892 compared pairs were collected from valid annotations for 8,878 prompts after two months of annotation.
- Evaluation as a metric: ImageReward is proposed as a zero-shot automatic metric for comparing text-to-image models and selecting individual samples.
- Evaluation as a metric: ImageReward aligns well with human model rankings, whereas zero-shot FID and CLIP do not in the reported comparison.
- Evaluation as a metric: ImageReward has larger within-model score interquartile ranges than CLIP, indicating stronger distinguishability among individual samples; its model-level medians also roughly follow human rankings.
3 ReFL: Reward Feedback Learning Improves Text-to-Image Diffusion
ReFL directly fine-tunes latent diffusion models using human-preference reward feedback, addressing the lack of generation likelihoods that limits standard RLHF approaches. Its key insight is that ImageReward scores become reliable during the latter denoising steps, enabling feedback before completion while improving training stability through random step selection and regularization.
- Motivation: Generate-and-then-filter methods can be expensive and inefficient, motivating direct improvement of generation quality in few trials.
- Challenge: Latent diffusion models lack likelihoods for multistep denoising generations, preventing them from directly adopting standard RLHF approaches.
- ReFL: Insight and Solution: ImageReward scores are uniformly low through step 15, begin separating high-quality generations between steps 15 and 30, and generally distinguish final quality from step 30 onward.
- ReFL: Insight and Solution: ReFL directly back-propagates a reward-model-derived preference loss through a randomly selected latter denoising step rather than waiting for the final step.
- ReFL: Insight and Solution: Randomizing the denoising step avoids instability associated with retaining only the final-step gradient, while loss reweighting and pre-training-loss regularization reduce rapid overfitting.
4 Experiment
The experiments evaluate ImageReward as a human-preference predictor and ReFL as a diffusion-model tuning method. ImageReward outperforms the tested scoring baselines, while ReFL achieves the strongest reported preference outcomes among optimization methods.
- ImageReward: Human Preference Prediction: Increasing the training dataset size significantly improves ImageReward’s preference accuracy.The ablation study reports that adding annotation data improves performance and suggests further gains from collecting more annotations.
- ImageReward: Human Preference Prediction: In human evaluation, ImageReward selects images preferred over random, CLIP, Aesthetic, and BLIP, with average win rates of 77.1%, 69.3%, 69.8%, and 65.8%, respectively.The evaluation selects top-three images from sets containing 9, 25, or 64 generated images, followed by ranking by three annotators.
- ReFL: Diffusion-Model Optimization: ReFL achieves the highest win rate against Stable Diffusion v1.4 and is preferred over the compared optimization methods.The evaluation compares methods using real user prompts and challenging benchmark prompts under matched training settings.
- ImageReward: Human Preference Prediction: BLIP provides a stronger ImageReward backbone than CLIP in the reported ablation, even when CLIP uses a relatively larger training dataset.The authors attribute part of the difference to BLIP’s bootstrapping and image-grounded text encoder.
- ReFL: Diffusion-Model Optimization: The evaluation reports that RAFT and Reward Weighted do not improve over the baseline, while ReFL provides direct reward-based gradient feedback.The authors relate the comparison to prompt distribution: ReFL is evaluated with more widely distributed and complex prompts, whereas competing methods use constructed or non-user prompt data.
- ReFL: Diffusion-Model Optimization: RAFT is constrained by expert-generator quality, whereas Reward Weighted remains affected by non-preferred images because its reward coefficients do not eliminate their influence.The authors identify these as limitations of the compared methods rather than of ReFL.
5 Related Work
Prior text-to-image evaluation methods use general image-generation metrics or text-image similarity and do not comprehensively capture human preference. Earlier human-feedback approaches for text-to-image generation were limited in study scope, prompt diversity, annotation consistency, or indirect optimization.
- Text-to-image Generation and Evaluation: Inception Score and Fréchet Inception Distance are commonly used after fine-tuning, while CLIP-based methods evaluate individual images through text-image similarity.Aesthetic provides an additional CLIP-based prediction of image aesthetics.
- Text-to-image Generation and Evaluation: These metrics do not comprehensively capture human preference across factors such as text-image alignment, fidelity, and aesthetics.The related-work discussion distinguishes holistic human-preference reward modeling from narrower similarity or aesthetic scoring.
- Learning from Human Feedback: Human-feedback methods in language applications commonly train reward models from human preference data to align generative systems with human intent.The paper contrasts this established language-modeling practice with the smaller body of work for text-to-image generation.
- Text-to-image Human Feedback: Prior text-to-image studies used closed-domain synthetic prompts, noisy online clicks, indirect data filtering, or augmentation rather than a general-purpose scorer with rigorous annotation.The paper positions ImageReward and ReFL as addressing these limitations through quality-controlled annotation and direct diffusion-model tuning.
6 Conclusion
The paper presents ImageReward and ReFL as complementary tools for aligning text-to-image generation with human preferences. ImageReward is trained through a structured annotation pipeline that combines prompt analysis, image rating, and preference ranking.
- Annotation System Design: Annotator quality is managed through training, agreement tests, inspection, and relabeling of invalid annotations.
- The pipeline was designed because large-scale preference annotation requires consensual criteria despite individual preferences being easy to identify.
- Text-Image Rating: Ratings assess alignment, fidelity, and harmlessness, while ranking guidelines specify trade-offs when these criteria conflict.
- Annotation System Design: The annotation pipeline spans prompt categorization, problem identification, text-image rating, and ranking generated images by preference.
A.4 Human Annotation Analysis
The analysis links prompt categories and prompt composition to image quality and failure patterns. Body defects are the most frequent reported problem, while function-phrase proportion shows non-monotonic relationships with some errors.
- 136,892 comparison pairs were collected from 8,878 valid prompts selected from 10,000 annotation candidates.
- Average Score Distribution: Abstract prompts receive the lowest average scores across alignment, fidelity, and overall satisfaction.
- Problem Distribution: 21.14% is the average frequency of body problems, which occur most often in People and Arts categories.
- Problem Distribution: 3.16% is the average frequency of repeated-generation problems, concentrated in Word Knowledge, People, Arts, and Vehicles.
- Function-Phrase Analysis: Repeated-generation frequency first increases, then decreases, and rises again as function phrases grow from 0% to 80%.
- Function-Phrase Analysis: Body and fuzzy problems generally decline as function-phrase proportions increase, whereas toxic, pornographic, and violent content shows no strong correlation.
D.1 Human Evaluation
The human-evaluation analysis compares preference-model selections and examines the prompt distributions used for training. ImageReward’s selected images achieve the strongest overall ranking separation, while its prompt set is slightly more evenly distributed than PickScore’s.
- Human Evaluation: Human evaluation ranks images generated from the same prompt across datasets, including Best of 64 and ReFL variants.
- Prompt Distribution: The ImageReward training prompt set is slightly more evenly distributed than the compared PickScore training prompt set.
E Implementation Details of Related LDM Optimization Methods
The paper compares reward-guided diffusion fine-tuning methods based on dataset construction, loss reweighting, or repeated high-reward data selection. ReFL is motivated by ImageReward’s informative scores at later denoising steps.
- Existing human-feedback methods for latent diffusion models mainly acquire new datasets or change loss-function coefficients.
- Dataset Filtering: Dataset Filtering constructs preferred and non-preferred training examples by selecting generated images with high or low reward scores.
- Reward Weighted: Reward Weighted uses reward values as loss coefficients while adding model-generated images to the pre-training dataset.
- RAFT: RAFT repeatedly generates 100,000 images, ranks them with ImageReward, and selects 10,000 images for latent-diffusion fine-tuning.
- Figure 19 examines correlations between denoising-step rate, Spearman ρ, and ImageReward scores.
F.2 ReFL compares other fine-tuning methods
ReFL is compared with other fine-tuning methods through win-rate and qualitative evaluations. It achieves the highest reported win rate, with additional examples provided for inspection.
- ReFL achieves the highest win rate among the compared fine-tuning methods.
- Additional qualitative examples compare ReFL with other fine-tuning methods.
G Additional Results of ImageReward Compared to Other Typical Image Scorers
The section presents additional qualitative comparisons involving ImageReward and fine-tuning methods. Comparisons select top-ranked images by method-specific scores or rewards across sampled prompts.
- Additional qualitative examples compare ImageReward with typical image scoring methods.
- Fine-tuning comparisons include ReFL, reward-weighted dataset filtering, the original method, and RAFT.
- Each method selects its top 3 images using the corresponding scores or rewards.
- Prompts are sampled from DiffusionDB, while the annotated dataset provides more than 64 generated images for selection.
H Limitations
The paper identifies limitations in annotation data, reward-model training, and the use of reward models to improve generative models.
- The annotation set has about 9k prompts and 137k expert-comparison pairs, but larger and more diverse data are still needed.
- DiffusionDB prompts may contain bias because real-world text-to-image use extends beyond unusual prompts.
- Single-person annotation with quality control may provide less consistency than multi-person fitting annotation.
- Reward-model training may benefit from parameter-efficient tuning and stronger, larger text-image backbones.
- ReFL remains an approximation of original RLHF and requires more fundamentally grounded, unbiased, and efficient feedback-learning algorithms.
J Reproducibility
The paper provides code and detailed information covering ImageReward training and evaluation, including one-command scripts for evaluations and ReFL.
- The repository openly provides code and detailed information for ImageReward and ReFL.
- ImageReward training details are documented through objective, dataset, architecture, hyperparameter, and experimental-setup specifications.
- Evaluation procedures, including model ranking and human-preference prediction, are organized into executable bash scripts.
- ReFL is also provided through one-command-to-run bash scripts with further algorithm details in the paper.