Source-linked AI summary
Human Preference Score: Better Aligning Text-to-Image Models with Human Preference
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, Hongsheng Li
TL;DR
Text-to-image models often misalign with human preferences, and existing evaluation metrics do not fully track human choices. The paper collects human choices, fine-tunes CLIP into a preference classifier and HPS, then uses HPS to adapt Stable Diffusion. HPS better predicts preferences, and the adapted model generates images users prefer more.
Problem
Generated text-to-image images can contain awkward artifacts, while mainstream metrics do not fully match human choices.
Method
The authors collect human choices, fine-tune CLIP into a human preference classifier, derive HPS, and adapt Stable Diffusion with HPS-guided LoRA training.
Results
HPS better predicts human choices than existing mainstream metrics, and HPS-guided adaptation produces more preferable images that better capture human intentions.
Takeaways & Limitations
Human preference can serve as an additional image-quality dimension and guide generative models toward images better aligned with human intentions.
Takeaways & Limitations
The dataset reflects a small, potentially biased population of Stable Foundation Discord users and includes many prompts written by experienced Stable Diffusion users.
Abstract
from arXiv · showhide
Recent years have witnessed a rapid growth of deep generative models, with text-to-image models gaining significant attention from the public. However, existing models often generate images that do not align well with human preferences, such as awkward combinations of limbs and facial expressions. To address this issue, we collect a dataset of human choices on generated images from the Stable Foundation Discord channel. Our experiments demonstrate that current evaluation metrics for generative models do not correlate well with human choices. Thus, we train a human preference classifier with the collected dataset and derive a Human Preference Score (HPS) based on the classifier. Using HPS, we propose a simple yet effective method to adapt Stable Diffusion to better align with human preferences. Our experiments show that HPS outperforms CLIP in predicting human choices and has good generalization capability toward images generated from other models. By tuning Stable Diffusion with the guidance of HPS, the adapted model is able to generate images that are more preferred by human users. The project page is available here: https://tgxs002.github.io/align_sd_web/ .
1. Introduction
Text-to-image models can produce images that diverge from human preferences, while mainstream metrics may not capture those choices. The paper addresses this gap with a human-choice dataset, a preference classifier, and HPS-guided adaptation of Stable Diffusion.
- Generated images can contain awkward limb combinations and facial expressions, forcing users to cherry-pick results that better match their preferences.
- IS and FID may not correlate with human choices because they emphasize image features without fully representing human perception or user intention.
- The dataset contains 98,807 generated images from user-provided prompts and 25,205 human choices for studying human preference.
- The authors fine-tune CLIP into a human preference classifier, define HPS from it, and use HPS to guide Stable Diffusion adaptation.
- The adapted Stable Diffusion model better captures human intentions and generates more preferable images, mitigating artifacts illustrated in Figure 1.
2. Related Works
Related work spans text-to-image generation, diffusion models, Stable Diffusion improvements, generated-image datasets, and human-feedback approaches. The paper positions its method as targeting human-preference misalignment rather than customization, prompt engineering, or exact text-image matching alone.
- Text-to-image generative models: Text-to-image research progressed from caption-conditioned generation and GANs to open-domain systems such as DALL·E.
- Diffusion models: Diffusion models generate images by reversing a diffusion process, and later systems including DALL·E 2, GLIDE, Imagen, ERNIE-ViLG, and Stable Diffusion brought this approach to public attention.
- Stable Diffusion improvements: Prior work improves Stable Diffusion through object customization, compositional guidance, and automatic prompt engineering, whereas this method targets human-preference misalignment.
- Datasets of generated images: Generated-image databases support vision research, but Lexica is closed-source and provides prompts and related generation information alongside Stable Diffusion images.
- Human feedback: Human-feedback methods have improved models through interaction or reward functions, while related text-to-image work emphasizes prompt augmentation or exact text-image alignment.
3. Human Preference Dataset
The paper builds a large dataset of user choices among images generated from the same prompts in the Stable Foundation Discord channel. It uses these choices to study metric correlation with human preferences and train a preference classifier.
- Data collection: The dataset is extracted from “dreambot” Discord interactions using a pre-defined grammar and pattern matching.DiscordChatExporter downloads the channel history in JSON format for processing.
- Dataset statistics: 98,807 images from 25,205 prompts are organized into preferred and non-preferred images selected within each prompt.Most prompts have four images, while others have three or two.
- Dataset evaluation: Table 1 compares IS and FID for preferred versus non-preferred images using LAION-5B images aligned with the text inputs as FID references.The table reports both metrics for the two image groups.
- Dataset statistics: Choices come from 2,659 users, with images spanning a broad range of themes.Each user contributes at most 267 choices.
- Uses: The dataset supports both evaluating metric correlation with human preferences and training the human preference classifier.The same collected data is used in Sections 4 and 5 for these purposes.
4. Existing Metrics
The paper evaluates mainstream image-quality and prompt-alignment metrics against human choices and finds that they do not reliably track preference. It also examines CLIP-based and aesthetic predictors on a 5,000-choice task.
- Overview: Current mainstream evaluation metrics are not well correlated with human preferences on the collected dataset.The section tests whether standard metrics distinguish preferred from non-preferred images.
- Metrics by Inception Net: IS shows no significant difference between preferred and non-preferred images.FID is computed against pseudo-ground-truth images retrieved from LAION using sampled text prompts, and likewise reveals no significant difference.
- Method connection: Figure 5 summarizes the pipeline for training the preference classifier and adapting Stable Diffusion to generate preferable images.The figure depicts the use of a special identifier as a negative prompt during inference.
- Metrics by CLIP: CLIP score measures cosine similarity between prompt and image embeddings, while the aesthetic classifier predicts from the image without conditioning on the prompt.The evaluation counts prediction accuracy on 5,000 dataset samples.
- Metrics by CLIP: The aesthetic classifier achieves better-than-chance accuracy in predicting user choice, indicating an aesthetic component in human decision-making.Unlike CLIP score, its prediction does not use the text prompt.
5. Human Preference Score
The paper derives Human Preference Score from a CLIP model fine-tuned on human choices. The resulting classifier is trained to increase similarity for preferred images and decrease it for non-preferred images.
- Human preference classifier: The human preference classifier is a ViT-L/14 CLIP model fine-tuned on prompts paired with one preferred and multiple non-preferred images.Each training sample contains two, three, or four images for the same prompt.
- Human preference classifier: Training maximizes prompt–preferred-image similarity while minimizing similarity for non-preferred images.This fine-tuning is intended to align the model more closely with human preferences.
- Human Preference Score: HPS is derived from the trained human preference classifier as a scaled cosine similarity between visual and text encoder embeddings.The cosine similarity is multiplied by 100 for visualization.
6. Better Aligning Stable Diffusion with Human Preferences
The paper adapts Stable Diffusion using HPS-labeled generated images and LoRA, teaching the model to distinguish non-preferred images through a special prompt prefix. During inference, that prefix is used negatively to avoid such images.
- Method rationale: The method treats preference misalignment as missing awareness rather than insufficient model capacity.It explicitly distinguishes preferred from non-preferred images during adaptation.
- Training data: Generated training images are categorized as preferred or non-preferred using the previously trained HPS classifier.The training data combines generated images with real images for regularization.
- Stable Diffusion adaptation: Non-preferred images receive a special prompt prefix so Stable Diffusion can learn the concept of non-preference through LoRA adaptation.The original model parameters remain frozen while low-rank residuals augment projection matrices.
- Inference: During inference, the special identifier is supplied as the negative prompt in classifier-free guidance to avoid non-preferred images.Figure 5 presents the same training-to-inference design using [Identifier].
- HPS and CLIP: HPS correlates positively with CLIP score but emphasizes image aesthetic quality more than direct image–text matching.The paper interprets this difference as related to an “alignment tax.”
7. Experiments
The experiments evaluate HPS against human choices, test its generalization, and use it to adapt Stable Diffusion. HPS aligns better with human preferences than CLIP, while HPS-guided adaptation improves prompt alignment and reduces unnatural limbs.
- The experiments first validate HPS reliability, then evaluate Stable Diffusion adaptation guided by human preference.
- HPS reliability: HPS significantly outperforms CLIP in predicting human choices, with accuracy exceeding that of human participants.The classifier is trained on human-choice data and evaluated for alignment with human preferences.
- Stable Diffusion adaptation: The adapted model better captures prompt intent and generates more preferable images than the original and regularization-only models.Qualitative comparisons use the same prompt and random seed within each row, with prompts sampled from DiffusionDB.
- HPS reliability: HPS aligns more closely with human preferences than CLIP when comparing images generated by DALL·E and Stable Diffusion.Its agreement with humans is close to the agreement among human participants, indicating generalization across models.
- HPS reliability: HPS has a positive correlation with CLIP score but emphasizes image aesthetic quality more than direct image-text matching.The comparison uses images generated by Stable Diffusion from prompts sampled from COCO Captions.
- Stable Diffusion adaptation: Training with HPS-labeled generated images mitigates unnatural-limb artifacts compared with regularization-only training.The regularization-only model removes generated images from training while using regularization images for the same number of steps.
8. Limitations
The dataset captures preferences from a limited and potentially biased user population, with many prompts written by experienced Stable Diffusion users.
- The dataset may reflect the preferences of a small, Stable Foundation Discord–active population rather than people worldwide.The authors retain public-figure prompts and images to preserve dataset diversity.
- Many prompts were written by experienced Stable Diffusion users and may be tuned to exploit the model rather than reflect normal language habits.
9. Conclusion
The paper finds that mainstream generative-model metrics do not adequately track human preferences, while fine-tuned CLIP can better align with human choices. It then uses HPS to adapt Stable Diffusion toward more preferable images.
- Mainstream evaluation metrics are not well aligned with human preferences, whereas fine-tuned CLIP better aligns with human choices.
- The authors adapt a generative model with guidance from HPS to generate more preferable images.They describe the adaptation method as simple yet effective.
A. Datasheet
The dataset supports research on human aesthetic preferences for generated images by pairing prompts and image groups with user choices. It documents collection, composition, use, distribution, and important limitations, including noise, sensitive content, and population bias.
- The dataset was created for academic research on human aesthetic preference in generated images.
- Each instance contains one prompt, 2–4 generated images, and one human preference choice among those images.
- The dataset omits image-generation parameters because comparisons within each batch use the same parameters except for the random seed.
- The dataset may contain noisy choices, sensitive user-written prompts, some NSFW content, and preferences biased toward Stable Foundation Discord users.
- Data were collected automatically from public messages and chat history in the Stable Foundation Discord server, with no preprocessing applied to images or prompts.
B. More Dataset Examples
The examples compare original and adapted Stable Diffusion outputs generated from identical prompts and random seeds. The adapted model is presented as producing fewer artifacts and better capturing user intentions.
- C. More Visualization: The adapted model generates images with fewer artifacts than the original model.
- C. More Visualization: The adapted model better captures users’ intentions than the original model.
- B. More Dataset Examples: Figure 9 shows collected examples generated by Stable Diffusion, with preferred images marked by red borders.
- C. More Visualization: Figures 10 and 11 compare original and adapted model outputs generated with the same prompt and random seed.
- C. More Visualization: Figure 12 shows the interface used for the user study.