Source-linked AI summary
Aligning Text-to-Image Models using Human Feedback
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Shixiang Shane Gu
TL;DR
Text-to-image models often struggle to align generated images with prompts, especially for compositional details. The paper trains a reward function from binary human feedback and fine-tunes the model with reward-weighted likelihood, improving alignment for specified attributes and compositions while exposing fidelity tradeoffs.
Problem
Text-to-image models often fail to generate images well-aligned with prompts, including reliable visual text and compositional content.
Method
The method collects binary human feedback, trains a reward function to predict alignment judgments, and fine-tunes the model using reward-weighted likelihood.
Results
Up to 47% improvement in image-text alignment is achieved on human evaluation, with mildly degraded image fidelity.
Takeaways & Limitations
Human feedback can effectively improve alignment for specified counts, colors, backgrounds, and compositional combinations in text-to-image models.
Takeaways & Limitations
The results do not address all failure modes of existing text-to-image models, and balancing alignment with fidelity remains challenging.
Abstract
from arXiv · showhide
Deep generative models have shown impressive results in text-to-image synthesis. However, current text-to-image models often generate images that are inadequately aligned with text prompts. We propose a fine-tuning method for aligning such models using human feedback, comprising three stages. First, we collect human feedback assessing model output alignment from a set of diverse text prompts. We then use the human-labeled image-text dataset to train a reward function that predicts human feedback. Lastly, the text-to-image model is fine-tuned by maximizing reward-weighted likelihood to improve image-text alignment. Our method generates objects with specified colors, counts and backgrounds more accurately than the pre-trained model. We also analyze several design choices and find that careful investigations on such design choices are important in balancing the alignment-fidelity tradeoffs. Our results demonstrate the potential for learning from human feedback to significantly improve text-to-image models.
1. Introduction
The paper proposes human-feedback fine-tuning to improve text-image alignment, using human assessments to train a reward function and reward-weighted likelihood to update the model. Experiments report stronger alignment for specified colors, counts, backgrounds, and compositions, while highlighting an alignment-fidelity tradeoff.
- Current text-to-image models often fail on reliable visual text and compositional image generation, motivating improved prompt alignment.
- The method samples diverse outputs for alignment-focused prompts, collects binary human feedback, trains a reward function, and fine-tunes with reward-weighted likelihood.An auxiliary prompt-classification objective identifies the original prompt among perturbed alternatives.
- 27K human-feedback image-text pairs are used to fine-tune Stable Diffusion for objects with specified colors, counts, and backgrounds.
- Up to 47% improvement in image-text alignment is achieved on human evaluation, at the expense of mildly degraded image fidelity.
- The learned reward function predicts human quality assessments more accurately than the CLIP score, and rejection sampling based on it can improve image-text alignment.
- Naive human-feedback fine-tuning can reduce image fidelity, so design choices such as auxiliary losses and training-data diversity require careful investigation.
2. Related Work
Related work shows strong progress in text-to-image generation but persistent alignment failures, alongside human-feedback methods and embedding-based alignment evaluation. Prior fine-tuning work mainly addresses personalization, whereas this paper applies fine-tuning to alignment.
- Text-to-image systems use varied generative architectures with large-scale language encoders and have achieved impressive generation results.
- Existing models frequently fail at reliable visual text and compositional generation, prompting techniques such as character-aware encoders and structured language representations.
- Few-image fine-tuning methods such as DreamBooth personalize text-to-image diffusion models, while this work targets alignment using human feedback.
- Human feedback has been applied across translation, question answering, story generation, and instruction-following systems.
- Image-text alignment is commonly evaluated with embedding-based scores from pretrained multimodal models such as CLIP and CoCa.
3. Main Method
The method aligns text-to-image models through human-labeled prompts, reward learning, and reward-weighted fine-tuning, with regularization to preserve natural image generation.
- 3. Main Method: The method generates prompt-targeted images, collects binary human judgments, trains a reward model, and fine-tunes the generator using reward-weighted log likelihood.Prompts test specified object counts, colors, backgrounds, and their combinations.
- 3. Main Method: The reward function maps CLIP image-text embeddings to a scalar prediction of human feedback, trained by minimizing mean-squared error.Feedback uses y ∈ {0, 1}, where 1 denotes good alignment and 0 denotes bad alignment.
- 3. Main Method: The study limits prompts mainly to count, color, and background categories and defers broader categories, ranking feedback, and RLHF-style updates to future work.The authors note that RLHF could offer benefits but would require extensive hyperparameter tuning and engineering.
- 3. Main Method: Prompt classification augments good image-text pairs with semantically different prompts and adds an auxiliary loss to improve reward-model generalization to unseen images and text prompts.The combined reward-learning loss is LMSE(φ) + λLpc(φ), where λ is a penalty parameter.
- 3. Main Method: The generator loss combines reward-weighted negative log likelihood on model-generated data with pre-training loss on Dpre to reduce overfitting and produce more natural images.The model-generated data can combine human-labeled and additional unlabeled generated images.
4. Experiments
Experiments evaluate human-feedback fine-tuning across setup, human alignment judgments, qualitative samples, reward learning, rejection sampling, and dataset ablations. The method improves prompt adherence, while fidelity depends on dataset diversity and optimization choices.
- Experimental Setup: The experiments use Stable Diffusion v1.5 as the baseline, freeze its CLIP language encoder, and fine-tune only the diffusion module.The reward model uses CLIP embeddings as input to an MLP.
- Experimental Setup: The study generates 27K images from 2,700 English prompts, with additional unlabeled and pre-training data used during fine-tuning.The pre-training data is a 625K filtered subset of LAION-5B.
- Human Evaluation: On 120 prompts, human raters compare fine-tuned and original models for alignment and fidelity using nine independent ratings per query.The prompts include 60 seen and 60 unseen prompts, with ties allowed for similar quality.
- Text-Image Alignment Results: 50% of samples from the fine-tuned model receive at least two-thirds of votes for image-text alignment, compared with 15% versus 10% for image fidelity.The authors attribute the fidelity decrease to alignment-focused labels, limited human-data diversity, and a small pre-training subset.
- Qualitative Comparison: Qualitative comparisons show better adherence to specified colors, counts, and backgrounds, including unseen objects and unseen artistic text categories.The original model often omits prompt details, whereas the fine-tuned model produces reasonable images for unseen categories.
- Reward Learning: The learned reward predicts human preferences more accurately than CLIP, while prompt-classification loss improves performance on seen and unseen prompts.Reward accuracy on unseen prompts is approximately 80%, and reduced human-data size lowers accuracy on both prompt types.
- Reward Learning: Rejection sampling improves alignment to 46% two-thirds preference without sacrificing fidelity, but fine-tuning adds a 10% alignment gain while sacrificing 17% fidelity against that baseline.Rejection sampling generates 16 images and selects the four highest-reward outputs, increasing inference-time computation and leaving the model unchanged.
- Ablation Studies: Adding unlabeled and pre-training datasets improves FID without affecting image-text alignment, whereas human-only fine-tuning can overfit and reduce image quality.More diverse datasets produce more natural images on unseen prompts.
5. Discussion
Human-feedback fine-tuning improves alignment in specified count, color, and background domains, but balancing alignment against fidelity remains challenging and several failure modes remain unaddressed.
- Human-feedback fine-tuning improves image-text alignment for specified object counts, colors, and backgrounds.
- The method’s alignment-fidelity tradeoff requires careful investigation of design choices such as auxiliary losses and diverse training data.
- The results do not address all failure modes of existing text-to-image models, positioning the method as a starting point for further study.
- Oversaturated colors may result from similar images being highly ranked in the training set, motivating more nuanced human feedback on diverse failure modes.
- The human dataset is limited to count, color, and background categories with binary good-or-bad feedback, leaving subjective categories and ranking-based feedback for future work.
- Reward-weighted likelihood is used for model updates, while reinforcement-learning fine-tuning is proposed as a future direction that may mitigate reward overfitting through online samples and KL regularization.
A. Qualitative Comparison
Qualitative comparisons show that the fine-tuned model handles specified colors, counts, backgrounds, styles, and broad prompt distributions, while count control remains imperfect.
- The fine-tuned model generates unseen tigers with specified colors.
- The fine-tuned model generates unseen wolves with specified counts, although the counts are not always perfect.
- The fine-tuned model generates cakes with specified backgrounds.
- The fine-tuned model retains the ability to generate rabbits in specified styles.
- The fine-tuned model maintains performance across a wide distribution of text prompts.
B. Image-text Dataset
The image-text dataset uses systematically combined prompts spanning colors, counts, backgrounds, and objects, with human raters evaluating alignment and fidelity on held-out prompts.
- The dataset contains 2774 prompts formed by combining category terms with objects.
- Prompt categories include 9 colors, 6 numbers, 8 backgrounds, and 25 objects.
- Evaluation uses 120 text prompts and asks raters to compare anonymized sets of four images for image-text alignment and fidelity.
- Each evaluation query receives ratings from 9 independent human raters.
C. Additional Results
Additional comparisons evaluate fine-tuned and rejection-sampled Stable Diffusion outputs against the original model for image-text alignment and fidelity, using evaluation objects that include bear, tiger, rabbit, rose, and wolf.
- Table 5 compares fine-tuned-model images with original Stable Diffusion images using alignment and fidelity win, tie, and lose percentages.
- Evaluation uses five objects: bear, tiger, rabbit, rose, and wolf.
- Table 6 compares rejection-sampled original Stable Diffusion images with the original model using alignment and fidelity win, tie, and lose percentages.
- Table 7 compares fine-tuned-model images with rejection-sampled original-model images using alignment and fidelity win, tie, and lose percentages.
D. Experimental Details
The experiments use Stable Diffusion v1.5 as the baseline generator and a CLIP-based MLP reward model, with fidelity measured using FID on MS-CoCo validation data. Training uses AdamW in half-precision, while a smaller pre-training dataset limits the impact of larger batches.
- Model architecture: Stable Diffusion v1.5 serves as the baseline generative model, pretrained on large image-text datasets.
- Model architecture: The reward model combines ViT-L/14 CLIP image-text embeddings with a two-layer MLP using 1024 hidden dimensions per layer.ReLU activations are used between layers.
- Training: AdamW trains the model in half-precision on 4 40GB NVIDIA A100 GPUs.The optimizer uses β1 = 0.9, β2 = 0.999, ϵ = 1e −8 and weight decay 1e −2.
- Evaluation: FID is computed on 40,504 randomly selected caption-image pairs from the MS-CoCo val2014 dataset after resizing images to 256×256.The measurement uses the pytorch-fid Python implementation.
- Training: Using a bigger batch does not make a big difference, which the authors expect is due to the small pre-training dataset.
E. Pseudocode
The pseudocode defines a CLIP-based reward function trained on human labels and augmented with a prompt-classification objective. It also generates negative prompts by randomly varying counts, colors, objects, and locations while excluding the original prompt.
- Reward learning: The reward function encodes an image and text prompt with pretrained CLIP, concatenates their embeddings, and predicts a score with an MLP.
- Reward learning: Reward training minimizes mean squared error between predicted rewards and human labels.
- Reward learning: A prompt-classification loss encourages the original text prompt to receive class 0 relative to perturbed prompts, scaled by λ after division by temperature T.
- Perturbated text prompts: Perturbed prompts are generated by randomly sampling a count, color, location, and object from predefined lists.
- Perturbated text prompts: Each generated prompt follows the template “count color object location” and is retained only when it differs from the original text.