Aligning Text-to-Image Models using Human Feedback

• Current text-to-image models often fail on reliable visual text and compositional image generation, motivating improved prompt alignment.Source: §1 · p. 1
• The method samples diverse outputs for alignment-focused prompts, collects binary human feedback, trains a reward function, and fine-tunes with reward-weighted likelihood.An auxiliary prompt-classification objective identifies the original prompt among perturbed alternatives.Source: §1 · p. 1, Figure 1 · p. 2
• 27K human-feedback image-text pairs are used to fine-tune Stable Diffusion for objects with specified colors, counts, and backgrounds.Source: §1 · p. 1
• Up to 47% improvement in image-text alignment is achieved on human evaluation, at the expense of mildly degraded image fidelity.Source: §1 · p. 2
• The learned reward function predicts human quality assessments more accurately than the CLIP score, and rejection sampling based on it can improve image-text alignment.Source: §1 · p. 2
• Naive human-feedback fine-tuning can reduce image fidelity, so design choices such as auxiliary losses and training-data diversity require careful investigation.Source: §1 · p. 2

• Text-to-image systems use varied generative architectures with large-scale language encoders and have achieved impressive generation results.Source: §2 · p. 2
• Existing models frequently fail at reliable visual text and compositional generation, prompting techniques such as character-aware encoders and structured language representations.Source: §2 · p. 2
• Few-image fine-tuning methods such as DreamBooth personalize text-to-image diffusion models, while this work targets alignment using human feedback.Source: §2 · p. 3
• Human feedback has been applied across translation, question answering, story generation, and instruction-following systems.Source: §2 · p. 3
• Image-text alignment is commonly evaluated with embedding-based scores from pretrained multimodal models such as CLIP and CoCa.Source: §2 · p. 3

3. Main Method

• The method generates prompt-targeted images, collects binary human judgments, trains a reward model, and fine-tunes the generator using reward-weighted log likelihood.Prompts test specified object counts, colors, backgrounds, and their combinations.Source: §3 · p. 3, §3.1 · p. 3, §3.3 · p. 4
• The reward function maps CLIP image-text embeddings to a scalar prediction of human feedback, trained by minimizing mean-squared error.Feedback uses y ∈ {0, 1}, where 1 denotes good alignment and 0 denotes bad alignment.Source: §3.2 · p. 3
• The study limits prompts mainly to count, color, and background categories and defers broader categories, ranking feedback, and RLHF-style updates to future work.The authors note that RLHF could offer benefits but would require extensive hyperparameter tuning and engineering.Source: §3.2 · p. 3, §3.1 · p. 3, §3.3 · p. 4
• Prompt classification augments good image-text pairs with semantically different prompts and adds an auxiliary loss to improve reward-model generalization to unseen images and text prompts.The combined reward-learning loss is LMSE(φ) + λLpc(φ), where λ is a penalty parameter.Source: §3.2 · p. 4
• The generator loss combines reward-weighted negative log likelihood on model-generated data with pre-training loss on Dpre to reduce overfitting and produce more natural images.The model-generated data can combine human-labeled and additional unlabeled generated images.Source: §3.3 · p. 4

Experimental Setup

• The experiments use Stable Diffusion v1.5 as the baseline, freeze its CLIP language encoder, and fine-tune only the diffusion module.The reward model uses CLIP embeddings as input to an MLP.Source: §4.1 · pp. 4–5
• The study generates 27K images from 2,700 English prompts, with additional unlabeled and pre-training data used during fine-tuning.The pre-training data is a 625K filtered subset of LAION-5B.Source: §4.1 · p. 5

Human Evaluation

• On 120 prompts, human raters compare fine-tuned and original models for alignment and fidelity using nine independent ratings per query.The prompts include 60 seen and 60 unseen prompts, with ties allowed for similar quality.Source: §4.2 · pp. 5–6

Text-Image Alignment Results

• 50% of samples from the fine-tuned model receive at least two-thirds of votes for image-text alignment, compared with 15% versus 10% for image fidelity.The authors attribute the fidelity decrease to alignment-focused labels, limited human-data diversity, and a small pre-training subset.Source: §4.2 · p. 6

Qualitative Comparison

• Qualitative comparisons show better adherence to specified colors, counts, and backgrounds, including unseen objects and unseen artistic text categories.The original model often omits prompt details, whereas the fine-tuned model produces reasonable images for unseen categories.Source: Figure 2 · p. 5, §4.2 · p. 6

Reward Learning

• The learned reward predicts human preferences more accurately than CLIP, while prompt-classification loss improves performance on seen and unseen prompts.Reward accuracy on unseen prompts is approximately 80%, and reduced human-data size lowers accuracy on both prompt types.Source: §4.3 · p. 7
• Rejection sampling improves alignment to 46% two-thirds preference without sacrificing fidelity, but fine-tuning adds a 10% alignment gain while sacrificing 17% fidelity against that baseline.Rejection sampling generates 16 images and selects the four highest-reward outputs, increasing inference-time computation and leaving the model unchanged.Source: §4.3 · pp. 7–8

Ablation Studies

• Adding unlabeled and pre-training datasets improves FID without affecting image-text alignment, whereas human-only fine-tuning can overfit and reduce image quality.More diverse datasets produce more natural images on unseen prompts.Source: Figure 5 · p. 7, §4.4 · p. 8

• Human-feedback fine-tuning improves image-text alignment for specified object counts, colors, and backgrounds.Source: §5 · p. 8
• The method’s alignment-fidelity tradeoff requires careful investigation of design choices such as auxiliary losses and diverse training data.Source: §5 · p. 8
• The results do not address all failure modes of existing text-to-image models, positioning the method as a starting point for further study.Source: §5 · p. 9
• Oversaturated colors may result from similar images being highly ranked in the training set, motivating more nuanced human feedback on diverse failure modes.Source: §5 · p. 9
• The human dataset is limited to count, color, and background categories with binary good-or-bad feedback, leaving subjective categories and ranking-based feedback for future work.Source: §5 · p. 9
• Reward-weighted likelihood is used for model updates, while reinforcement-learning fine-tuning is proposed as a future direction that may mitigate reward overfitting through online samples and KL regularization.Source: §5 · p. 9

• The fine-tuned model generates unseen tigers with specified colors.Source: Figure 7 · p. 12
• The fine-tuned model generates unseen wolves with specified counts, although the counts are not always perfect.Source: Figure 8 · p. 13
• The fine-tuned model generates cakes with specified backgrounds.Source: Figure 9 · p. 14
• The fine-tuned model retains the ability to generate rabbits in specified styles.Source: Figure 10 · p. 15
• The fine-tuned model maintains performance across a wide distribution of text prompts.Source: Figure 11 · p. 16

• The dataset contains 2774 prompts formed by combining category terms with objects.Source: B. Image-text Dataset · p. 17
• Prompt categories include 9 colors, 6 numbers, 8 backgrounds, and 25 objects.Source: B. Image-text Dataset · p. 17
• Evaluation uses 120 text prompts and asks raters to compare anonymized sets of four images for image-text alignment and fidelity.Source: B. Image-text Dataset · p. 17
• Each evaluation query receives ratings from 9 independent human raters.Source: B. Image-text Dataset · p. 17

• Table 5 compares fine-tuned-model images with original Stable Diffusion images using alignment and fidelity win, tie, and lose percentages.Source: Table 5 · p. 17
• Evaluation uses five objects: bear, tiger, rabbit, rose, and wolf.Source: C. Additional Results · p. 17
• Table 6 compares rejection-sampled original Stable Diffusion images with the original model using alignment and fidelity win, tie, and lose percentages.Source: Table 6 · p. 18
• Table 7 compares fine-tuned-model images with rejection-sampled original-model images using alignment and fidelity win, tie, and lose percentages.Source: Table 7 · p. 18

Model architecture

• Stable Diffusion v1.5 serves as the baseline generative model, pretrained on large image-text datasets.Source: D. Experimental Details · p. 18
• The reward model combines ViT-L/14 CLIP image-text embeddings with a two-layer MLP using 1024 hidden dimensions per layer.ReLU activations are used between layers.Source: D. Experimental Details · p. 18

Training

• AdamW trains the model in half-precision on 4 40GB NVIDIA A100 GPUs.The optimizer uses β1 = 0.9, β2 = 0.999, ϵ = 1e −8 and weight decay 1e −2.Source: D. Experimental Details · p. 18

Evaluation

• FID is computed on 40,504 randomly selected caption-image pairs from the MS-CoCo val2014 dataset after resizing images to 256×256.The measurement uses the pytorch-fid Python implementation.Source: D. Experimental Details · p. 18

Training

• Using a bigger batch does not make a big difference, which the authors expect is due to the small pre-training dataset.Source: D. Experimental Details · p. 18

Reward learning

• The reward function encodes an image and text prompt with pretrained CLIP, concatenates their embeddings, and predicts a score with an MLP.Source: E. Pseudocode · p. 19
• Reward training minimizes mean squared error between predicted rewards and human labels.Source: E. Pseudocode · p. 19
• A prompt-classification loss encourages the original text prompt to receive class 0 relative to perturbed prompts, scaled by λ after division by temperature T.Source: E. Pseudocode · p. 19

Perturbated text prompts

• Perturbed prompts are generated by randomly sampling a count, color, location, and object from predefined lists.Source: E. Pseudocode · p. 20
• Each generated prompt follows the template “count color object location” and is retained only when it differs from the original text.Source: E. Pseudocode · p. 20