Rich Human Feedback for Text-to-Image Generation

• T2I models increasingly support content creation and related applications, but generated images still commonly contain artifacts, implausibility, text misalignment, and low aesthetic quality.Source: §1 · p. 1
• Existing distributional metrics and human-preference models do not reliably capture nuances in individual images.Source: §1 · p. 1
• Single-score alignment metrics and complex question-answering pipelines still do not localize misaligned image regions.Source: §1 · p. 2
• RichHF-18K records artifact and misalignment regions, misaligned prompt words, and four fine-grained scores across 18K generated images.Source: §1 · p. 2
• RAHF predicts localized regions, misaligned keywords, and fine-grained scores to provide more interpretable feedback than scalar image-quality ratings.Source: §1 · p. 2

Text-to-image generation

• T2I generation has evolved through GAN, VAE, diffusion, and latent diffusion architectures.Source: §2 · p. 2

Text-to-image evaluation and reward models

• Recent evaluation and reward models assess generated images along multiple quality dimensions.Source: §2 · p. 2
• Most existing feedback datasets rely on binary ratings or preference rankings rather than detailed actionable annotations.Source: §2 · p. 2
• The annotation interface combines image-region points, prompt-word labels, and plausibility, alignment, aesthetics, and overall-quality scores.Source: Figure 1 · p. 3
• Unlike artifact-only prior work, this paper combines artifact regions, misalignment regions, misaligned keywords, and four fine-grained scores for T2I generation.Source: §2 · p. 3

Dataset and annotations

• RichHF-18K contains artifact/implausibility and misalignment heatmaps, four fine-grained scores, and a misaligned-keyword sequence.Source: §3.1 · p. 3
• Annotators mark flawed image regions, label misaligned prompt keywords, and assign plausibility, alignment, aesthetics, and overall scores.Source: §3.1 · p. 3

Annotation consolidation

• Three annotators label each image-text pair; scores are averaged, keywords use majority voting, and point annotations are converted into heatmaps.Source: §3.2 · p. 3

Dataset construction

• The dataset samples diverse, balanced image-text pairs from Pick-a-Pic, yielding 17K annotated pairs split into 16K training and 1K validation samples.Source: §3.3 · p. 3

Rich feedback model

• The model uses separate vision and text streams, self-attention for fusion, heatmap and score heads, and a Transformer decoder for text generation.Source: Figure 3 · p. 4

Score statistics

• Collected score distributions provide both negative and positive samples for training a reward model.Source: §3.4 · p. 4

Annotator agreement

• Around 25% of samples have perfect annotator agreement, while around 85% have maxdiff at most 0.25 after score standardization.Source: §3.4 · p. 4

Model architecture

• RAHF combines ViT image tokens with embedded prompt tokens through bidirectional Transformer self-attention for rich-feedback prediction.The fused representations support information flow from text to image tokens and from image to text tokens.Source: §4.1.1 · p. 4

Prediction tasks

• Three prediction pathways output implausibility and misalignment heatmaps, fine-grained scalar scores, and misaligned prompt keywords.Heatmaps use convolutional and deconvolutional layers; keyword prediction marks misaligned tokens with a special suffix.Source: §4.1.1 · p. 5

Model variants

• The augmented-prompt variant uses three task-specific heads, conditioning each prediction on an output-type string instead of using seven separate heads.The task string identifies outputs such as an implausibility heatmap during training and inference.Source: §4.1.2 · p. 5

Quantitative results

• Both RAHF variants significantly outperform ResNet-50 or CLIP on the reported score and text-image alignment prediction tasks.For implausibility heatmaps, only the augmented-prompt version outperforms ResNet-50; the multi-head version performs worse.Source: §4.2.2 · p. 7

Limitations

• Misalignment heatmap prediction generally performs worse than artifact/implausibility heatmap prediction, possibly because misalignment regions are less well-defined and annotations noisier.The paper identifies annotation ambiguity as a possible explanation rather than a confirmed cause.Source: §4.2.2 · p. 8

Qualitative results

• Qualitative examples show predicted heatmaps identifying artifact or implausibility regions and objects that do not correspond to the prompt.The paper illustrates these predictions in separate implausibility and misalignment heatmap figures.Source: Figure 5 · p. 5, Figure 6 · p. 5, §4.2.2 · p. 8

Score-guided finetuning

• RAHF scores select high-plausibility generated images for finetuning Muse, testing whether feedback-guided data improves a different generator family.Muse is based on masked transformers, unlike the Stable Diffusion variants used to collect RichHF-18K.Source: §5 · p. 8, Finetuning generative models with… · p. 8

Score-guided generation

• RAHF aesthetic scores are used as Classifier Guidance for Latent Diffusion, illustrating that different fine-grained scores can improve different generation aspects.The paper gives an aesthetic-guidance example alongside plausibility-score finetuning.Source: Finetuning generative models with… · p. 8

Region inpainting

• Predicted implausibility heatmaps are thresholded and dilated into masks, after which Muse inpainting regenerates masked regions and selects outputs by predicted plausibility.This pipeline targets problematic regions rather than regenerating the entire image.Source: Region inpainting with predicted… · p. 8

Results

• Inpainting produces more plausible images with fewer artifacts and demonstrates generalization to a generator unlike those used to train RAHF.The reported examples use Muse inpainting with predicted heatmaps and plausibility scores.Source: Region inpainting with predicted… · p. 8, §5 · p. 8

Conclusions

• RichHF-18K is presented as the first rich human feedback dataset for image generation.The dataset contains fine-grained scores, implausibility and misalignment regions, and misalignment keywords.Source: §6 · p. 8
• A multimodal Transformer predicts the rich feedback, and predicted feedback is demonstrated to improve image generation.The paper presents these improvements as initial demonstrations rather than an exhaustive set of uses.Source: §6 · p. 8

Limitations

• Misalignment-heatmap performance is weaker than implausibility-heatmap performance, partly because some misalignment cases are ambiguous to label.The authors identify improving misalignment-label quality as future work.Source: §6 · p. 8
• The feedback data are limited to generative models beyond Pick-a-Pic, and additional uses such as reinforcement-learning rewards remain unexplored.The paper also mentions predicted heatmap weighting and misaligned sequences as possible future directions.Source: §6 · p. 8

Annotation categories

• Implausibility categories cover distorted bodies, objects, or text, nonsensical representations, excessive blur, and other artifacts.The guidelines distinguish several human, animal, object, text, and image-quality failure types.Source: §8.1 · p. 12
• Text-misalignment labels identify missing entities, incorrect attributes, actions, numbers, positions, and other inconsistencies.Annotators mark the corresponding prompt words and, for wrong actions, the affected image regions.Source: §8.2 · p. 12

Collection procedure

• Annotators mark artifact and misalignment points, label misaligned keywords, and assign plausibility, alignment, aesthetics, and overall scores in one interface.Each point has an effective radius to cover nearby flawed regions.Source: §8.3 · p. 12, §8.4 · p. 13

Dataset composition

• The final training set is around 300 images short of 16K after inappropriate samples are filtered, and its content attributes are relatively balanced.The balanced distribution is reported across the PaLI attributes of the training samples.Source: §8.3 · p. 13

Limitations

• Annotation limitations include minor over-annotation of faces and hands and limited subject diversity dominated by human, dog, and cat prompts.The authors use balanced PaLI categories but leave more diverse image and prompt annotation for future work.Source: §8.4 · p. 13

Model and training

• The model combines a ViT B16 image encoder, a T5-base multimodal encoder, and predictors for scores, heatmaps, and text misalignment.The implementation uses separate prediction heads for the three feedback types.Source: §9 · p. 13
• Training uses batch size 256 for 20K iterations with AdamW and a scheduled learning rate on 64 Google Cloud TPU v3 chips.The learning rate warms up for 2,000 iterations before reciprocal-square-root decay.Source: §9 · p. 13

Augmentation

• Image augmentation randomly crops images and corresponding heatmaps, then applies brightness, contrast, hue, saturation, JPEG-noise, and grayscale transformations.Cropping occurs with 50% probability, while the augmented version and grayscale conversion are each applied probabilistically.Source: §9 · pp. 13–14

Generative-model finetuning

• Prompt synthesis uses five handcrafted seed prompts and PaLM 2 to generate similar prompts with specified lengths and object categories.The procedure is used to construct training prompts for finetuning generative models with predicted scores.Source: §9 · p. 14

Feedback prediction

• RAHF locates implausibility regions across subjects including human hands, animals, vehicles, and concept art.The examples are presented as more accurate artifact and implausibility localization.Source: §10 · p. 14, Figure 11 · p. 15
• RAHF identifies misalignment across animals, objects, and outdoor scenes, including a subtle Segway-handlebar discrepancy.The examples emphasize localization rather than only assigning a global alignment score.Source: §10 · p. 14, Figure 12 · p. 16
• RAHF predicts scores close to human ground-truth scores in the qualitative examples.Ground-truth scores are averages from three annotators.Source: §10 · p. 14, Figure 13 · p. 17
• RAHF predicts the majority of misaligned keywords marked by human annotators.The qualitative examples focus on prompt-word prediction for text-image misalignment.Source: §10 · p. 14, Figure 14 · p. 18

Generation improvement

• Predicted scores are used to select finetuning examples for Muse and as classifier guidance for Latent Diffusion, with enhanced generation quality shown qualitatively.The examples compare generation before and after finetuning and with versus without classifier guidance.Source: §10 · p. 14, Figure 15 · p. 18
• Predicted implausibility heatmaps are converted into masks for Muse region inpainting, producing regions described as significantly more plausible.The illustrated pipeline proceeds from original artifacts to heatmaps, processed masks, and inpainted images.Source: §10 · p. 14, Figure 16 · p. 19