Source-linked AI summary
Rich Human Feedback for Text-to-Image Generation
Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, Junjie Ke, Krishnamurthy Dj Dvijotham, Katie Collins, Yiwen Luo, Yang Li, Kai J Kohlhoff, Deepak Ramachandran, Vidhya Navalpakkam
TL;DR
Generated T2I images still exhibit implausibility, artifacts, text-image misalignment, and low aesthetic quality, while existing feedback often reduces quality to scalar scores. The paper collects richer localized, keyword-level, and multi-score feedback in RichHF-18K, then trains RAHF to predict it and use it for generation improvement. The authors report improvements through data selection, reward guidance, and inpainting, including on Muse beyond the Stable Diffusion-derived training data.
Problem
Generated T2I images still contain artifacts, implausibility, text-image misalignment, and low aesthetic quality, while existing evaluation often compresses quality into single scores.
Method
The paper collects RichHF-18K with region annotations, misaligned prompt words, and four scores, then trains RAHF, a multimodal Transformer, to predict this feedback.
Results
RAHF-predicted feedback improves image generation through problematic-region inpainting and score-based finetuning or reward guidance, including on Muse.
Takeaways & Limitations
Rich human feedback provides an automatic, explainable pipeline that can guide evaluation and generation improvement beyond single scalar ratings.
Takeaways & Limitations
Misalignment heatmap performance is worse, partly because misalignment cases are ambiguous and annotations may be noisy; broader generative-model coverage remains needed.
Abstract
from arXiv · showhide
Recent Text-to-Image (T2I) generation models such as Stable Diffusion and Imagen have made significant progress in generating high-resolution images based on text descriptions. However, many generated images still suffer from issues such as artifacts/implausibility, misalignment with text descriptions, and low aesthetic quality. Inspired by the success of Reinforcement Learning with Human Feedback (RLHF) for large language models, prior works collected human-provided scores as feedback on generated images and trained a reward model to improve the T2I generation. In this paper, we enrich the feedback signal by (i) marking image regions that are implausible or misaligned with the text, and (ii) annotating which words in the text prompt are misrepresented or missing on the image. We collect such rich human feedback on 18K generated images (RichHF-18K) and train a multimodal transformer to predict the rich feedback automatically. We show that the predicted rich human feedback can be leveraged to improve image generation, for example, by selecting high-quality training data to finetune and improve the generative models, or by creating masks with predicted heatmaps to inpaint the problematic regions. Notably, the improvements generalize to models (Muse) beyond those used to generate the images on which human feedback data were collected (Stable Diffusion variants). The RichHF-18K data set will be released in our GitHub repository: https://github.com/google-research/google-research/tree/master/richhf_18k.
1. Introduction
T2I outputs still contain artifacts, implausibility, text-image misalignment, and low aesthetic quality, while existing evaluation often compresses image quality into single scores. The paper addresses this gap with RichHF-18K and RAHF, which provide and predict localized, keyword-level, and multi-score feedback.
- T2I models increasingly support content creation and related applications, but generated images still commonly contain artifacts, implausibility, text misalignment, and low aesthetic quality.
- Existing distributional metrics and human-preference models do not reliably capture nuances in individual images.
- Single-score alignment metrics and complex question-answering pipelines still do not localize misaligned image regions.
- RichHF-18K records artifact and misalignment regions, misaligned prompt words, and four fine-grained scores across 18K generated images.
- RAHF predicts localized regions, misaligned keywords, and fine-grained scores to provide more interpretable feedback than scalar image-quality ratings.
2. Related works
Prior T2I research progressed through multiple generative architectures, while evaluation increasingly used human feedback and reward models. The paper targets the remaining lack of detailed, actionable feedback about image regions and prompt keywords.
- Text-to-image generation: T2I generation has evolved through GAN, VAE, diffusion, and latent diffusion architectures.
- Text-to-image evaluation and reward models: Recent evaluation and reward models assess generated images along multiple quality dimensions.
- Text-to-image evaluation and reward models: Most existing feedback datasets rely on binary ratings or preference rankings rather than detailed actionable annotations.
- Text-to-image evaluation and reward models: The annotation interface combines image-region points, prompt-word labels, and plausibility, alignment, aesthetics, and overall-quality scores.
- Text-to-image evaluation and reward models: Unlike artifact-only prior work, this paper combines artifact regions, misalignment regions, misaligned keywords, and four fine-grained scores for T2I generation.
3. Collecting rich human feedback
RichHF-18K combines region heatmaps, four scores, and misaligned-keyword sequences collected from generated image-text pairs. The paper describes balanced data selection, multi-annotator consolidation, and model architectures that predict these feedback types.
- Dataset and annotations: RichHF-18K contains artifact/implausibility and misalignment heatmaps, four fine-grained scores, and a misaligned-keyword sequence.
- Dataset and annotations: Annotators mark flawed image regions, label misaligned prompt keywords, and assign plausibility, alignment, aesthetics, and overall scores.
- Annotation consolidation: Three annotators label each image-text pair; scores are averaged, keywords use majority voting, and point annotations are converted into heatmaps.
- Dataset construction: The dataset samples diverse, balanced image-text pairs from Pick-a-Pic, yielding 17K annotated pairs split into 16K training and 1K validation samples.
- Rich feedback model: The model uses separate vision and text streams, self-attention for fusion, heatmap and score heads, and a Transformer decoder for text generation.
- Score statistics: Collected score distributions provide both negative and positive samples for training a reward model.
- Annotator agreement: Around 25% of samples have perfect annotator agreement, while around 85% have maxdiff at most 0.25 after score standardization.
4. Predicting rich human feedback
RAHF fuses image and prompt tokens to predict fine-grained scores, implausibility and misalignment heatmaps, and misaligned keywords. Its augmented-prompt variant improves task-specific heatmap prediction over the baselines, while misalignment heatmaps remain harder to predict.
- Model architecture: RAHF combines ViT image tokens with embedded prompt tokens through bidirectional Transformer self-attention for rich-feedback prediction.The fused representations support information flow from text to image tokens and from image to text tokens.
- Prediction tasks: Three prediction pathways output implausibility and misalignment heatmaps, fine-grained scalar scores, and misaligned prompt keywords.Heatmaps use convolutional and deconvolutional layers; keyword prediction marks misaligned tokens with a special suffix.
- Model variants: The augmented-prompt variant uses three task-specific heads, conditioning each prediction on an output-type string instead of using seven separate heads.The task string identifies outputs such as an implausibility heatmap during training and inference.
- Quantitative results: Both RAHF variants significantly outperform ResNet-50 or CLIP on the reported score and text-image alignment prediction tasks.For implausibility heatmaps, only the augmented-prompt version outperforms ResNet-50; the multi-head version performs worse.
- Limitations: Misalignment heatmap prediction generally performs worse than artifact/implausibility heatmap prediction, possibly because misalignment regions are less well-defined and annotations noisier.The paper identifies annotation ambiguity as a possible explanation rather than a confirmed cause.
- Qualitative results: Qualitative examples show predicted heatmaps identifying artifact or implausibility regions and objects that do not correspond to the prompt.The paper illustrates these predictions in separate implausibility and misalignment heatmap figures.
5. Learning from rich human feedback
The authors use predicted RAHF scores and heatmaps to improve generation by filtering finetuning data, applying classifier guidance, and inpainting problematic regions. These interventions improve different generation aspects and generalize to Muse, a model family unlike the feedback-training generators.
- Score-guided finetuning: RAHF scores select high-plausibility generated images for finetuning Muse, testing whether feedback-guided data improves a different generator family.Muse is based on masked transformers, unlike the Stable Diffusion variants used to collect RichHF-18K.
- Score-guided generation: RAHF aesthetic scores are used as Classifier Guidance for Latent Diffusion, illustrating that different fine-grained scores can improve different generation aspects.The paper gives an aesthetic-guidance example alongside plausibility-score finetuning.
- Region inpainting: Predicted implausibility heatmaps are thresholded and dilated into masks, after which Muse inpainting regenerates masked regions and selects outputs by predicted plausibility.This pipeline targets problematic regions rather than regenerating the entire image.
- Results: Inpainting produces more plausible images with fewer artifacts and demonstrates generalization to a generator unlike those used to train RAHF.The reported examples use Muse inpainting with predicted heatmaps and plausibility scores.
6. Conclusions and limitations
The work introduces RichHF-18K and a multimodal Transformer that predicts rich human feedback, with demonstrations of improving image generation. Limitations include weaker misalignment-heatmap performance, limited model diversity, and unexplored uses of the feedback.
- Conclusions: RichHF-18K is presented as the first rich human feedback dataset for image generation.The dataset contains fine-grained scores, implausibility and misalignment regions, and misalignment keywords.
- Conclusions: A multimodal Transformer predicts the rich feedback, and predicted feedback is demonstrated to improve image generation.The paper presents these improvements as initial demonstrations rather than an exhaustive set of uses.
- Limitations: Misalignment-heatmap performance is weaker than implausibility-heatmap performance, partly because some misalignment cases are ambiguous to label.The authors identify improving misalignment-label quality as future work.
- Limitations: The feedback data are limited to generative models beyond Pick-a-Pic, and additional uses such as reinforcement-learning rewards remain unexplored.The paper also mentions predicted heatmap weighting and misaligned sequences as possible future directions.
8. Data collection details
The dataset collection process defines categories for implausibility and text misalignment, gathers multimodal annotations through a unified interface, and consolidates annotations from three annotators. The resulting training set contains around 16K samples with relatively balanced content attributes, while subject diversity and some over-annotation remain limitations.
- Annotation categories: Implausibility categories cover distorted bodies, objects, or text, nonsensical representations, excessive blur, and other artifacts.The guidelines distinguish several human, animal, object, text, and image-quality failure types.
- Annotation categories: Text-misalignment labels identify missing entities, incorrect attributes, actions, numbers, positions, and other inconsistencies.Annotators mark the corresponding prompt words and, for wrong actions, the affected image regions.
- Collection procedure: Annotators mark artifact and misalignment points, label misaligned keywords, and assign plausibility, alignment, aesthetics, and overall scores in one interface.Each point has an effective radius to cover nearby flawed regions.
- Dataset composition: The final training set is around 300 images short of 16K after inappropriate samples are filtered, and its content attributes are relatively balanced.The balanced distribution is reported across the PaLI attributes of the training samples.
- Limitations: Annotation limitations include minor over-annotation of faces and hands and limited subject diversity dominated by human, dog, and cat prompts.The authors use balanced PaLI categories but leave more diverse image and prompt annotation for future work.
9. Experimental details
The experiments use a multimodal model with image, text, score, heatmap, and text-misalignment components, trained with specified optimization and augmentation procedures. A separate prompt-synthesis process supplies prompts for finetuning generative models with predicted scores.
- Model and training: The model combines a ViT B16 image encoder, a T5-base multimodal encoder, and predictors for scores, heatmaps, and text misalignment.The implementation uses separate prediction heads for the three feedback types.
- Model and training: Training uses batch size 256 for 20K iterations with AdamW and a scheduled learning rate on 64 Google Cloud TPU v3 chips.The learning rate warms up for 2,000 iterations before reciprocal-square-root decay.
- Augmentation: Image augmentation randomly crops images and corresponding heatmaps, then applies brightness, contrast, hue, saturation, JPEG-noise, and grayscale transformations.Cropping occurs with 50% probability, while the augmented version and grayscale conversion are each applied probabilistically.
- Generative-model finetuning: Prompt synthesis uses five handcrafted seed prompts and PaLM 2 to generate similar prompts with specified lengths and object categories.The procedure is used to construct training prompts for finetuning generative models with predicted scores.
10. Additional qualitative examples
Qualitative examples show RAHF predicting artifact, misalignment, score, and keyword feedback across varied images. They also illustrate using predicted scores and heatmaps to improve generation through finetuning, classifier guidance, and region inpainting.
- Feedback prediction: RAHF locates implausibility regions across subjects including human hands, animals, vehicles, and concept art.The examples are presented as more accurate artifact and implausibility localization.
- Feedback prediction: RAHF identifies misalignment across animals, objects, and outdoor scenes, including a subtle Segway-handlebar discrepancy.The examples emphasize localization rather than only assigning a global alignment score.
- Feedback prediction: RAHF predicts scores close to human ground-truth scores in the qualitative examples.Ground-truth scores are averages from three annotators.
- Feedback prediction: RAHF predicts the majority of misaligned keywords marked by human annotators.The qualitative examples focus on prompt-word prediction for text-image misalignment.
- Generation improvement: Predicted scores are used to select finetuning examples for Muse and as classifier guidance for Latent Diffusion, with enhanced generation quality shown qualitatively.The examples compare generation before and after finetuning and with versus without classifier guidance.
- Generation improvement: Predicted implausibility heatmaps are converted into masks for Muse region inpainting, producing regions described as significantly more plausible.The illustrated pipeline proceeds from original artifacts to heatmaps, processed masks, and inpainted images.