PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Prompts for Text-to-Image Generation

Yaru Hao, Zewen Chi, Li Dong, Furu Wei

arXiv:2212.09611v2cs.CLcs.CV

TL;DR

Text-to-image prompts are often model-specific and misaligned with user input, making manual prompt engineering laborious. The paper introduces PROMPTIST, which combines supervised fine-tuning with reinforcement learning to adapt inputs into model-preferred prompts while preserving user intentions. On Stable Diffusion, the method outperforms manual prompt engineering, and reinforcement learning is especially beneficial for out-of-domain prompts.

  • Problem

    Prompt engineering for text-to-image models is laborious, model-specific, and difficult to transfer while preserving user intentions.

  • Method

    PROMPTIST fine-tunes a pretrained language model on manually engineered prompts, then uses reinforcement learning with relevance and aesthetic rewards to explore adapted prompts.

  • Results

    PROMPTIST outperforms manual prompt engineering in automatic metrics and human preference ratings, with reinforcement learning bringing 71% average improvement on out-of-domain COCO data.

  • Takeaways & Limitations

    Language models can serve as prompt interfaces that optimize user input into model-preferred prompts, with reinforcement learning improving generalization to out-of-domain examples.

  • Takeaways & Limitations

    The framework is currently applied only to text-to-image models, and its Lexica training prompts contain biases toward artwork and portraits.

Abstract

from arXiv · show

Well-designed prompts can guide text-to-image models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts. Specifically, we first perform supervised fine-tuning with a pretrained language model on a small collection of manually engineered prompts. Then we use reinforcement learning to explore better prompts. We define a reward function that encourages the policy to generate more aesthetically pleasing images while preserving the original user intentions. Experimental results on Stable Diffusion show that our method outperforms manual prompt engineering in terms of both automatic metrics and human preference ratings. Moreover, reinforcement learning further boosts performance, especially on out-of-domain prompts. The pretrained checkpoints are available at https://aka.ms/promptist. The demo can be found at https://aka.ms/promptist-demo.

1 Introduction

Prompt design strongly affects text-to-image quality, but manual, model-specific engineering is laborious and difficult to transfer. PROMPTIST automatically adapts user inputs through supervised fine-tuning and reinforcement learning, improving results over manual prompts and original inputs.

  • Motivation: Text-to-image prompt design is important because limited text encoders can make user intentions harder for models to understand.The introduction specifically identifies the relatively small capacity of encoders such as Stable Diffusion’s CLIP text encoder.
  • Motivation: Manual prompt engineering is laborious, sometimes infeasible, and often does not transfer across model versions.Prior methods typically add model-specific modifiers to the original input.
  • Method: PROMPTIST adapts user inputs to model-preferred prompts using supervised fine-tuning followed by reinforcement learning.The model is fine-tuned on manually engineered prompts, then trained to explore optimized prompts using a reward based on relevance and aesthetics.
  • Results: Experiments on Stable Diffusion found that optimized prompts outperform human-engineered prompts and original inputs under automatic and human evaluations.Human preference ratings showed consistent improvements for both in-domain and out-of-domain prompts.
  • Contribution: The framework is presented as a general prompt optimization approach for aligning user input with model-preferred prompts.The contribution emphasizes automatic adaptation rather than manual prompt construction.

2 Methods

PROMPTIST adapts user inputs into model-preferred prompts through supervised fine-tuning followed by reinforcement learning. Its reward combines relevance to the original input, aesthetic quality, and a KL penalty, while PPO optimizes prompt generation.

  • 2.1 Supervised fine-tuning: PROMPTIST adapts user inputs into model-preferred prompts using a pretrained language model initialized from human-engineered prompt examples.Parallel training data pairs original inputs with manually engineered prompts; supervised fine-tuning initializes the policy before reinforcement learning.
  • 2.1 Supervised fine-tuning: Human-engineered prompts are converted into parallel data by extracting main content, removing or shuffling modifiers, and using the remaining text as source inputs.The targets contain content plus style modifiers such as artist names and popular elements.
  • 2.2 Reward definition: The reward measures relevance and aesthetics, combining CLIP-based similarity to the original input with an aesthetic predictor’s score.Relevance is computed between generated images and the original prompt, while the aesthetic predictor is trained on human ratings.
  • 2.2 Reward definition: An additional KL penalty between the reinforcement-learning policy and the supervised-finetuned model mitigates overoptimization.The coefficient η controls the penalty.
  • 2.3 Reinforcement learning: PPO trains the prompt policy as a language-generating agent that sequentially selects output tokens to maximize accumulated expected reward.The policy and value models are initialized from supervised-finetuned weights, while clipped probability ratios limit large policy updates.

3 Experiments

Experiments evaluate prompt adaptation on Stable Diffusion using automatic rewards, human preferences, and comparisons across in-domain and out-of-domain prompts. Results show gains over manual engineering, with reinforcement learning providing especially strong improvements on unseen inputs.

  • Evaluation setup: Stable Diffusion experiments use automatic reward metrics, human preference ratings, and held-out prompts from Lexica, DiffusionDB, and COCO.Evaluation includes in-domain Lexica augmentations, in-domain DiffusionDB prompts, and out-of-domain COCO data.
  • Prompt optimization results: Optimized prompts achieve higher reward improvements than manual engineering, especially for rephrased and out-of-domain inputs.These results indicate that automatic prompt engineering helps align user-friendly prompts with text-to-image model preferences.
  • Automatic metrics: 0.25 to 0.26: reinforcement learning improves DiffusionDB relevance score, while aesthetic score improves significantly over human-engineered and supervised fine-tuned prompts.The reported relevance score of 0.26 is described as sufficient for generated images to remain relevant to the input prompt.
  • Reinforcement learning: 31%, 24%, and 71%: reinforcement learning improves average rewards on Lexica main content, DiffusionDB, and out-of-domain COCO data, respectively.The largest listed gain occurs on out-of-domain COCO prompts, which were unseen during supervised fine-tuning.
  • Reinforcement learning: 43% and 127%: reinforcement learning improves rephrasings of Lexica main content and target prompts, respectively, despite those augmentations not being used directly.The paper reports stronger gains on prompt types that supervised fine-tuning cannot optimize well.
  • Supervised fine-tuning: Source prompt augmentation during supervised fine-tuning consistently improves performance on both in-domain held-out and out-of-domain data.The augmentation strategy increases source-prompt diversity for better generalization to unseen domains.

4 Related work

The paper situates its work within manual and automatic prompt engineering, learning from human feedback, and text-to-image synthesis research.

  • Manual prompt design has improved model performance but requires time and experience and can be sub-optimal, especially for text-to-image visual styles.
  • Automatic prompt methods search, paraphrase, or generate prompts, while continuous prompt methods optimize prompt parameters directly on downstream tasks.
  • Learning from human feedback has been applied to dialogue systems and other human-in-the-loop machine learning problems.
  • Text-to-image synthesis models generate images conditioned on text and have been developed using GANs, autoregressive models, and diffusion-based models.

5 Conclusion

The paper proposes automatic prompt optimization that aligns user input with model-preferred prompts. On Stable Diffusion, prompt adaptation outperforms human prompt engineering and supervised fine-tuning, while reinforcement learning improves out-of-domain generalization.

  • The framework automatically optimizes prompts so user input and model-preferred prompts are aligned.
  • On Stable Diffusion, prompt adaptation outperforms human prompt engineering and supervised fine-tuning on automatic metrics and human evaluation.
  • Reinforcement learning exploration improves generalization over out-of-domain examples by going beyond teacher forcing.
  • The framework is described as flexible for aligning human intentions with model-favored languages and potentially applicable to other prompt-guided generation tasks.

Limitations

The experiments rely on human-engineered prompts crawled from Lexica, whose distribution contains art-style and portrait biases. The framework has currently been applied only to text-to-image models.

  • Lexica prompts tend to favor artwork over realistic photographs because many contain artist names.
  • Lexica prompts contain a relatively high proportion of portrait examples compared with other categories.
  • The framework currently applies only to text-to-image models, with extension to text-only and text-to-video models left for future work.

A Hyperparameter settings

Table 6 presents the hyperparameter settings used for supervised fine-tuning and reinforcement learning.

  • Table 6 reports hyperparameter settings for both supervised fine-tuning and reinforcement learning.

B Computational budget

The implementation used V100 GPUs, while Table 7 reports the computational budgets for supervised fine-tuning and reinforcement learning.

  • Experiments were implemented on V100 GPUs with 32GB of memory.
  • Table 7 presents the computational budget for supervised fine-tuning and reinforcement learning.
  • The reported budget comparison covers both SFT and RL training stages.

C Results on Stable Diffusion v1.5.

The paper reports results for the proposed method on Stable Diffusion v1.5.

  • The evaluation includes Stable Diffusion v1.5.
  • Table 8 is dedicated to results on Stable Diffusion v1.5.
  • The section presents Stable Diffusion v1.5 as the evaluation setting.

D Comparisons with heuristic baseline

The heuristic baseline combines frequent tags with user inputs, but tag performance varies substantially across domains, whereas the proposed framework performs well across domains.

  • Heuristic baseline: The heuristic baseline uses the top 15 frequent tags from human-engineered prompts.These tags are randomly combined into six groups of common tags.
  • Heuristic baseline: The common-tag baseline concatenates selected tags with user input and evaluates the resulting reward.
  • Cross-domain comparison: Tag3 performs well on COCO and Lexica but poorly on DiffusionDB.This illustrates substantial variation in heuristic-tag performance across domains.
  • Cross-domain comparison: The proposed framework performs well across domains and improves substantially over the common tags.

E Results on different categories and lengths of prompts

The method is evaluated across prompt categories and lengths, with optimized prompts generally improving reward despite slight variation between groups.

  • Prompt categories: The evaluation covers MC, MCM, RMC, RTP, in-domain DiffusionDB, and out-of-domain COCO prompt categories.
  • Prompt categories: Optimized prompts are generally effective across the evaluated prompt categories.
  • Prompt lengths and semantics: Performance varies slightly across different prompt lengths and semantic categories, while reward generally improves.
  • Prompt lengths and semantics: Reinforcement learning uses large-scale in-domain and out-of-domain prompts spanning varied lengths and semantic categories.
Loading 2212.09611v2…