PaperlayerView exact revision on arXiv

Source-linked AI summary

Best Prompts for Text-to-Image Models and How to Find Them

Nikita Pavlichenko, Dmitry Ustalov

arXiv:2209.11711v3cs.HCcs.CLcs.CV

TL;DR

The paper addresses arbitrary prompt keywords and the lack of tools for assessing their aesthetic effects. It uses human pairwise comparisons to rank keyword sets and search for them, reporting improved image aesthetics over popular-keyword and no-keyword baselines.

  • Problem

    Prompt keywords are often chosen by intuition, while tools and analysis for evaluating their aesthetic effects are lacking.

    Source: §1 · p. 1

  • Method

    The authors use crowd workers' pairwise image comparisons to rank keyword sets and use genetic search to find promising sets.

    Source: §1 · p. 1, §2 · p. 2, §4.3 · p. 5

  • Results

    The proposed keywords significantly improve image quality over the fifteen most popular keywords and over using no keywords.

    Source: §4.2 · p. 5, §4.3 · p. 5

  • Takeaways & Limitations

    Released comparisons, images, and code let others extend the keyword search through different initializations, more iterations, or other optimization methods.

    Source: §4.3 · p. 5

    The authors cannot conclude that their keyword set is best: the genetic algorithm may reach a local minimum, tried only 56 of the 100 popular keywords, and rank-based metrics are not sensitive enough to determine convergence.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

Recent progress in generative models, especially in text-guided diffusion models, has enabled the production of aesthetically-pleasing imagery resembling the works of professional human artists. However, one has to carefully compose the textual description, called the prompt, and augment it with a set of clarifying keywords. Since aesthetics are challenging to evaluate computationally, human feedback is needed to determine the optimal prompt formulation and keyword combination. In this paper, we present a human-in-the-loop approach to learning the most useful combination of prompt keywords using a genetic algorithm. We also show how such an approach can improve the aesthetic appeal of images depicting the same descriptions.

1 Introduction

Text-to-image prompts often rely on intuitively chosen keywords, but tools for evaluating their aesthetic effects are lacking. The paper proposes crowd-based evaluation and reports a keyword set that improves image aesthetics.

1 Introduction

  • Prompt keywords are commonly chosen by human intuition, and prompts can look arbitrary.The authors also note that users often judge prompt quality from a single generation or task, with no available analysis of keyword effects.Source: §1 · p. 1
  • The paper proposes large-scale human evaluation of prompt templates using crowd workers.It applies the approach to find keywords for Stable Diffusion.Source: §1 · p. 1
  • The authors introduce a generation-quality evaluation method, propose keywords that improve image aesthetics, and release data and code for reproducibility and further work.They identify finding better keywords and applying the approach to other models as possible uses of the released resources.Source: §1 · p. 1

2 Prompts and How to Evaluate Them

Prompts combine an image description with keywords placed before and after it. The proposed evaluation ranks keyword sets through human pairwise comparisons of generated images.

2 Prompts and How to Evaluate Them

  • A prompt places keywords before and after the image description.The paper illustrates this structure with a calico-cat prompt whose description is surrounded by artistic and stylistic keywords.Source: §2 · p. 2
  • Because aesthetic quality is difficult to evaluate computationally, the method infers keyword-set rankings from pairwise human judgments of images generated for the same description.The resulting average rank per description is used as the keyword set's quality measure.Source: §2 · p. 2
  • The method generates four images per description and keyword set, then has crowd workers choose the better image in sampled pairs without seeing the keyword set.For each description, Bradley-Terry ranking [1] aggregates comparisons into an ordering of keyword sets; their average rank across descriptions measures quality.Source: §2 · p. 2

3 Iterative Estimation of the Best Keyword Set

The approach supports iterative keyword search: it evaluates new candidates with additional human comparisons and uses genetic optimization to propose subsequent candidates.

3 Iterative Estimation of the Best Keyword Set

  • New keyword sets can be evaluated with a small additional number of comparisons, enabling iterative discrete optimization.The process evaluates each new candidate using comparisons with already evaluated sets and then applies the quality-evaluation steps.Source: §3 · pp. 2–3
  • The genetic algorithm proposes candidates from the currently evaluated keyword sets, with the highest-ranked sets used for crossover and mutation.The resulting candidate is evaluated through the same image-generation and pairwise-comparison scheme.Source: §3 · p. 3, Figure 2 · p. 3
  • For each new candidate, the method samples comparisons with prior sets to sustain a total of kn log2 n comparisons.The candidate's quality is then assessed using the evaluation procedure described earlier.Source: §3 · p. 3

4 Experiment

The experiment evaluates genetic-algorithm prompt-keyword search for Stable Diffusion using crowdsourced aesthetic comparisons. The authors report better results than the popular-keyword baseline, while noting limits on search coverage and convergence assessment.

4.1 Setup

  • Setup sampled 12 raw descriptions from each of six categories, generated four Stable Diffusion images per description, and used pairwise crowdsourcing with k=3.Source: §4.1 · pp. 3–4

4.2 Results

  • The genetic algorithm found keyword sets significantly better than Top-15, and any keyword set significantly outperformed no keywords; most results held on validation, with noisier metrics.Source: Table 1 · p. 4, §4.2 · p. 5

4.1 Setup

  • The search used a genetic algorithm with 100-bit keyword masks, initialized with empty and Top-15 sets, and capped outputs at 15 keywords to avoid overly long prompts.Source: §4.1 · p. 4

4.3 Discussion

  • A random-forest analysis identified “colorful background” as the most important keyword, while the most important keywords differed from widely used ones such as “trending on artstation.”Source: §4.3 · p. 5
  • The authors cannot conclude the found set is best: the search tried only 56 of 100 popular keywords, and rank-based metrics cannot establish convergence.Source: §4.3 · p. 5
  • The released comparisons, images, and code let others test new candidates against existing images and add those results to the dataset.Source: §4.3 · p. 5

5 Related Work

The related work contrasts scalar aesthetic ratings with pairwise comparisons and places keyword optimization among prompt-engineering research.

5 Related Work

  • Existing aesthetic datasets use human ratings on 1-to-5 scales, whereas the authors favor pairwise comparisons because people perceive scales subjectively.The paper also notes that prior approaches train aesthetic evaluators rather than evaluating generative models.Source: §5 · p. 5
  • The authors situate their work after prompt-engineering research enabled by GPT-3 and text-to-image findings that added keywords help [7].Source: §5 · pp. 5–6
  • To their knowledge, they are the first to apply prompt engineering to finding the best keywords.Source: §5 · p. 6

6 Conclusion

The conclusion reports improved Stable Diffusion results over popular community keywords and presents the evaluation approach as applicable beyond text-to-image prompts.

6 Conclusion

  • The selected Stable Diffusion keyword set produced better results than the most popular keywords used by the community.Source: §6 · p. 6
  • The evaluation approach can assess arbitrary prompt templates, including in text-to-text settings.Source: §6 · p. 6
  • The authors release code and data and encourage further experiments to find better keyword sets.Source: §6 · p. 6

Appendix

The appendix describes how the authors selected candidate keywords and sourced image descriptions for evaluation.

A Keyword Selection

  • The authors counted comma-separated phrases from Stable Diffusion Discord prompts, took the 100 most popular, then manually removed common phrases that were not keywords.Source: A Keyword Selection · p. 8

B Image Descriptions

C Annotation

Annotation used pairwise image-set choices, with worker qualification and golden tasks used to screen annotators.

C Annotation

  • Workers compared two four-image sets for a description without prompt keywords and selected the more appealing set.Source: C Annotation · p. 8
  • Workers first had to distinguish Stable Diffusion from DALL-E Mini images; failing a golden task led to disqualification, and pages completed in under 15 seconds triggered suspension.Source: C Annotation · p. 8
  • 12,724 workers annotated 597,830 pairs, with 84% accuracy on golden tasks.Source: C Annotation · p. 8

D Keywords Optimization

A simple genetic algorithm searches keyword sets represented as binary masks, selecting high-average-rank candidates and varying them through crossover and mutation.

  • Keyword sets are encoded as 100-bit masks, with each bit indicating whether its keyword is appended to the prompt.Source: D Keywords Optimization · p. 8
  • The search starts from an empty mask and a mask containing the 15 most popular keywords, then selects the two masks with the highest average rank.Source: D Keywords Optimization · p. 8
  • Crossover swaps a random segment between selected masks, while mutation flips offspring bits with 1% probability.Source: D Keywords Optimization · p. 8

E Keyword Importance

The section describes keyword-set rankings and estimates the importance of the top 15 keywords with a random forest, while cautioning that importance is not equivalent to quality.

Loading 2209.11711v3…