PaperlayerView exact revision on arXiv

Source-linked AI summary

Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models

Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, Tovi Grossman

arXiv:2304.09337v1cs.HCcs.AIcs.MM

TL;DR

Crafting prompts that reflect creative intent remains difficult and often requires trial and error. Promptify combines LLM-based prompt suggestions with image organization and iterative refinement, and a 14-participant study found it more useful than Automatic1111. Its current suggestion engine does not generate negative prompts, and model randomness can limit keyword-specific control.

  • Problem

    Users struggle to craft prompts that models interpret according to their creative intentions, while existing models provide little support for discovering useful keywords.

  • Method

    Promptify combines LLM-based subject and style suggestions with CLIP-based image layout, clustering, and keyword-guided prompt refinement.

  • Results

    Promptify was significantly more useful than Automatic1111 and reduced reported mental demand and frustration in a 14-participant user study.

  • Takeaways & Limitations

    Promptify supports iterative exploration by helping users generate, organize, compare, and refine image collections around desired features.

  • Takeaways & Limitations

    The LLM suggestion engine currently does not suggest negative prompts, although Stable Diffusion supports them.

Abstract

from arXiv · show

Text-to-image generative models have demonstrated remarkable capabilities in generating high-quality images based on textual prompts. However, crafting prompts that accurately capture the user's creative intent remains challenging. It often involves laborious trial-and-error procedures to ensure that the model interprets the prompts in alignment with the user's intention. To address the challenges, we present Promptify, an interactive system that supports prompt exploration and refinement for text-to-image generative models. Promptify utilizes a suggestion engine powered by large language models to help users quickly explore and craft diverse prompts. Our interface allows users to organize the generated images flexibly, and based on their preferences, Promptify suggests potential changes to the original prompt. This feedback loop enables users to iteratively refine their prompts and enhance desired features while avoiding unwanted ones. Our user study shows that Promptify effectively facilitates the text-to-image workflow and outperforms an existing baseline tool widely used for text-to-image generation.

1 INTRODUCTION

Promptify addresses the difficulty of crafting prompts by supporting interactive exploration, image organization, and iterative refinement. Its 14-participant evaluation found it more useful than Automatic1111 for text-to-image generation.

  • Motivation: Prompt crafting is difficult because users must iteratively refine prompts so models interpret their creative intentions correctly.Existing models also lack built-in support for discovering useful keywords and alternative prompt possibilities.
  • System: Promptify supports subject ideation, style expansion, image generation and browsing, and refinement of the original prompt.Users can begin with brief subject and style inputs and receive expanded suggestions.
  • System: Promptify suggests keywords from generated images to help users enhance desired features and avoid unwanted ones during prompt refinement.The suggestion engine uses GPT-3.5 for subject suggestions and few-shot prompting for community-like style keywords.
  • Evaluation: In a 14-participant study, Promptify was significantly more useful than Automatic1111 and supported more appealing first-attempt images with less mental demand and frustration.Participants also found it easier to compare images and ignore unwanted results.
  • System: Promptify provides an interface for flexibly organizing generated images and clustering them by similarity.This supports browsing image collections while users explore prompt alternatives.

2 RELATED WORK

Related work spans text-to-image model development, prompt engineering, design exploration, and interfaces for generative image workflows. Promptify extends these directions by combining prompt assistance with image exploration and organization.

  • Text-to-Image Generative Models: Text-to-image models generate images from language descriptions, progressing from early GAN and transformer systems to diffusion models such as DALL-E-2 and Stable Diffusion.Stable Diffusion generates 512×512 images and is open source, motivating its use as Promptify’s basis.
  • Prompt Engineering: Prompt engineering studies examine how keywords, phrases, and prompting techniques communicate desired outputs to generative models.For text-to-image systems, prior work has focused on identifying useful keywords and testing phrasing effects.
  • Design Exploration and Semantic Editing: Graphics and HCI systems support generating, managing, and exploring design variations through galleries, semantic controls, and dimensionality reduction.These systems provide precedents for exploring alternatives and navigating high-dimensional design spaces.
  • User Interfaces for Image Generation: Generative-image interfaces help users iteratively explore directions, edit content, and translate intentions into text-and-image prompts.Examples include GANzilla, GANravel, Opal, and 3DALL-E.
  • User Interfaces for Image Generation: Promptify contributes a CLIP-similarity-based interface that positions and clusters images for more efficient browsing than a traditional folder view.This connects prompt exploration with visual organization of generated variations.

3 FORMATIVE INTERVIEWS

Formative interviews show that beginners face prolonged prompt-writing and refinement challenges, rely on community resources, and generate large image batches to manage model randomness.

  • 3 FORMATIVE INTERVIEWS: Six Reddit users were interviewed about their experiences generating desirable Stable Diffusion images, with emphasis on beginner challenges.The participants had varied levels of experience and were recruited from active community members.
  • 3.1 Prompt Writing is Difficult for Beginners: Beginners reported substantial difficulty writing effective prompts, sometimes spending months or several evenings experimenting before achieving satisfactory results.These difficulties appeared when creating images for new campaigns, subjects, or styles.
  • 3.1 Prompt Writing is Difficult for Beginners: Limited discoverability of effective keywords led participants to learn from online communities by examining shared prompts and hyperparameters.Participants described following Reddit examples and searching for similar prompts when their own attempts failed.
  • 3.1 Prompt Writing is Difficult for Beginners: Existing prompt retrieval and generation tools could produce stylistically narrow suggestions that users had to filter manually.One participant used regular expressions to remove keywords associated with unwanted styles.
  • 3.2 Iterative Experimentation is Required: Participants often refined prompts iteratively, including controlled comparisons of image sets with and without a new keyword.This process helped users determine how individual keywords affected generated outputs.
  • 3.3 Managing Model Randomness is a Challenge: Model randomness made it difficult to preserve a desired image essence, especially when simple prompts produced varied styles.Participants generated batches of 30–50 images, while users with less powerful GPUs used batches of 2–10.

4 DESIGN GOALS

Promptify’s design goals target novice users’ difficulties with prompt engineering, iterative generation, and examination of many generated images. The system is intended to support creative-intent-preserving refinement and comparison.

  • Target Users: Promptify is designed for novice users, who formative interviews identified as facing significant challenges with text-to-image models.The goals were guided by those interviews.
  • D1. Assisting Users in Prompt Engineering: Promptify assists prompt engineering by generating subject ideas and suggesting keywords that can help users achieve creative objectives.This goal addresses prompt writing as a significant hurdle in prior research and formative study.
  • Generation: Promptify supports an iterative workflow covering subject and style ideation, prompt construction, and refinement based on generated results.The workflow follows users’ needs while developing prompts that reflect creative intentions.
  • D3. Facilitating Examination and Organization of Generated Images: Promptify facilitates comparing, contrasting, and managing large image collections to support exploration of serendipitous results.This goal responds to participants’ reported need to generate substantial numbers of images.

5 THE PROMPTIFY SYSTEM

Promptify supports iterative text-to-image prompt exploration through automatic suggestion, image organization, and refinement feedback. Users can extend subjects and styles, browse image collections, and select modifiers to reinforce desired features or avoid unwanted ones.

  • 5 THE PROMPTIFY SYSTEM: Promptify integrates automatic prompt extension, similarity-based image layout and clustering, and refinement suggestions into an iterative feedback loop.The loop helps users generate images, inspect model interpretations, and edit prompts for subsequent generations.
  • 5.1 Automatic Prompt Suggestion: An atomic input is a basic subject or style description that Promptify expands through separate subject and style suggestion features.Users may manually edit suggestions or opt out of either extension.
  • 5.1.1 Subject Ideation: Users can steer subject ideation by entering natural-language instructions and selecting among generated subject extensions.For example, an initial lion description can be expanded into a detailed scene and revised to change its setting.
  • 5.1.1 Subject Ideation: The workflow moves from atomic subject ideation to style extension, image generation and clustering, then modifier-based prompt refinement.Users may skip suggestion features and manually write prompt portions at any stage.
  • 5.1.1 Subject Ideation: Promptify applies a short style description to suggested subjects and extends it with relevant modifiers, such as lighting, colors, and artistic details.A user-entered style like “studio ghibli” is expanded into a richer style description.
  • 5.1.2 Style Extension: The suggestion menu colors subject phrases orange and style modifiers violet; its first suggestion automatically enters the prompt field for further editing.Users retain control over the suggested prompt text.
  • 5.2 Image Layout and Clustering: Generated images appear on a zoomable 2D canvas where users can reorganize and cluster them by similarity, with cluster colors shown on a minimap.A scale slider adjusts spacing to reduce disruptive overlap while browsing collections.
  • 5.3 Suggesting Modifiers Based on Prior Generation: Refinement modifiers are keywords or phrases associated with individual images or their clusters, enabling users to enhance preferred features or avoid unwanted ones.Users access these modifiers through an image information menu before revising prompts.

6 BACKEND IMPLEMENTATION

Promptify’s backend uses GPT-3.5 with different prompting strategies for subject and style suggestions, CLIP-based image organization, and CLIP Interrogator modifiers for prompt refinement.

  • 6 BACKEND IMPLEMENTATION: Promptify uses GPT-3.5 for prompt suggestions, with zero-shot prompting for subject extension and few-shot prompting using community data for style extension.The implementation is designed to accommodate future useful zero-shot or few-shot prompts.
  • 6 BACKEND IMPLEMENTATION: Zero-shot subject prompting provides instructions without concrete examples to preserve a broad space of exploratory subject ideas.The system asks the LLM to generate three detailed suggestions for an atomic subject.
  • 6 BACKEND IMPLEMENTATION: The system logs the initial query and response, then resends the original and additional user instructions to generate modified suggestions.This supports iterative natural-language steering of subject ideation.
  • 6 BACKEND IMPLEMENTATION: Style extension uses ten semantically similar community prompts retrieved by KNN from a filtered corpus of 500k DiffusionDB prompts.Sentence similarity selects examples for constructing the LLM’s few-shot prompt.
  • 6 BACKEND IMPLEMENTATION: The final style-extension prompt combines retrieved examples with the user’s style input, allowing the LLM to autocomplete additional modifiers.Few-shot examples guide suggestions toward community-like prompt language.
  • 6 BACKEND IMPLEMENTATION: Image layout computes CLIP image embeddings, reduces them to two dimensions with t-SNE, and clusters screen positions using affinity propagation.Coordinates are scaled to maintain a 128-pixel minimum distance, while a slider spreads images apart.
  • 6 BACKEND IMPLEMENTATION: The full-stack system combines Flask and React Flow, GPT-3.5-Turbo, SentenceTransformers embeddings, CLIP embeddings, and Stable Diffusion image generation.The reported hardware generates a 512x512 image in 4 seconds.

7 USER EVALUATION

The evaluation combined comparative testing, free-form observation, and interviews to examine Promptify’s effectiveness and users’ experiences. Fourteen completed studies compared Promptify with Automatic1111 in a within-subject workflow.

  • 7 USER EVALUATION: The user evaluation comprised comparative evaluation, free-form usage observation, and semi-structured interviews focused on system effectiveness, feature use, and obstacles.The comparative evaluation directly contrasted Promptify with Automatic1111.
  • 7 USER EVALUATION: Fourteen participants completed the study, and none had significant Stable Diffusion experience.Fifteen participants were recruited, but one professional visual artist withdrew after repeated NSFW filter blocks while attempting Egon Schiele-style images.
  • 7 USER EVALUATION: The study used a within-subject comparison with counterbalanced system order and the Automatic1111 txt2img interface.Participants performed the same image-generation and refinement workflow with both systems.
  • 7 USER EVALUATION: The comparative evaluation asked participants to generate, review, refine, and repeatedly regenerate images before producing a larger final collection.The procedure included an initial batch of 10 images and a final batch of 50 images.
  • 7 USER EVALUATION: Participants chose favorite and least favorite images during refinement, supporting assessment of how systems helped reinforce desired features and discourage unwanted ones.The workflow repeated refinement twice more before the final generation.
  • 7 USER EVALUATION: After comparative testing, participants freely used Promptify to create images and then discussed their experiences in 10–20-minute semi-structured interviews.Interviews addressed feature effectiveness and encountered difficulties.

7.3 Subjective Ratings

Subjective ratings measured aesthetic appeal, alignment with artistic vision, image-collection management, and task load across the compared systems.

  • 7.3 Subjective Ratings: Participants rated each condition using 5-point Likert scales, except mental-demand, effort, and frustration questions using the 7-point NASA-TLX scale.The ratings were collected after each comparative-evaluation condition.
  • 7.3 Subjective Ratings: Aesthetic-pleasing ratings covered both initial and refined prompts during workflow Steps 1–5.This measured participants’ perceptions of the generated images at both stages.
  • 7.3 Subjective Ratings: Participants rated whether generated images aligned with their artistic vision for both initial and refined prompts.The question was associated with Steps 1–5.
  • 7.3 Subjective Ratings: Image-management ratings assessed tracking favorites, comparing images, and ignoring disliked images across Steps 5–7.These ratings targeted participants’ experience handling large generated collections.

Task Workload.

Promptify received higher average ratings across the evaluated workload measures and was significantly more useful than the baseline. Users also experienced lower mental demand and frustration and found image comparison and filtering easier.

  • Promptify received higher average scores across all reported workload ratings than Automatic1111.The ratings covered mental demand, effort, and frustration.
  • Promptify was significantly more useful for supporting the text-to-image generation workflow than the baseline tool (Z = 11.0, p < .05).
  • Images generated with Promptify users’ initial prompts were significantly more aesthetically pleasing than baseline images (Z = 2.5, p < .05).Promptify’s initial prompts averaged 34.43 words versus 8.6 words for Automatic1111 after stopword removal.
  • Promptify users reported significantly lower mental demand (Z = 9.5, p < .05) and less frustration (Z = 11.0, p < .05).
  • Promptify made it significantly easier to compare and contrast images (Z = 13.0, p < .05) and ignore unwanted images (Z = 13.5, p < .05).These ratings concerned searching within a large image collection.

7.5 User Feedback on Promptify Features

Participants generally rated Promptify’s features as useful and easy to use, while describing how they supported ideation, style refinement, image organization, and prompt modification. Usefulness was lower when suggestions were less relevant to users’ specific goals or when style terms were unfamiliar.

  • 7.5.1 Subject Ideation.: Participants rated Promptify’s subject suggestion engine useful (4.43, STD=0.76) and easy to use (4.21, STD=0.70).They said subject ideation supplied details they might not have considered independently.
  • 7.5.1 Subject Ideation.: Subject extensions were less relevant for participants who already had a concrete target image in mind.One example involved generating a sheep baking a pie.
  • 7.5.2 Style Extension.: Style extension was the most useful feature, with usefulness of 4.57 (STD=0.65) and ease of use of 4.56 (STD=0.51).Participants said it helped them achieve desired styles rapidly and supplied specific style phrases.
  • 7.5.2 Style Extension.: Some participants worried that unfamiliar artist names suggested by style extension could impede prompt comprehension.Participants with stronger knowledge of artistic styles were described as potentially benefiting more from the feature.
  • 7.5.3 Image Layout and Clustering.: Image layout and clustering received usefulness of 4.29 (STD=0.61) and ease of use of 4.36 (STD=0.63).Participants used the feature to manage large batches, compare prompts, group favorites, and identify themes.
  • 7.5 User Feedback on Promptify Features: Figure 9 reports subjective feature ratings, with rows sorted by mean usefulness score and showing median and inter-quartile range.Participants largely agreed that the features were easy to use and useful.
  • 7.5.3 Image Layout and Clustering.: CLIP Interrogator prompt modifiers were rated useful (3.79, STD=0.89) and easy to use (3.86, STD=0.86).Participants used selected modifiers to reinforce favored styles in later generations.
  • 7.5.4 Prompt Modifiers from CLIP Interrogator.: Figure 10 compares first-batch images generated with Promptify in the top row and Automatic1111 in the bottom row.

8 DISCUSSION AND FUTURE WORK

Promptify combines steerable LLM prompt suggestions with image organization and clustering to support iterative text-to-image exploration. The discussion identifies benefits alongside limitations from keyword suggestion, model randomness, prompt length, and fixed generation parameters.

  • 8.2 Automatic Prompt Suggestion with LLMs: Promptify’s few-shot prompting approach can adapt to different prompt formats, whereas fine-tuning is described as expensive and vulnerable to catastrophic forgetting and overfitting.The authors report that initial fine-tuning exploration could cause repeated artist suggestions and loss of prior knowledge useful for suggesting artists and styles.
  • 8.2 Automatic Prompt Suggestion with LLMs: The current LLM engine does not suggest negative prompts, and CLIP Interrogator modifiers did not show statistically significant evidence of better prompt refinement.The paper suggests region- or object-focused captioning and stronger visual-language models as future directions for image-based modifier suggestions.
  • 8.3 Model Randomness and Effectiveness of Individual Keywords: Diffusion randomness and longer prompts can reduce correspondence between intended keywords and generated images, with specific details increasingly overlooked as prompts grow.Promptify mitigates this by semantically organizing images and allowing users to remove undesired results, but the authors note that fixed parameters limited the current studies.
  • 8.4 Contrasting Promptify with Existing Tools: Promptify separately suggests subject matter and style, lets users steer ideation through language, and organizes generated images for browsing and comparison.Its image layout and clustering interface is based on CLIP embeddings, while subject suggestions are steerable and style keywords use few-shot prompting with community examples.
  • 8.4 Contrasting Promptify with Existing Tools: Promptify extends existing prompt-support tools by combining separate subject and style suggestions, steerable ideation, and image layout and clustering.The paper contrasts this combination with tools that share only subsets of Promptify’s features.
  • 8.3 Model Randomness and Effectiveness of Individual Keywords: The study fixed random seed, noise scheduler, and CFG Scale, leaving integration of img2img, inpainting, ControlNet, and other parameters to future work.These constraints define the current experimental scope rather than the full range of Stable Diffusion controls.

9 CONCLUSION

Promptify is an interactive system for exploring and creating text prompts through separate subject and style suggestions, image organization, and image-based refinement. In a 14-participant study, it was significantly more useful than a widely used baseline, supported more visually appealing first attempts, and required significantly less cognitive load.

  • 9 CONCLUSION: Promptify supports interactive prompt exploration through separate LLM suggestions for subject matter and style, image organization, comparison, and image-based keyword refinement.Subject suggestions are steerable through natural language, while style keywords are generated with few-shot prompting from community examples.
  • 9 CONCLUSION: 14 participants found Promptify significantly more useful than a widely used baseline tool, while creating more visually appealing images on their first attempt with significantly less cognitive load.The study evaluated Promptify against Automatic1111 for text-to-image generation.
Loading 2304.09337v1…