PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation

Yingchaojie Feng, Xingbo Wang, Kam Kwai Wong, Sijia Wang, Yuhong Lu, Minfeng Zhu, Baicheng Wang, Wei Chen

arXiv:2307.09036v2cs.AIcs.HC

TL;DR

PromptMagician addresses the difficulty of refining ambiguous natural-language prompts for desired text-to-image results. It retrieves related prompt-image pairs, recommends important keywords, and visualizes images and keywords for interactive exploration. Usage scenarios, a user study, and expert interviews demonstrate its effectiveness and usability, while broader hyper-parameter tuning remains future work.

  • Problem

    Complex and ambiguous natural-language prompts make it difficult to obtain desired text-to-image results and refine prompts efficiently.

  • Method

    PromptMagician retrieves related prompt-image pairs from DiffusionDB, extracts important keywords, and co-embeds images and keywords for interactive exploration.

  • Results

    Two usage scenarios, a user study, and expert interviews demonstrate PromptMagician’s effectiveness and usability.

  • Takeaways & Limitations

    The system supports iterative prompt refinement and personalized evaluation by jointly exploring generated images, retrieved images, and recommended keywords.

  • Takeaways & Limitations

    PromptMagician currently supports text prompts and guidance scales, while tuning additional hyper-parameters remains future work.

Abstract

from arXiv · show

Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refinement, PromptMagician introduces a multi-level visualization for the cross-modal embedding of the retrieved images and recommended keywords, and supports users in specifying multiple criteria for personalized exploration. Two usage scenarios, a user study, and expert interviews demonstrate the effectiveness and usability of our system, suggesting it facilitates prompt engineering and improves the creativity support of the generative text-to-image model.

1 INTRODUCTION

PromptMagician addresses the difficulty of crafting effective text-to-image prompts by combining prompt recommendations with visual exploration of generated and retrieved images. Its studies report effectiveness and usability for interactive prompt engineering.

  • Natural-language complexity and ambiguity make effective prompt development difficult, especially for novice users seeking desired images.
  • PromptMagician generates image collections across hyper-parameter values, retrieves related prompt-image pairs, and supports personalized filtering of image results.Users can specify evaluation criteria such as image quality or abstract perception to focus exploration on relevant subsets.
  • Its prompt recommendation model identifies important and relevant keywords from retrieved image clusters to guide prompt improvement.
  • The visual system helps users explore and evaluate model-generated results while conducting interactive prompt engineering.
  • Two usage scenarios, a user study, and expert interviews demonstrate the system’s effectiveness and usability.

2 RELATED WORK

Prior work supports prompt engineering and large-scale image exploration, but PromptMagician targets the distinct evaluation and refinement needs of text-to-image generation. It combines semantic image exploration with prompt and hyper-parameter considerations.

  • Prompt engineering has been studied for model interaction, while text-to-image systems require users to refine prompts for subjective visual outputs.
  • Most prior prompt-formulation systems target text-to-text models with label-based quantitative evaluation, whereas this work focuses on text-to-image outputs and evaluations.
  • Image-exploration research uses visual summaries and semantic layouts, but PromptMagician additionally considers image styles and model hyper-parameters for prompt refinement.
  • Text-to-image generation translates natural-language descriptions into realistic images using encoder-decoder architectures with GAN-based or diffusion-based decoders.

3 OVERVIEW

PromptMagician combines generated and retrieved image exploration with prompt recommendation and visual summarization. Its design responds to users’ needs for varied results, inspiration, overview, evaluation, and iterative refinement.

  • 3.1 Background: DiffusionDB contains 14 million prompt-image and hyper-parameter data pairs, including 1.8 million unique text prompts.
  • 3.2 Design Requirements: The study targets ordinary image-creation users who lack professional-tool expertise and recruited 9 participants from local universities.
  • 3.2 Design Requirements: The system generates collections of images using varied guidance scales and random seeds so users can compare results efficiently.
  • 3.2 Design Requirements: It retrieves similar prior image works and their prompts to inspire users’ prompt refinement.
  • 3.2 Design Requirements: Visual summaries help users overview image characteristics and navigate large collections toward subsets for detailed exploration.
  • 3.2 Design Requirements: The workflow combines user inputs, Stable Diffusion generation, DiffusionDB retrieval, keyword identification, and joint visualization of images and keywords.

4 PROMPT RECOMMENDATION

PromptMagician recommends prompt keywords by retrieving, embedding, and hierarchically clustering similar DiffusionDB images, then mining and matching keywords to their most related clusters.

  • 4.1 Image Retrieval: The model retrieves images similar to the user prompt from DiffusionDB, using image features because similar prompts can generate substantially different images.CLIP aligns text and image features, and cosine distance measures prompt-image similarity.
  • 4.2 Image Embedding: Retrieved images and their prompts are concatenated into 1,024-dimensional representations after separate 512-dimensional CLIP encodings.This joint representation captures semantic features such as subjects and styles.
  • 4.3 Hierarchical Clustering: Hierarchical clustering organizes semantically similar images into a bottom-up tree without pre-specifying the number of clusters.Leaves begin as individual clusters and progressively merge into larger non-leaf clusters.
  • 4.3 Hierarchical Clustering: Clusters are constrained to 3–20 child nodes and distant merged sub-clusters are discarded to preserve clusters suitable for keyword mining.These constraints reduce overly broad clusters that contain generic keywords and obscure specific ones.
  • 4.4 Prompt Keyword Mining: Cluster-level TF-IDF identifies keywords that are more important in the current cluster than across the retrieved prompt set.Generic keywords may have high term frequency but low inverse document frequency, making them less likely to rank as special.
  • 4.4 Prompt Keyword Mining: The mining process incorporates multi-word n-grams, removes stop words, preserves connecting words inside phrases, and selects top keywords by importance.Examples include “unreal engine” and “trending on Artstation.”
  • 4.5 Prompt-Cluster Matching: Each keyword is assigned to the cluster with its highest normalized TF-IDF value, linking keywords to the image group where they are most relevant.This mapping supports joint visualization of images and text for user exploration and comprehension.
  • 4.5 Prompt-Cluster Matching: Redundant individual words are removed when an n-gram expresses the same cluster association, retaining phrases such as “unreal engine” instead.The words “unreal” and “engine” are eliminated when the combined phrase has similar importance in the same cluster.

5 SYSTEM DESIGN

PromptMagician combines multiple coordinated views for exploring generated and retrieved images, inspecting prompt keywords, and interactively evaluating and refining image results. Its visualization links images and keywords through semantic embeddings and hierarchical clusters, while image evaluation supports criteria-based filtering; natural-language evaluation remains future work.

  • Interface: PromptMagician combines four views for prompt input, image browsing, image evaluation, and local exploration.The views support hyper-parameter customization, exploration of generated and retrieved images, criteria-based filtering, and detailed inspection of selected results.
  • Image Browser View: The Image Browser View co-embeds generated and retrieved images with prompt keywords in a multi-level semantic visualization.Images are positioned by semantic embeddings, keywords near related image clusters, and representative images reduce overview-level clutter.
  • Image Evaluation View: Users can specify opposing keyword criteria, receive image ratings, and brush rating distributions to filter images from multiple aspects.The system supports objective and subjective criteria, including aesthetic properties, and can generate an opposing keyword with “not” when needed.
  • Image Evaluation View: The image rating uses Softmax over cosine similarities between each image and two opposing text descriptions, producing a value on [0,1].Values near 0 or 1 indicate greater similarity to the corresponding keyword, and opposing keywords reduce ambiguity relative to a single keyword.
  • Image Evaluation View: The evaluation strategy is currently limited to opposing keywords, while extending it to natural-language sentences is left for future work.The authors state that sentences could better distinguish image nuances but do not implement that extension here.

6 USAGE SCENARIOS

The usage scenarios show PromptMagician supporting both targeted prompt refinement and open-ended creation. Users inspect image collections, explore recommended keywords, apply selection criteria, revise prompts, and obtain more satisfactory results.

  • Targeted prompt refinement: In the targeted-style scenario, the user rejects initial cat images, explores similar images, and uses “cute” and “ugly” criteria to select relevant results.Brushing toward “cute” images reveals a Totoro style and supports deeper inspection of selected results.
  • Targeted prompt refinement: Recommended keywords “Hayao Miyazaki,” “Studio Ghibli,” and “Totoro” reveal style associations, but the user discards “Totoro” because it changes the subject.She keeps the other two keywords to refine the original prompt and narrows the hyper-parameter range.
  • Open-ended creation: In the open-ended scenario, unexpected film-like results lead the user to explore retrieved robot and city images and clarify the intended subjects.Recommended keywords help the user add a second sentence describing cyberpunk robots walking through a futuristic city.
  • Open-ended creation: The user addresses vague robots by filtering for detailed images and adding “8k,” “highly detailed,” and “unreal engine” to the prompt.The resulting collection contains more images with high details and textures, from which the user selects a final outcome.

7 USER STUDY

The user study evaluates PromptMagician’s recommendation model, overall system effectiveness and usability, and creativity support against two Stable Diffusion-based baselines. The baselines isolate the effects of keyword recommendation and retrieved-image exploration while providing alternative prompting support.

  • Study goals: The user study evaluates recommendation helpfulness, overall effectiveness and usability, and creativity support against two baseline systems.The study compares interactive prompt engineering and image creation across these evaluation targets.
  • Participants: Twelve participants aged 24–32, including students from several disciplines, took part and had limited knowledge of generative models.Participants had varying experience with text-to-image tools but lacked sufficient knowledge of how to use the models effectively.
  • Baselines: All three systems use Stable Diffusion as the text-to-image backbone, enabling comparison of their interaction and prompting features.The systems differ in retrieval, recommendation, and automatic prompting support rather than backbone model.
  • Baselines: Baseline A retains similar-image retrieval but removes prompt keyword recommendation, approximating inspiration from public artwork platforms.This isolates the contribution of PromptMagician’s keyword recommendation feature.
  • Baselines: Baseline B shows only generated images and uses Promptist to automatically refine prompts for aesthetic quality.It excludes image retrieval and PromptMagician’s keyword recommendation feature.

7.3 Procedure and Tasks

The study procedure combines system introduction, target-replication training, open-ended creation across three systems, and questionnaire-based evaluation followed by interviews. These stages assess usability, effectiveness, creativity support, and participant feedback.

  • Introduction: Participants received a 15-minute introduction covering the research background, protocol, system views, and usage examples.Demographic information and consent to record operations and results were collected during this stage.
  • Target replication training: A 15-minute target-replication task familiarized participants with the systems using given images without textual descriptions or generation metadata.The training used the Jeopardy evaluation methodology and included PromptMagician and Promptist.
  • Open-ended creation: During 60 minutes of open-ended creation, participants used all three systems for up to 20 minutes each under a broad theme.They selected a theme, wrote a prompt, and created images without detailed target constraints.
  • Evaluation and interview: A five-point Likert questionnaire assessed effectiveness, usability, and creativity support, followed by semi-structured interviews for additional feedback.The questionnaire covered PromptMagician’s system and recommendation model and creativity support across all three systems.

7.4 Results Analysis

Participants generally found PromptMagician’s keyword recommendations and visual exploration features useful, intuitive, and easy to use. Compared with two baselines, the system received higher creativity-support ratings across all evaluated dimensions.

  • Prompting Model: Participants considered the recommended keywords meaningful, understandable, prompt-related, and helpful for improving stylization and image quality.Users also reported that keywords could reveal indirect associations and help control image attributes such as sky color and aesthetics.
  • Visual System: Semantic grouping, multi-level visualization, and criterion-based filtering helped participants compare images, narrow exploration, and understand keyword effects.The visual system also supported keyword validation by exposing common combinations and helping users exclude irrelevant or overly general terms.
  • Usability: All participants rated the system easy to learn and use, while appreciating its intuitive workflow and generation-time reminders.One participant suggested adding a mini-map to improve Image Browser navigation.
  • Creativity Support Comparison: PromptMagician outperformed two baselines on every evaluated Creativity Support Index dimension.The comparison assessed the three systems during creation tasks while excluding the inapplicable collaboration dimension.
  • Creativity Support Comparison: Compared with Baseline A, keyword recommendations and contextual highlighting made alternative-keyword selection and analysis less tedious.Participants also reported that this design motivated more creation attempts.
  • Creativity Support Comparison: Compared with Baseline B, co-visualizing keywords and images helped users understand keyword effects and stimulated greater creativity.Baseline B’s automatic addition of art disciplines and artist names was perceived as straightforward but sometimes produced unexpected image styles.

7.5 Notable Observations

Users employed three recurring prompt-engineering patterns: progressively adding detail, replacing detailed descriptions with summaries, and selectively substituting keywords. These patterns suggest opportunities for adaptive keyword recommendations.

  • Prompting Patterns: Users commonly began with basic sentences and progressively added keywords to enhance image details.This was the most common observed prompting pattern.
  • Prompting Patterns: When detailed descriptions failed, users used summarizing suggestions such as “summer afternoon” to achieve desired color effects with less confusion.The summary represented the coloring of sky and clouds more effectively than a detailed description.
  • Prompting Patterns: Users selectively replaced keywords when substitutions improved generated composition, posture, or background.Replacing “carpet” with “pillow” improved both body posture and background in one example.
  • Implications: These prompting patterns and user intents could support adaptive recommendations for selecting and refining keywords.The observation connects recurring user behavior with a proposed direction for recommendation design.

8 DISCUSSION

The discussion frames PromptMagician as a breadth-first, multimodal workflow for creative exploration, evaluation, and prompt refinement. It also identifies open challenges in aesthetic evaluation, iterative stopping, hyper-parameter control, and model-assisted prompt improvement.

  • Design Implications: PromptMagician uses breadth-first exploration to turn abstract ideas into concrete visuals and streamline early-stage creative coordination and refinement.The approach combines database querying with ad-hoc image generation rather than relying only on conventional image browsing.
  • Design Implications: Experts viewed machine evaluation of large image collections as useful, while suggesting art-based evaluators and multiple concepts for multidimensional aesthetic assessment.The discussion gives “uncanny valley” detection as an example of an aesthetic aspect requiring richer evaluation.
  • Design Implications: Users often struggled to decide when to stop fine-tuning because of uncertainty about model capability or failure to achieve desired results.The paper identifies guidance for the iterative process as a need currently addressed mainly through expert experience and heuristics.
  • Fine-tuning: Multimodal conditioning and interactive modification methods provide related routes for fine-tuning generated outputs beyond prompt changes.The discussion cites text-guided inpainting, ControlNet, and DragGan as examples of interactive image modification.
  • Generalizability: The system could generalize to content moderation and model analysis by examining harmful prompts and large sets of generated outputs.These applications include scrutinizing word groups and assessing precision, stylization, and diversity.
  • Limitations and Future Work: PromptMagician currently exposes text prompts and guidance scales, leaving other quality-relevant hyper-parameters for future tuning support.The planned extension aims to add flexible control without increasing learning cost or usage complexity.
  • Limitations and Future Work: Future work could use large language models to recommend keywords or revise prompts from images of user interest.The proposed direction is intended to extend the existing DiffusionDB-based retrieval and keyword-identification workflow.

9 CONCLUSION

PromptMagician combines visual analytics, retrieval, keyword recommendation, and interactive exploration to support iterative prompt engineering for text-to-image creation. Usage scenarios, a user study, and expert interviews evaluated its effectiveness and usability.

  • System and Evaluation: PromptMagician co-embeds retrieved images and recommended keywords in 2D space to support interactive exploration and personalized evaluation.The recommendation model mines special and related keywords from DiffusionDB through semantic retrieval and hierarchical keyword extraction.
  • Findings: The paper reports that PromptMagician recommends useful keywords and facilitates interactive exploration for iterative prompt refinement.It also presents insights for designing and improving prompting methods and visual systems.
Loading 2307.09036v2…