PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement

Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, Tianyi Zhang

arXiv:2403.04014v1cs.HCcs.AI

TL;DR

Novices face difficulty crafting prompts that align text-to-image models with their creative intentions and often lack style expertise. PromptCharm addresses this with automated prompt refinement, style exploration, attention-based explanations, attention adjustment, and inpainting. In user studies, participants created higher-quality images that better matched expectations than with two comparison variants.

  • Problem

    Novice users struggle to write prompts that models interpret effectively while preserving creative intentions and often lack expertise in image-style modifiers.

  • Method

    PromptCharm is a mixed-initiative system combining automated prompt refinement, style exploration, model-attention visualization, attention adjustment, and image inpainting.

  • Results

    PromptCharm participants created more aesthetically pleasing images that better matched expectations than users of two baselines; one study reported 5.9 versus 4.4 and 5.9 versus 4.8.

  • Takeaways & Limitations

    PromptCharm supports novice text-to-image creation through a rich feedback loop that connects prompt exploration, model explanations, and direct image refinement.

  • Takeaways & Limitations

    Promptist may be outperformed by alternative prompt-refinement methods, and inpainting sometimes fails to provide intended results when the masked area is large.

Abstract

from arXiv · show

The recent advancements in Generative AI have significantly advanced the field of text-to-image generation. The state-of-the-art text-to-image model, Stable Diffusion, is now capable of synthesizing high-quality images with a strong sense of aesthetics. Crafting text prompts that align with the model's interpretation and the user's intent thus becomes crucial. However, prompting remains challenging for novice users due to the complexity of the stable diffusion model and the non-trivial efforts required for iteratively editing and refining the text prompts. To address these challenges, we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-modal prompt engineering and refinement. To assist novice users in prompting, PromptCharm first automatically refines and optimizes the user's initial prompt. Furthermore, PromptCharm supports the user in exploring and selecting different image styles within a large database. To assist users in effectively refining their prompts and images, PromptCharm renders model explanations by visualizing the model's attention values. If the user notices any unsatisfactory areas in the generated images, they can further refine the images through model attention adjustment or image inpainting within the rich feedback loop of PromptCharm. To evaluate the effectiveness and usability of PromptCharm, we conducted a controlled user study with 12 participants and an exploratory user study with another 12 participants. These two studies show that participants using PromptCharm were able to create images with higher quality and better aligned with the user's expectations compared with using two variants of PromptCharm that lacked interaction or visualization support.

1 INTRODUCTION

PromptCharm addresses novice difficulty in crafting prompts that preserve creative intent and produce aesthetically pleasing images. It combines automated refinement, interactive support, and model explanations, and studies report better image outcomes than comparison variants.

  • Novices struggle to write prompts that models interpret effectively while preserving creative intentions and often lack expertise in image-style modifiers.
  • Existing interactive prompting approaches provide alternatives or keywords but often lack rich feedback about how prompts influence generated images.
  • PromptCharm combines automated prompt optimization, image-style exploration, attention visualization, attention adjustment, and image inpainting in a mixed-initiative refinement loop.
  • 24 participants took part in two within-subjects studies comparing PromptCharm with Baseline and Promptist variants.
  • 5.9 versus 4.4 and 5.9 versus 4.8 were the reported comparison scores for PromptCharm against Baseline and Promptist, respectively.Both comparisons were statistically significant, with p=0.02 and p=0.04, respectively.

2 RELATED WORK

Related work spans text-to-image generation, prompt engineering, interactive generative design, and human-AI collaboration. PromptCharm contributes a mixed-initiative system combining prompt refinement, modifier exploration, explanations, attention adjustment, and inpainting.

  • Stable Diffusion uses latent diffusion and cross-attention layers, and PromptCharm adopts it as an open-source text-to-image generation pipeline.
  • Prior work includes prompting guidelines and automated prompt generation methods that optimize input prompts or extend them with image-style keywords.
  • Interactive systems support prompt exploration, but PromptCharm additionally provides model explanations to help users interpret and improve generated images.
  • PromptCharm differs from image-focused tools by supporting iterative improvement through automated refinement, modifier exploration, attention adjustment, and inpainting.
  • Its design follows human-AI collaboration principles by explaining system behavior and enabling user feedback during interaction.

3 USER NEEDS AND DESIGN RATIONALE

The design rationale derives five user needs for interactive text-to-image prompting, including recommendation, control, exploration, version tracking, and explanations. PromptCharm addresses these needs through editable automation and feedback-driven refinement.

  • A literature review identifies five user needs: prompt revision, automation-control balance, option exploration, version control, and explanations for generated content.
  • Novices often need help starting and revising prompts because they struggle with prompt writing.
  • PromptCharm uses Promptist to revise prompts while allowing users to delete, explore, or replace image modifiers.
  • Attention visualizations show token importance and corresponding image regions, helping users identify misalignment with their intent.
  • Users can adjust keyword attention or mask and regenerate undesired regions without rewriting prompts.

4 DESIGN AND IMPLEMENTATION

PromptCharm implements a multi-stage workflow for prompt refinement, style exploration, attention-based explanation and correction, and iterative creation. Users can inspect model behavior and directly modify prompts, attention, or image regions.

  • PromptCharm includes text prompt refinement, modifier suggestion and exploration, attention-based explanation and refinement, inpainting, and version control.
  • Text Prompt Refinement, Suggestion, and Exploration: Promptist automatically rephrases users’ prompts while retaining their original intentions.
  • Text Prompt Refinement, Suggestion, and Exploration: Users can compare refined and initial prompts, generate images, replace modifiers with similar or dissimilar styles, and explore popular modifiers.
  • Model Attention-based Explanation and Refinement: DAAM aggregates cross-attention scores across layers and projects a selected prompt token’s scores onto the generated image.
  • Model Attention-based Explanation and Refinement: PromptCharm visualizes token saliency and highlights image regions associated with hovered prompt tokens.
  • Model Attention-based Explanation and Refinement: Users can adjust attention to keywords when model attention misaligns with the intended image content.

5 USAGE SCENARIO

PromptCharm’s usage scenario follows Alice as she iteratively improves a text-to-image creation through prompt refinement, attention adjustment, style exploration, and localized inpainting.

  • Initial creation: Alice begins with a prompt for a wolf, child, and full moon, but the generated image misplaces the child and lacks aesthetic quality.She finds online prompt examples cumbersome to experiment with and turns to PromptCharm.
  • Prompt refinement: PromptCharm generates a revised prompt with additional style modifiers that Alice can inspect before generating a new image.The example modifiers include “by greg rutkowski”, “thomas kinkade”, and “trending on artstation.”
  • Attention adjustment: When the child is missing and multiple wolves appear, Alice reduces the model attention to “wolf” by a factor of 0.5.After regeneration, the image includes both the human child and wolf.
  • Style exploration: Alice explores the “thomas kinkade” modifier through database examples, then replaces it with the dissimilar style “baarle ilya kuvshinov”.The style exploration helps her understand the visual effects associated with the modifier.
  • Localized refinement: Alice uses inpainting to regenerate an unwanted image area without modifying the broader prompt.Comparing the latest iteration with the initial creation, she observes improved image quality and visual effects.

6 USER STUDY 1: CLOSE-ENDED TASKS

Study 1 compared PromptCharm with a plain baseline and Promptist in close-ended image-creation tasks. PromptCharm produced the strongest image similarity and was generally preferred, while attention adjustment and style exploration were especially useful features.

  • Study design: PromptCharm was compared with Baseline and Promptist in a within-subjects study of 12 participants completing three close-ended image-creation tasks.Baseline provided text editing and version history; Promptist additionally automated prompt refinement.
  • User performance: PromptCharm achieved the best performance on all three tasks, with average SSIM of 0.648 versus 0.479 for Baseline and 0.574 for Promptist.SSIM measures structural similarity between each generated image and its target image; higher SSIM indicates better performance.
  • User performance: PromptCharm performed significantly better than the baselines on the two tasks containing multiple objects, whereas Promptist was not significantly different from PromptCharm on the simpler single-subject task.The authors attribute the stronger results on challenging tasks to multi-modal prompting and rich feedback.
  • User performance: Participants explored an average of 4.8 modifiers and made an average of 4.3 attention adjustments, with all participants rating attention adjustment as helpful.Style exploration helped participants assess modifiers before generation, while attention adjustment supported image refinement.
  • User ratings: Nine of 12 participants wanted to use PromptCharm in the future, and its median future-use rating was 6 on a 7-point Likert scale.Attention adjustment received the highest feature rating, with a median of 6.5; image inpainting received a neutral median rating of 5.
  • Cognitive overhead: PromptCharm did not produce statistically greater mental demand than either baseline, while participants reported better performance and less frustration than with Baseline.Differences in reported performance and frustration were not significant between PromptCharm and Promptist in the cited comparisons.

7 USER STUDY 2: OPEN-ENDED TASKS

Study 2 evaluated PromptCharm in open-ended image-creation tasks with 12 participants, finding stronger perceived image quality, expectation alignment, feature usefulness, and preference than the baselines without added workload.

  • Study design: 12 participants completed open-ended tasks using PromptCharm and the same Baseline and Promptist comparisons as Study 1.Participants created images from provided subjects based on their own creative ideas.
  • User performance: PromptCharm received median ratings of 6 for aesthetic appeal and expectation alignment, compared with Baseline medians of 5 and 4.5 and Promptist medians of 5 and 5.These assessments were self-reported because the open-ended tasks lacked an objective performance measure.
  • User behavior: Participants explored 7.3 modifiers on average, and 8 of 12 explicitly said PromptCharm helped them discover diverse image styles.Participants also used attention adjustment to address missed subjects in longer, more complex prompts.
  • User preference and feedback: 11 of 12 participants wanted to use PromptCharm for future image creation, with median willingness rated 6 on a 7-point scale.All participants rated modifier exploration and version control as helpful, while 11 of 12 rated attention control as helpful.
  • Cognitive load: PromptCharm did not significantly differ from either baseline in mental demand, hurry, or frustration, suggesting its additional interaction did not add measured workload in open-ended tasks.The reported p-values were 0.21 and 0.42 for mental demand, 0.25 and 0.55 for hurry, and 0.18 and 0.19 for frustration.
  • User preference and feedback: 11 of 12 participants ranked PromptCharm as most helpful and preferred it over both baselines, citing greater control and a more efficient feedback loop.Participants valued being able to adjust attention, erase unwanted elements, and explore modifiers while refining images.
  • User feedback: Two participants requested textual explanations for image modifiers as a potential improvement.This feedback identifies a remaining usability need in image-style exploration.

8 DISCUSSION

The discussion frames PromptCharm as a mixed-initiative approach that enriches feedback and helps novices overcome prompt-selection barriers. It also identifies boundaries involving study freedom, model coverage, exploration scale, and algorithmic choices.

  • Enriching the feedback loop: Attention visualization helps users identify missed or misattended prompt keywords and refine prompts or images more directly.The discussion links this support to users’ reported frustration when model explanations were absent.
  • Enriching the feedback loop: PromptCharm combines image-style exploration, attention visualization, attention adjustment, and image inpainting to support iterative human-AI refinement.The design aims to let users understand how feedback is incorporated and choose flexible ways to provide further feedback.
  • Addressing conceptual gaps: PromptCharm addresses a selection barrier: novices may know the desired image style but not which modifiers to use.Browsing style keywords and related images helps users contextualize modifiers and understand model-refined prompts.
  • Addressing conceptual gaps: In close-ended tasks, style exploration supported better replication of a particular target image and style.Participants compared generated images with targets, examined attention, and replaced modifiers that produced mismatched styles.
  • User expertise: Participants with one year of text-to-image experience performed similarly to less-experienced participants in Study 1, with average SSIM values of 0.642 and 0.649.The comparison supports the reported applicability of PromptCharm to users with limited experience.
  • Limitations and future work: The open-ended study’s three manually designed scenes may have limited participants’ creative freedom, and PromptCharm was evaluated only with novice users.The authors propose free-form usage and evaluation with experts as future work.
  • Limitations and future work: PromptCharm was evaluated only with Stable Diffusion; reuse with closed-source models would require alternative attention-adjustment and explanation designs.The authors also identify evaluation with other text-to-image models as future work.
  • Limitations and future work: The current system supports examining one generated image per version, limiting large-batch exploration for experienced users.Future designs could organize batches such as 30 images per prompt using layouts based on color or semantic similarity.

9 CONCLUSION

PromptCharm assists novice users with text-to-image generation through multi-modal prompting and refinement. User studies found that it produced images more closely resembling targets, more aesthetically pleasing, and better matched to expectations than two baselines.

  • PromptCharm is a mixed-initiative system for multi-modal prompting and refinement in text-to-image generation.It assists users in refining prompts and AI-generated images.
  • PromptCharm provides automated prompt refinement, style and keyword exploration, attention visualization, attention adjustment, inpainting, and version control.These features support iterative refinement of both prompts and generated images.
  • In a 12-participant close-ended study, PromptCharm users created images that closely resembled target images compared with two baseline methods.The comparison used two baseline methods in the user study.
  • In a separate 12-participant open-ended study, users found PromptCharm images more aesthetically pleasing and better matched to their expectations than baseline images.The paper reports this as an outcome of the open-ended study.

A.1 Task Description

The close-ended user study used predefined image-creation tasks, with examples of participant-created images documented in Figure A1.

  • The close-ended user study included predefined tasks for participants to complete.The task description is presented in Table A1.
  • Figure A1 presents examples of images created by users in Study 1.

B.1 Task Description

The open-ended study involved image-creation tasks and examples illustrating attention adjustment for selected prompt keywords.

  • The open-ended user study included image-creation tasks described in Table B2.
  • Figure B2 presents examples of images created by users in Study 2.
  • The attention-adjustment examples use prompt keywords such as taxi, snowy, rain, and giraffe.The examples include city-street, library, desert, and animal-scene prompts.
Loading 2403.04014v1…