PaperlayerView exact revision on arXiv

Source-linked AI summary

RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions

Yunlong Wang, Shuyuan Shen, Brian Y. Lim

arXiv:2302.09466v3cs.HCcs.AI

TL;DR

Text-to-image models may not precisely express the contexts and emotions conveyed by natural-language prompts. RePrompt uses interview-informed, explainable prompt editing based on intuitive text features, and studies found improved emotional expression, especially for negative emotions. The method’s feature design remains limited by word-level modeling and omission of image properties.

  • Problem

    It remains unclear how accurately text-to-image models capture user intent and emotional expression from natural-language prompts.

  • Method

    RePrompt uses interview-informed word-level text features, a proxy model, and model explanations to create an automatic prompt-editing rubric.

  • Results

    RePrompt improved emotional expression in simulation and user studies, especially for negative emotions, while results for image-text alignment were mixed.

  • Takeaways & Limitations

    Explainable prompt engineering can tune human requests to help generative models produce more precise emotional output.

  • Takeaways & Limitations

    The approach uses simplistic word-level features, can break meaningful phrases, and does not model image properties such as styles or color themes.

Abstract

from arXiv · show

Generative AI models have shown impressive ability to produce images with text prompts, which could benefit creativity in visual art creation and self-expression. However, it is unclear how precisely the generated images express contexts and emotions from the input texts. We explored the emotional expressiveness of AI-generated images and developed RePrompt, an automatic method to refine text prompts toward precise expression of the generated images. Inspired by crowdsourced editing strategies, we curated intuitive text features, such as the number and concreteness of nouns, and trained a proxy model to analyze the feature effects on the AI-generated image. With model explanations of the proxy model, we curated a rubric to adjust text prompts to optimize image generation for precise emotion expression. We conducted simulation and user studies, which showed that RePrompt significantly improves the emotional expressiveness of AI-generated images, especially for negative emotions.

1 INTRODUCTION

Text-to-image models let laypeople create visual art from natural-language prompts, but generated images may not precisely express intended emotions. RePrompt addresses this gap by automatically producing human-readable prompt revisions informed by user strategies and explainable modeling.

  • Natural-language image generation expands visual-art creation for laypeople, but emotional expression may not match the input context.
  • Users commonly rely on inefficient trial-and-error prompt editing because prompts are not always optimized for the generative model.
  • RePrompt automatically edits prompts to make AI-generated images more precisely express the original text’s meaning and emotion.
  • The authors report an interview study of 19 laypersons examining how people understand text-to-image models and revise prompts for emotional expression.
  • RePrompt combines intuitive text features, a proxy model, and model explanations to create an automatic prompt-editing rubric.
  • Simulation and user studies with 197 participants found significantly improved emotional expression, especially for negative emotions.

2 BACKGROUND AND RELATED WORK

Prior work established text-to-image generation, prompt engineering, and emotion-related image evaluation, but precise emotional expression from complex natural-language prompts remains insufficiently understood. RePrompt builds on interpretable prompt engineering to address this problem.

  • 2.1 Text-to-Image Generative AI and Vision-Language Model: Natural-language ambiguity and imperfect generative models make it important to assess how accurately prompts capture user intent and how prompts can be optimized.
  • 2.1 Text-to-Image Generative AI and Vision-Language Model: Text-to-image systems use vision-language models such as CLIP to connect text and images through shared semantic representations.
  • 2.1 Text-to-Image Generative AI and Vision-Language Model: Earlier emotion-generation work used constrained short prompts and small studies, leaving complex and natural emotional expressions unclear.
  • 2.2 Prompt Engineering for Generative AI: Prompt engineering adjusts prompts to improve model outputs and can substitute for fine-tuning when model parameters are inaccessible.
  • 2.2 Prompt Engineering for Generative AI: Many automated prompt methods are not human-readable, whereas interpretable systems emphasize transparency, controllability, and understanding.
  • 2.2 Prompt Engineering for Generative AI: RePrompt extends this interpretable direction to text-to-image generation, where research remains nascent and prior phrasing changes showed limited effects.

3 RESEARCH QUESTIONS AND METHODOLOGY

The paper asks how people perceive emotional image generation, how prompts can be automatically refined, and how effective RePrompt is. It combines interviews, an explainable feature-based pipeline, and simulation and user-study evaluations.

  • The study examines emotional expressiveness, automatic prompt refinement, and RePrompt’s effectiveness through three research questions.
  • Interviews used thematic analysis to study how laypersons developed prompt-editing strategies through trial and error with DALL-E 2.
  • RePrompt uses word-level features and machine-learning models to identify prompt properties associated with better generated images, then revises prompts automatically.
  • The method evaluates generated images using computational metrics and an evaluation user study against other text-prompting methods.
  • 3.3 Text-to-Image Models: The proxy-model training setup generated 10,000 images with VQGAN-CLIP, while DALL-E 2 was used as a comparative generative model.
  • 3.4 Emotional Text Dataset: The emotional dataset contains situation texts and emotion labels from EmpatheticDialogues, selected for natural emotion expressions grounded in real experiences.

4 INTERVIEW STUDY

The interview study investigated how laypeople judged emotional images and revised prompts while interacting with DALL-E 2. Participants initially used trial and error, then developed strategies favoring concise, concrete, objective descriptions.

  • 4.4.1 Participants’ Understandings of the Text-to-Image Model.: All participants began with trial-and-error editing but adapted their strategies as they learned more about DALL-E 2.
  • 4.4.1 Participants’ Understandings of the Text-to-Image Model.: Participants found that DALL-E 2 handled simple, concrete text better, often ignored context with many subjects, and sometimes produced gibberish.
  • 4.4.1 Participants’ Understandings of the Text-to-Image Model.: Participants reported failures with specific concepts, including movie and commodity names, which the model did not portray correctly.
  • 4.4.1 Participants’ Understandings of the Text-to-Image Model.: Objective descriptions of the expected image, including direct descriptions of desired objects and actions, often worked better for participants.
  • 4.4.1 Participants’ Understandings of the Text-to-Image Model.: Participants found failed edits difficult to interpret because the AI was not consistently accurate or clear about why outputs diverged from prompts.
  • 4.4.2 Participants’ Editing Strategies.: Thematic analysis connected trial-and-error failures to poor user experience and motivated support for more systematic prompt refinement.

5 TECHNICAL METHOD OF REPROMPT

RePrompt automatically refines emotional text prompts by modeling how interpretable word-level features affect CLIP-based image alignment, then using explanations to tune those features. Its rubric targets actionable feature ranges, including adjective count and concreteness, while accounting for CLIP’s semantic-alignment limitations.

  • 5.1 Feature Curation: RePrompt uses interpretable POS-based counts and concreteness features to connect prompt structure with generated-image quality.The 20 features are easy to adjust by adding, deleting, or replacing words, and were informed by interview findings and prior work.
  • 5.2 CLIP Score for Image Quality Assessment: CLIP Score measures image-emotion and image-text alignment through cosine similarity rather than aesthetic quality.The method treats semantic closeness between image and text embeddings as the relevant image-quality measure.
  • 5.2 CLIP Score for Image Quality Assessment: CLIP can produce meaningless scores for junk text, although the study’s checked dataset contained no such input.The authors note that spell checking can address this issue before applying CLIP-based evaluation.
  • 5.3 Proxy Model and Feature Analysis: RePrompt trains a proxy classifier on 10,000 emotional texts and VQGAN-CLIP images, with LightGBM achieving AUC = 0.60 for IEA and AUC = 0.73 for ITA.Alignment scores are binarized around their mean, and competing models are compared using 5-fold cross-validation.
  • 5.4 Feature Analysis by Model Explanations: The pipeline applies SHAP to select salient features and partial dependence plots to identify value ranges associated with high predicted image-emotion alignment.These explanations support automated rubric construction while avoiding repeated calls to the generative model during feature analysis.
  • 5.4 Feature Analysis by Model Explanations: For adjectives, the rubric recommends more than 1 adjective and mean adjective concreteness above 2.0, implemented by adding relevant concrete adjectives.The adjective findings aligned with interview results and supplied quantitative, actionable guidance for prompt refinement.

6 EVALUATION

RePrompt was evaluated against original, manual, and label-appended prompts using simulation and user studies measuring image-emotion and image-text alignment. It improved image-emotion alignment overall, with the clearest user-rated advantage for negative emotions, while image-text findings differed between computational and human evaluations.

  • 6.3 Simulation Study: RePrompt produced significantly higher image-emotion alignment than the original, manual, and label-appended prompts in simulation.Manual editing also improved alignment over the original, but scored below label appending.
  • 6.3 Simulation Study: Simulation showed lower image-text alignment for RePrompt and manual editing than the original, whereas label appending increased it.The lower scores suggest that edited prompts changed the original text more substantially.
  • 6.4.3 Results of Ratings.: User ratings likewise found higher image-emotion alignment for all editing methods than the original, with RePrompt significantly outperforming the alternatives.The practical differences between RePrompt and other conditions were small.
  • 6.4.3 Results of Ratings.: Human ratings showed higher image-text alignment for all editing conditions than the original, contradicting the simulation result.The authors attribute the discrepancy to preserved key meanings, rating-order effects, and the strong correlation between image-emotion and image-text ratings.
  • 6.4.3 Results of Ratings.: RePrompt’s user-rated advantage in image-emotion alignment occurred only for negative emotions, while positive-emotion differences were not perceived.The authors relate this pattern to human sensitivity to negative stimuli and weaker modeling of positive emotions by CLIP.
  • 6.4.3 Results of Ratings.: Ranking measures found no significant condition differences for either image-emotion or image-text alignment, despite weak correlations with rating scores.The authors note that ranking questions may have influenced participants’ rankings.

7 SUMMARY OF RESULTS

The paper identifies layperson strategies for improving emotional image generation and operationalizes them in RePrompt, an explainable automatic prompt-refinement method. Simulation and user studies indicate improved image-emotion alignment, especially for negative emotions, while image-text alignment results remain mixed.

  • 7 SUMMARY OF RESULTS: Laypersons improved prompts by emphasizing emotion, simplifying text, increasing concreteness, and shifting from first-person expression to third-person image descriptions.Participants also found that the AI handled simple, concrete, and objective descriptions better than complex or abstract text.
  • 7 SUMMARY OF RESULTS: RePrompt converts these editing strategies into intuitive text features, predictive models, and an explanation-based rubric for automatic prompt refinement.The rubric is designed to improve the emotional expression of generated images.
  • 7 SUMMARY OF RESULTS: RePrompt improved image-emotion alignment in simulation and user studies, especially for negative emotions, while positive-emotion differences were not perceived and image-text results were mixed.The authors suggest that human sensitivity and CLIP’s weaker positive-emotion modeling may contribute to the emotion-type pattern.

8 DISCUSSION

The discussion frames RePrompt as an explainable prompt-engineering approach whose rubric can transfer across models and potentially expressions, while identifying important measurement and modeling limitations. It also highlights emotion-valence sensitivity and possible applications in emotional expression.

  • 8.1 Design Implications and Limitations: CLIP-based generation shows emotion-valence sensitivity: image-emotion alignment tracks human perception for negative but not positive emotions.The authors suggest that nuanced positive emotions may be poorly modeled and that this matters for emotion-regulation applications.
  • 8.2 Generalization of RePrompt: Knowledge derived with VQGAN-CLIP transferred to DALL·E 2, although the study assumes this transferability is related to their shared use of CLIP.The authors also note that the rubric’s generalization was tested using the same EmpatheticDialogues dataset and another Twitter dataset.
  • 8.1 Design Implications and Limitations: RePrompt’s proxy-model explanations support human-readable prompt revisions that can improve human–AI communication by reducing information loss.The paper presents model explanation as a new application of explainable AI to prompt engineering.
  • 8.1 Design Implications and Limitations: RePrompt’s feature-level modeling is limited because word-based features can split meaningful phrases and omit image properties such as style and color.The authors describe more sophisticated linguistic features and image-aware modeling as future directions.
  • 8.2 Generalization of RePrompt: The RePrompt pipeline could generalize through adapted feature curation, proxy models, and explanation methods across other expressions and generative models.The paper names partial dependence plots, Anchors, and scalable Bayesian rule lists as possible explanation choices.
  • 8.3 Benefits, Potential Applications, and Future Work: Human-readable edits may help users understand models and reduce trial-and-error while supporting emotional expression through image creation.The paper presents mental-health applications as a future possibility rather than an evaluated use case.

9 CONCLUSION

The conclusion presents RePrompt as an automatic, explainable method for refining prompts to improve semantic and emotional precision in text-to-image generation. Simulation and user studies suggest that its edits improve image–emotion alignment, especially for negative emotions.

  • 9 CONCLUSION: RePrompt combines interview-informed prompt editing with a rubric for explainable, automatic improvement of emotional image generation.The rubric was inspired by participants’ strategic prompt-editing behavior.
  • 9 CONCLUSION: RePrompt’s edited prompts produced better image–emotion alignment, especially for negative emotions, in simulation and user studies.The evaluation combined objective measures with subjective ratings.

A.1 A.1 Examples of CLIP Scores

The appendix cautions that CLIP Scores can be meaningless for junk inputs and provides example IEA and ITA scores for DALL·E 2-generated images.

  • A.1 A.1 Examples of CLIP Scores: The study avoided junk-input contamination because spelling checks found no such inputs in its text dataset.The authors identify spelling checks as a way to address the junk-character problem.
  • A.1 A.1 Examples of CLIP Scores: Table 3 lists example CLIP Scores for image-emotion alignment and image-text alignment in DALL·E 2 outputs.The examples illustrate both IEA and ITA measurements rather than establishing a general performance ranking.

A.2 Model Explanations

The appendix illustrates feature-importance analysis and rubric curation for noun and verb features. These visuals document how model explanations inform prompt-editing rules.

  • A.2 Model Explanations: Figure 10 presents feature importance according to SHAP values in the image-emotion alignment model.The figure identifies the model features used to analyze image-emotion alignment.
  • A.2 Model Explanations: Figures 11 and 12 document rubric curation for nouns and verbs, respectively.The supplied captions identify the feature categories but do not state the resulting comparisons or rules.

A.3 Survey Screenshots in the Image Evaluation User Study

The survey screenshots document screening, language, emotion, and alignment questions used in the image evaluation user study. The materials also include statistical-analysis tables and an external emotional-dataset evaluation.

  • The survey covers screening, language, and emotion questions across the study materials.
  • Image evaluation materials assess text-emotion alignment, image-emotion alignment, and image-text alignment through ratings, rankings, and explanations.
  • The appendix reports statistical analyses for simulation and user evaluation studies, including fixed effects, random effects, significance levels, and model fit.
  • A correlation table examines CLIP scores against emotion-rating probability distributions from the Emotion6 dataset.
  • An external Twitter-derived dataset evaluation uses six basic emotions and 200 randomly selected texts per emotion for a balanced test set.
Loading 2302.09466v3…