Source-linked AI summary
PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
John Joon Young Chung, Eytan Adar
TL;DR
Text prompts can be inadequate for describing some visual ideas, and end-to-end T2I models offer little opportunity for intervention during generation. PromptPaint applies paint-like interactions to mix prompts and steer generation across canvas areas and stages. Studies characterize different strategies and report trade-offs, with user-study findings also identifying design challenges.
Problem
End-to-end T2I models offer little opportunity to intervene, and natural-language prompts may not express ambiguous concepts or fine-grained attributes.
Source: §1 · p. 1Method
PromptPaint treats prompts like paint, combining semantic mixing and directional prompts with prompt interventions and spatial stencils.
Source: §1 · pp. 1–2Results
Prompt mixing and directional prompts effectively add attributes, while prompt intervention and prompt stencil tend to transform images while maintaining similarity to the original.
Source: §1 · p. 2Takeaways & Limitations
The studies characterize steering strategies, user interactions, design trade-offs, and challenges to inform future generative tools.
Source: §1 · p. 2The studies did not compare PromptPaint’s interactions with those of other existing tools.
Source: §8.6 · p. 15
Abstract
from arXiv · showhide
While diffusion-based text-to-image (T2I) models provide a simple and powerful way to generate images, guiding this generation remains a challenge. For concepts that are difficult to describe through language, users may struggle to create prompts. Moreover, many of these models are built as end-to-end systems, lacking support for iterative shaping of the image. In response, we introduce PromptPaint, which combines T2I generation with interactions that model how we use colored paints. PromptPaint allows users to go beyond language to mix prompts that express challenging concepts. Just as we iteratively tune colors through layered placements of paint on a physical canvas, PromptPaint similarly allows users to apply different prompts to different canvas areas and times of the generative process. Through a set of studies, we characterize different approaches for mixing prompts, design trade-offs, and socio-technical challenges for generative models. With PromptPaint we provide insight into future steerable generative tools.
1 INTRODUCTION
PromptPaint addresses the difficulty of describing some visual ideas in language and the limited opportunities to shape end-to-end generation. It applies paint-like prompt mixing and iterative steering; studies characterize the approaches and report trade-offs and socio-technical challenges.
1 INTRODUCTION
- End-to-end generation offers little opportunity to intervene, while language may not express ambiguous concepts or fine-grained attributes.The introduction gives the example of specifying an image attribute such as “flatness” at 60%.Source: §1 · p. 1
- PromptPaint treats prompts like flexible paint, letting users mix semantics and apply different prompts across canvas areas and generation stages.Its controls include prompt mixing, directional prompts, prompt stencils, and prompt interventions.Source: §1 · pp. 1–2, Figure 1 · p. 2
- Prompt mixing and directional prompts effectively add attributes, while prompt intervention and prompt stencil tend to transform images while maintaining similarity to the original.The paper also reports that iterative steering can help users generate images aligned with their intentions.Source: §1 · p. 2
- The user study identifies trade-offs and challenges including model complexity and randomness, possible misalignment with expectations, and ownership influenced by expertise and alignment.The introduction also points to insights for designing future generative tools.Source: §1 · p. 2
2 BACKGROUND AND RELATED WORK
The background frames PromptPaint against challenges in translating painting practices into digital generation, the end-to-end design of T2I models, and interaction approaches for steering AI.
2.1 Painting From Physical to Digital
- Painting involves gradual judgments and iteration, while T2I models make some visual ideas hard to describe and remove the physical act associated with personal causation.The authors propose combining painting interactions with generative approaches to balance direct manipulation and intelligent interfaces.Source: §2.1 · pp. 2–3
2.2 AI Image Generation
- Diffusion-based T2I models are flexible and produce high-quality output, but prior technical methods for shaping images have largely neglected the interaction model.Examples include seed images, prompt combinations, masking, prompt-based editing, visual structures, and automatic prompt refinement.Source: §2.2 · p. 3
2.3 Interaction with AI Generation
- AI steering ranges from categories and sliders to examples and prompts, but prompts can be difficult when concepts are ambiguous or hard to verbalize.Prior work explores prompt mixtures, while few approaches let users intervene during generation; this paper investigates both in-process interaction and modularized results.Source: §2.3 · p. 3
3 INTERACTING WITH GENERATIVE MODELS LIKE PAINT MEDIUM
PromptPaint adapts paint-medium interactions to two generative-model tasks: exploring semantics between discrete inputs and gradually shaping an artifact. Its design uses palette mixing and temporal or spatial generation modularization.
3 INTERACTING WITH GENERATIVE MODELS LIKE PAINT MEDIUM
- The paint analogy links exploration between discrete semantics to color mixing and gradual artifact generation to applying colors over time and canvas areas.Discrete prompts or examples map to paint colors that can be combined beyond their original choices.Source: §3 · pp. 3–4, Figure 2 · p. 4
3.1 Mixing Colors: Exploring Vector Spaces
- Prompt mixing interpolates between discrete semantics, while directional prompts add a semantic direction for fine-grained control.A palette can show interpolation and shifting together through position and color, whereas sliders may require separate controls.Source: §3.1 · p. 4, Figure 3 · p. 4
3.2 Colors onto Canvas: Gradual Generation
- Temporal modularization changes prompts during generation, while spatial modularization generates content for selected canvas areas.The paper describes earlier prompts as shaping overall form and later prompts as shaping details; in-painting and out-painting can support partial generation.Source: §3.2 · pp. 4–5, Figure 4 · p. 5
4 PROMPTPAINT: INTERFACE
PromptPaint combines a canvas-based text-to-image tool with paint-like interactions for exploring prompts and gradually shaping images. Users can steer generation through prompt mixing and directional prompts, then apply prompts across canvas areas and generation steps.
4 PROMPTPAINT: INTERFACE
- PromptPaint pairs prompt mixing and directional prompts for exploring prompt space with prompt intervention and stencils for gradually building images.The canvas also supports moving and rotating images, brushing, erasing, lassoing, and layer management.Source: §4 · p. 5, §4.1 · p. 5
4.2.2 Prompt Mixing.
- Users mix prompts by combining palette circles, selecting a point on the resulting gradient, and detaching a prompt by touching it with another prompt in the mix.The current version supports mixing up to three prompts.Source: §4.2.2 · pp. 5–6, Figure 5 · p. 5
4.2.3 Directional Prompt.
- Directional prompts add a chosen attribute by setting two prompt endpoints and adjusting a slider to control its intensity.For example, the slider can add a slight amount of the “matte” attribute.Source: §4.2.3 · p. 6, Figure 7 · p. 6
4.2.4 Prompt Stencil.
- With a prompt stencil, users brush the canvas area to generate and see intermediate results, repeating the process to fill other areas.Guide Scale and steps adjust guidance intensity and the number of intermediate steps.Source: §4.2.4 · p. 6, Figure 8 · p. 6
- When a stencil overlaps existing imagery, the overcoat value controls similarity to that content: higher values produce less-similar results.Overcoating can also change an image using prompts different from those used to generate it.Source: Figure 9 · p. 6, Figure 10 · p. 6, §4.2.4 · p. 7
4.2.5 Prompt Intervention.
- Prompt intervention lets users change the selected prompt or directional-prompt slider during generation; earlier prompts tend to shape overall form and color, while later prompts tend to shape details.The result may sometimes change little even after an early prompt intervention.Source: §4.2.5 · p. 7, Figure 11 · p. 7
- Prompt paths, stopping and restarting, rollback, and past-generation selection support reviewing and iterating on earlier generation states.Users can roll back through the progress bar or undo, and set the number of steps processed in a restart round.Source: §4.2.5 · p. 7
5 PROMPTPAINT: TECHNICAL DETAILS
PromptPaint manipulates text-prompt vectors for prompt mixing, directional prompts, and intervention, and latent representations for stencil-based generation. The technical description covers these operations and the system’s web-app implementation.
5.1 Background: Diffusion-based T2I
- Latent diffusion uses a text encoder, denoiser and scheduler, and decoder; an encoder is needed when an existing canvas image is used for diffusion.PromptPaint manipulates prompt vectors for mixing, directional prompts, and intervention, and latent representations for prompt stencil.Source: §5.1 · p. 7
5.2 Prompt Vector Manipulations
- Prompt mixing interpolates embedded prompt vectors using weights determined by the user’s palette selection.The resulting interpolated vector guides generation.Source: §5.2 · pp. 7–8
- A directional prompt forms a vector between two prompt embeddings, weights it with the slider value, and adds it to the mixed-prompt vector used to guide denoising.Users can change the final prompt vector during generation for prompt intervention.Source: §5.2 · p. 8
5.3 Latent Representation Manipulation
- For stencil areas containing image content, PromptPaint manipulates latent representations according to a diffusion-step threshold controlled by the overcoat ratio.Below the threshold, the existing image’s latent representation is noised to match the step; otherwise, the scheduler’s reconstructed latent is used.Source: §5.3 · p. 8
- Unstenciled areas are treated as white-background regions and replaced at each step by existing-image latent representations with added noise, an approach chosen instead of inpainting to simulate overcoating.The stated aim is to maintain visual similarity when adding generation to an already-filled area.Source: §5.3 · p. 8
5.4 Implementation
- PromptPaint is a web app built with HTML, CSS, JavaScript, and React, with a WebSocket-based Flask server for real-time intermediate results.The implementation uses Stable Diffusion, CLIP, and the DDIM scheduler.Source: §5.4 · p. 8
6 CHARACTERIZATION STUDY
The characterization study compared prompt-steering strategies for adding attributes to generated images, finding distinct trade-offs between attribute addition and preserving the original image. It evaluated five strategies across attribute types and weights using crowd judgments.
6.1 Conditions
- The study compared prompt mixing, directional prompts, prompt stencil, prompt intervention, and text prompt concatenation as attribute-addition strategies.Stencil was repurposed to overcoat visual elements, while concatenation added the attribute text to the original prompt.Source: §6.1 · p. 8
6.2 Dataset
- The dataset paired images before and after adding objects, styles, or specific visual attributes, varying prompts, seeds, conditions, and weights across 4,368 image pairs.Specific visual attributes included colors, textures, lines, and shapes; generations used 50 diffusion steps and guide scale 7.5.Source: §6.2 · pp. 8–9
6.3 Metrics
- Workers rated attribute addition, persistence of the original attribute, image similarity, and how the new attribute was added.The first three used seven-level Likert scales; addition-approach options varied by attribute type.Source: §6.3 · p. 9
6.4 Procedure
- Amazon Mechanical Turk workers judged randomly sampled image pairs, with attention checks and response-quality filters applied before analysis.Workers saw ten pairs each; eligibility and filtering criteria included acceptance rate, prior HIT count, and response patterns.Source: §6.4 · p. 9, Table 2 · p. 9
6.5 Result
- Increasing weights generally raised attribute addition while reducing retention and similarity; specific visual attributes showed more gradual changes in retention and similarity.Prompt intervention changed specific-visual-attribute images less often than other conditions, even at higher weights.Source: §6.5 · pp. 9–10
- Prompt intervention added the least new-attribute content while best preserving the original image; prompt stencil, directional prompting, and prompt mixing followed that pattern.For objects and styles, stencil and intervention had similar remain and similarity scores, while mixing and directional prompting formed another group.Source: §6.5 · p. 9, Figure 13 · p. 10
- For semantically related object pairs, directional prompting and intervention often produced mixed-object outcomes at weight 0.5, while intervention led at weight 0.75.The compared pairs included tree-river, woman-man, dog-cat, and love-hate.Source: Figure 14 · p. 10, §6.5 · p. 10
- Prompt concatenation could add an attribute while retaining the original, but similarity fell and object pairs more often appeared separately than blended.Source: §6.5 · p. 10
7 USER STUDY
An observational study examined how hobbyists and novices used PromptPaint to steer and explore text-to-image generation. Participants generally responded positively, while their experiences also surfaced usability trade-offs, AI unpredictability, and differences in ownership.
7 USER STUDY
- Eight hobbyists or novices in visual arts completed an in-person study with four function tasks, think-aloud sessions, surveys, and an interview.The functions were tested in a fixed order: mixing, directional prompting, intervention, then stencil.Source: §7 · p. 10, §7.1 · p. 10, §7.2 · p. 11
7.3.1 Survey results.
- Participants generally rated PromptPaint positively for enjoyment, exploration, expressiveness, and immersion, though one participant was neutral or negative and one rated immersion negatively.Source: §7.3.1 · p. 11, Figure 15 · p. 12
- Participants often felt they collaborated with PromptPaint and retained some ownership even while judging AI contribution as greater.Source: §7.3.1 · p. 11, Figure 16 · p. 12
- All four functions were perceived positively, with stencil tending to support control and exploration most and intervention least; the small sample limited significant comparisons.Source: §7.3.1 · p. 11, Figure 17 · p. 12
7.3.2 Qualitative Results.
- Mixing and directional prompts helped participants explore or add concepts, while stencil enabled localized edits and intervention produced interesting mixes but was difficult to use.Distant prompts could shift images abruptly; stencil outputs could be incomplete or mismatched, and intervention timing was hard to judge from noisy previews.Source: §7.3.2 · pp. 11–12
- Participants valued gradual work on one canvas but sometimes wanted multiple results because diffusion is stochastic; some also found extensive manual control burdensome.Participants suggested balancing automation with manual intervention.Source: §7.3.2 · pp. 12–13
- AI complexity and randomness could misalign outputs with intentions, especially when users had concrete expectations; some participants first explored simpler prompts to understand the model.Others used interpolation or semantic shifts to learn how the model interpreted prompts.Source: §7.3.2 · p. 13
- Ownership varied; participants described contributing high-level ideas while AI supplied implementation, and expertise and expectation alignment shaped ownership.One participant also raised concerns about legal ownership and copyright.Source: §7.3.2 · p. 13
8 DISCUSSION
Prompt-mixing strategies trade off preserving existing image attributes against adding new ones, while paint-like interactions suggest design directions for other generative media. The discussion also identifies usability, ownership, and evaluation boundaries for PromptPaint.
8.1 Paint-like Interactions for Generative Tools
- Paint-like semantic placement could generalize across modalities, but 3D requires selectable parts and sequential media require specifications that model sequence.For in-generation interventions, earlier prompts shape overall form and later prompts render details; music could progress from song structures to embellishments.Source: §8.1 · pp. 13–14
8.2 Characterizing Approaches to Mix Prompts
- Prompt intervention and stencil tend to preserve original attributes and similarity, while mixing and directional prompts more often add new attributes.All four approaches let users adjust how much of the new attribute to add, unlike prompt concatenation.Source: §8.2 · p. 14
8.3 Interaction for T2I models
- Users found prompt intervention interesting but struggled to identify when to change prompts, and PromptPaint lacks negative prompts and structural conditions.More interpretable intermediate images could help, though learning to read noisy images adds user load; newer overcoating and inpainting models could also change stencil interactions.Source: §8.3 · p. 14
8.4 Design Trade-Offs in Generative Tools
- Generative tools must balance curation of multiple candidates with iterative steering, and simple automation with more manual control.The authors suggest learning user preferences for curation and treating automated controls as a low-threshold option and manual controls as a high-ceiling option.Source: §8.4 · p. 14
8.5 Ownership of Generated Artifacts
- Ownership depends partly on users’ values around manual effort, while training-data copying raises copyright concerns and legal ownership may trade off with dataset scale.The authors suggest letting users choose where to use generative AI and note image transformation as a possible way to balance ownership and scale.Source: §8.5 · pp. 14–15
8.6 Limitations
- PromptPaint was not compared with other existing tools; the studies instead compared prompt-combination functions and qualitatively examined usage patterns.The authors propose future comparisons along iteration support and the ability to show multiple results.Source: §8.6 · p. 15
9 CONCLUSION
The paper introduces PromptPaint as a paint-color-inspired way to interact with generative models. User studies characterize its steering approaches and usage, informing design insights for future generative tools.
9 CONCLUSION
- PromptPaint treats prompts like paint colors, letting users mix semantics and apply them across canvas regions and stages of generation.It uses prompt mixing, directional prompts, prompt intervention, and prompt stencil.Source: §9 · p. 15
- The approach aims to make end-to-end generation more flexible and gradual through iterative steering.Source: §9 · p. 15
- User studies characterize the four steering approaches and identify how people use the interactions.Source: §9 · p. 15