The Stable Artist: Steering Semantics in Diffusion Latent Space

1. Introduction

• Small prompt changes can produce entirely different images, making fine-grained control difficult in iterative text-guided generation.Users often make repeated prompt and parameter changes to approach an envisioned outcome.Source: §1 · p. 2
• Existing approaches have trade-offs: masks can discard structural information and global composition, P2P offers coarse control and one operation at a time, and Composable Diffusion only controls initial composition.The cited methods also do not support some texture or style changes, or subtler changes, respectively.Source: §1 · p. 2
• The Stable Artist uses Semantic Guidance (SEGA) to steer image generation through latent-space operations, supporting subtle edits, composition and style changes, and optimization of the artistic conception.SEGA can also probe diffusion-model latent spaces to study learned concepts and perform arithmetic between concepts.Source: §1 · p. 2
• The Stable Artist supports simultaneous editing with multiple concepts, control over edit extent and concept strength, and no explicit or attention-based masks or fine-tuning.Source: §1 · p. 2

2. The Stable Artist

• The Stable Artist combines classifier-free text conditioning with a variable number of semantic directions that can be enforced or suppressed.Unlike Safe Latent Diffusion [13], which suppresses one inappropriate direction, it supports multiple directions.Source: §2 · p. 2

2.1. Guided Diffusion

• At inference, classifier-free guidance adjusts the unconditioned noise estimate toward the text-conditioned estimate, with a guidance scale controlling the adjustment.The diffusion model iteratively denoises a Gaussian variable to produce samples of a learned data distribution.Source: §2.1 · p. 3

2.2. Semantic Guidance

• For one edit, SEGA combines noise estimates conditioned on the prompt and edit concept to move the unconditioned estimate toward or away from that concept.The editing direction determines whether the concept-conditioned direction is enforced or suppressed; the threshold and editing scale control which differences contribute and their strength.Source: §2.2 · pp. 3–4
• SEGA adds warm-up and momentum controls, then combines per-concept guidance directions as a weighted sum with weights summing to one.A higher warm-up value produces less significant image adjustments, and can preserve composition while altering fine-grained details; momentum builds during warm-up and accelerates guidance along consistently guided dimensions.Source: §2.2 · pp. 4–5

2.3. Interactively Steering Semantics

• Semantic guidance can manipulate a prompt-generated image by subtracting one concept direction and adding another, as in changing a king toward a queen.The paper illustrates this as moving from a prompt-guided royal-and-male region toward the intersection of royal and female concepts.Source: §2.3 · p. 5

• Steering away from “male” and toward “female” changes a king into a queen while retaining a compositionally similar image.The authors say this suggests diffusion-model latent spaces may be disentangled to some extent, an assumption requiring further research.Source: §3 · p. 5
• Multiple edits can change local objects and global environment while minimally interfering, including combinations of local and global changes.Examples include adding “carbon emissions” to a car, yielding a seemingly older vehicle, and reducing “safety,” yielding a roofless convertible.Source: §3 · p. 5

• SEGA targets relevant latent dimensions for continuous concept-strength control without user-provided or attention-derived masks.The authors report competitive-quality results without masking the image.Source: §4 · p. 5
• Two local composition changes and a global style change can occur simultaneously while most image details remain unchanged.Examples include preserved clouds, a foreground bush, a castle, and its hill; changes to a car likewise leave the foreground and background largely unaffected.Source: §4 · p. 5
• Crowd and tree edits can be progressively adjusted while the untargeted scene remains fairly consistent.Figure 4 uses a shared initial noise latent and increases editing-prompt strength from left to right and top to bottom; mixed edits can keep tree levels stable while reducing the crowd.Source: §4 · pp. 5–6, Figure 4 · p. 6

• Semantic guidance can optimize an image’s overall look and quality from a concise prompt instead of relying on long, error-prone prompt engineering.The authors describe this as direct style optimization in latent space.Source: §5 · p. 6
• Style transfer reproduces named artists’ styles and abstract instructions while leaving the original composition largely unchanged.Figure 5 shows edits to images generated from the same initial noise latent with the prompt “a house at a lake”; arrows mark editing direction.Source: §5 · p. 6, Figure 5 · p. 7
• For five image categories, averaged prompt embeddings guide high-quality results that stay close to the original while reflecting the target artistic direction.The categories include portrait photography, animation, concept art, character design, and modern architecture.Source: §5 · pp. 6–7

6. Qualitative Comparison

• For adding elements, Stable Artist preserves more of the original castle and river; for removal, it is more faithful than Composable Diffusion, alongside Prompt-to-Prompt.The model tends to cover removed components with new details rather than generate what would be revealed beneath them.Source: §6 · p. 7
• On style transfer, Stable Artist and Prompt-to-Prompt fuse the starting image content with the target style, while Composable Diffusion loses the original composition.With stronger Van Gogh conditioning, Composable Diffusion shifts toward replicating the painting.Source: §6 · p. 7
• For combined editing and style transfer, Stable Artist and Prompt-to-Prompt produce satisfactory results, while Composable Diffusion may miss the input descriptions.Prompt-to-Prompt accepts only one editing prompt, so both tasks were placed in one prompt, producing artifacts such as multiple boats.Source: §6 · p. 7
• Stable Artist outperforms Composable Diffusion overall and achieves better or equally good targeted editing performance as Prompt-to-Prompt while natively supporting multiple concepts.SEGA also supports conditioning beyond natural language, including images and averaged textual embeddings.Source: §6 · p. 7

7. Broader Impact on Society

• Stable Diffusion is trained on web-crawled data containing inappropriate content [14], and current versions show signs of inappropriate degeneration [13].The authors note that SEGA could also steer generation toward inappropriate material.Source: §7 · p. 8
• SEGA could potentially support fairness by detecting and steering related concepts, but the authors advocate further research in this direction.Source: §7 · p. 9
• The authors argue that generative models remain tools for human artists because creating artwork still involves substantial iterative feedback and creative thinking.They say Stable Artist increases interaction capabilities in this process.Source: §7 · p. 9

8. Conclusions

• SEGA steers the diffusion process along several latent-space directions, providing fine-grained control for sophisticated image composition and editing.Source: §8 · p. 9
• Further work should investigate how concepts are represented and targeted in diffusion-model latent spaces and how to quantify them.The authors also suggest automatic concept detection could offer tools to mitigate bias and address privacy concerns about memorized real people.Source: §8 · p. 9