Source-linked AI summary
Prompt-to-Prompt Image Editing with Cross Attention Control
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, Daniel Cohen-Or
TL;DR
Text-driven image editing is difficult because small prompt changes can destroy the original image, while mask-based alternatives are cumbersome and discard masked structure. Prompt-to-Prompt injects cross-attention maps during diffusion to preserve source layout while changing text, enabling diverse text-only edits with reported seamless editing across varied images. Its real-image use remains limited by imperfect inversion and low-resolution attention maps.
Problem
Text-to-image models lack simple, localized editing control because slight prompt changes can produce different images, while mask-based methods are cumbersome and ignore masked structure.
Method
Prompt-to-Prompt injects source-image cross-attention maps during diffusion, controlling token-to-pixel relations while editing the textual prompt.
Results
The method enables seamless text-only editing across diverse images, including localized and global manipulations while preserving source structure.
Takeaways & Limitations
Cross-attention maps provide an interpretable handle for navigating semantic edits while retaining spatial layout and geometry.
Takeaways & Limitations
Real-image editing is limited by visible inversion distortion, the need for a suitable prompt, and low-resolution attention maps that constrain precise localization.
Abstract
from arXiv · showhide
Recent large-scale text-driven synthesis models have attracted much attention thanks to their remarkable capabilities of generating highly diverse images that follow given text prompts. Such text-based synthesis methods are particularly appealing to humans who are used to verbally describe their intent. Therefore, it is only natural to extend the text-driven image synthesis to text-driven image editing. Editing is challenging for these generative models, since an innate property of an editing technique is to preserve most of the original image, while in the text-based models, even a small modification of the text prompt often leads to a completely different outcome. State-of-the-art methods mitigate this by requiring the users to provide a spatial mask to localize the edit, hence, ignoring the original structure and content within the masked region. In this paper, we pursue an intuitive prompt-to-prompt editing framework, where the edits are controlled by text only. To this end, we analyze a text-conditioned model in depth and observe that the cross-attention layers are the key to controlling the relation between the spatial layout of the image to each word in the prompt. With this observation, we present several applications which monitor the image synthesis by editing the textual prompt only. This includes localized editing by replacing a word, global editing by adding a specification, and even delicately controlling the extent to which a word is reflected in the image. We present our results over diverse images and prompts, demonstrating high-quality synthesis and fidelity to the edited prompts.
1 Introduction
Prompt-to-Prompt addresses the instability and cumbersome masking of text-driven image editing by using cross-attention to control edits through text alone while preserving image structure.
- Large-scale text-to-image models can produce a completely different image after even a slight prompt change, limiting direct text-based editing.
- Mask-based editing localizes changes but is cumbersome and discards structural information within the masked region.
- The method analyzes cross-attention maps that bind image pixels to prompt tokens and uses them to control generated-image semantics.
- Attention-map injection supports token replacement, global prompt edits, and reweighting a word’s semantic effect through a simple interface.
- Prompt-to-Prompt enables text-only editing without model training, fine-tuning, extra data, optimization, or pixel-space specifications across diverse images.
2 Related work
Prior text-driven editing methods either support limited global changes, require masks, or need per-image training. Prompt-to-Prompt instead uses spatial information from internal generative-model layers to edit local and global details through text alone.
- Attention injection preserves the source structure when prompts change, whereas using only the same random seed can produce an unrelated image structure.
- GAN- and diffusion-based methods improved text-driven image manipulation, but their capabilities vary across dataset diversity and editing scope.
- Text-only approaches were often limited to global editing, while localized methods could change textures but not complex structures and required training per input.
- Mask-guided methods restrict changes spatially but require user-provided masks, which can make editing less intuitive and discard masked structural content.
- Prompt-to-Prompt uses spatial information from internal generative-model layers to edit local or global details by modifying text prompts only.
3 Method
Prompt-to-Prompt edits images by manipulating cross-attention during diffusion while preserving source structure through shared attention maps and controlled injection. The method supports token replacement, object preservation, and tunable attention injection without additional training or optimization.
- 3 Method: Prompt-to-Prompt edits a source image using an edited text prompt while aiming to preserve its original structure and appearance.The source and target generations use controlled cross-attention manipulation during diffusion.
- 3.1 Cross-attention in text-conditioned Diffusion Models: Cross-attention maps bind image pixels to prompt tokens, and their spatial relations influence the generated image’s composition and geometry.The maps assign token-dependent weights to pixels and produce cross-attention features through a weighted combination of token values.
- 3.2 Controlling the Cross-attention: For word swaps, injecting source attention maps into the edited generation preserves composition while allowing the new token to change image content.The approach can replace “bicycle” with “car” while retaining the source image’s structure, but large structural changes may require softer constraints.
- 3.2 Controlling the Cross-attention: Attention injection can preserve an individual object while its context changes, as shown by retaining a butterfly’s structure and appearance.Only the butterfly token’s attention weights are injected in the object-preservation example.
- 3.2 Controlling the Cross-attention: The method runs source and edited prompts simultaneously with fixed internal randomness, applying an edit function to attention maps at each diffusion step.The algorithm takes source and target prompts plus a shared random seed, then returns both generated outputs.
- 3.2 Controlling the Cross-attention: Limiting attention injection to early diffusion steps balances fidelity to the source image against geometry freedom for adapting to the edited prompt.The timestamp parameter τ determines how long injection continues; early steps largely determine composition.
4 Applications
Prompt-to-Prompt supports text-only localized and global editing by controlling cross-attention while preserving image structure, and extends to fader control and real-image editing. Its real-image results depend on inversion quality, which can trade off reconstruction fidelity against editability.
- Text-Only Localized Editing: Replacing prompt words while injecting attention maps preserves spatial layout, geometry, semantics, and background, including for structural edits such as changing a bicycle to a car.Increasing the number of injection steps generally increases fidelity to the original image, although injecting throughout every step is not always optimal.
- Text-Only Localized Editing: Adding specifications such as “crushed” preserves the background while introducing additional details over the original image.
- Global editing: Global edits can preserve object locations and identities while adding snow, changing lighting, translating sketches into photorealistic images, or inducing artistic styles.
- Fader Control using Attention Re-weighting: Re-scaling a specified word’s cross-attention provides fader control over how strongly that word influences the generated image.This addresses cases where the desired magnitude of an attribute, such as snow on a mountain, is difficult to express textually.
- Real Image Editing: Real-image editing uses diffusion inversion to find an initial noise vector, but current DDIM-based inversion can produce unsatisfactory reconstructions.Reducing classifier-free guidance improves reconstruction but constrains significant manipulations, while attention-derived masks can restore unedited regions without user guidance.
5 Conclusions
The work uses cross-attention maps to connect prompt words with image spatial layouts, enabling textual control over local and global editing. It also identifies limitations in inversion, attention resolution, and moving existing objects.
- 5 Conclusions: Cross-attention maps provide interpretable spatial representations that tie prompt words to synthesized-image layouts.These maps support direct control of image attributes through prompt manipulation.
- 5 Conclusions: Textual prompt manipulation enables local and global image editing with incremental changes rather than regenerating an image from scratch.The approach is presented as a simple interface for navigating a semantic textual space.
- 5 Conclusions: The method remains limited by inversion distortion, the need for a suitable inversion prompt, low-resolution attention maps, and inability to spatially move existing objects.The authors propose higher-resolution cross-attention as one direction for more precise localized editing.
- 5 Conclusions: Attention maps can preserve unedited image regions without user-provided masks when inversion distortion is significant.The figure specifically notes retention of the cat’s identity after editing.
A.1 Diffusion Models
DDPMs define forward and reverse diffusion processes between data and noise, then learn a parameterized reverse transition for sampling. DDIMs can make sampling deterministic by setting σ_t to zero.
- A.1 Diffusion Models: DDPMs model a data distribution with a forward Markov chain that gradually adds noise from x_0 to x_T.The final latent approaches an isotropic Gaussian when the number of steps is sufficiently large.
- A.1 Diffusion Models: The forward process can express x_t directly as a linear combination of the original data and noise, without sampling intermediate latent vectors.The process is parameterized by a schedule β_0, . . . , β_T and α_t := ∏_{i=1}^t(1 − β_i).
- A.1 Diffusion Models: The reverse process approximates the intractable data-dependent transition with a parameterized Gaussian network p_θ(x_{t−1} | x_t).Its mean can be replaced by predicting the noise added to x_0.
- A.1 Diffusion Models: DDIM sampling becomes deterministic when σ_t = 0 at every step.The sampling noise scale σ_t can otherwise be controlled at each stage.
A.2 Cross-attention in Imagen
Imagen uses three text-conditioned diffusion models arranged from base generation through two super-resolution stages. Text conditioning enters the base model through cross-attention and hybrid-attention layers at specified resolutions.
- A.2 Cross-attention in Imagen: Imagen consists of a 64 × 64 text-to-image model followed by 64 × 64 → 256 × 256 and 256 × 256 → 1024 × 1024 super-resolution models.The three models predict noise ε_θ(z_t, c, t) using U-shaped networks, with z_t as the latent vector and c as the text embedding.
- A.2 Cross-attention in Imagen: The 64 × 64 Imagen model conditions on text through cross-attention at resolutions [16] [8] and hybrid-attention at resolutions [32] [16] [8].These layers occur within the U-Net’s downsampling and upsampling paths.
B Additional results
Additional examples cover Prompt-to-Prompt word swaps, specification additions, and attention re-weighting. The supplied examples include varied subjects and scenes illustrating these editing operations.
- B Additional results: The additional results include word swaps, such as changing “cat” to alternative animal tokens across a source prompt.The examples list cat-to-dog, chicken, squirrel, and elephant substitutions.
- B Additional results: The examples also demonstrate adding specifications to prompts describing a house, a flag, and a mountain.Figure 14 is identified as showing editing by adding a specification.
- B Additional results: Attention re-weighting is presented as a further editing operation, with an example involving a tiger sleeping in a field.Figure 15 is identified as showing attention re-weighting.