Source-linked AI summary
Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
Narek Tumanyan, Michal Geyer, Shai Bagon, Tali Dekel
TL;DR
Text-to-image diffusion models offer expressive synthesis but limited control over generated structure and semantic layout. The paper injects guidance-image spatial features and self-attention into a fixed, pretrained diffusion model to perform text-guided image-to-image translation without training or fine-tuning. Across diverse translation settings, it reports high-quality results and a significantly better balance between preserving guidance layout and deviating from guidance appearance than existing baselines.
Problem
Text-to-image models provide limited user control over generated structure and semantic layout, which matters for real-world content creation tasks.
Method
The method directly injects spatial features and self-attention extracted from a guidance image into target generation using a pretrained fixed diffusion model, without training or fine-tuning.
Results
The method produces high-quality translations across real and generated guidance images and achieves a significantly better balance between preserving guidance layout and deviating from its appearance.
Takeaways & Limitations
Manipulating internal diffusion features supports diverse text-guided image-to-image translation while preserving guidance shape and complying with target prompts.
Takeaways & Limitations
Performance relies on semantic association in diffusion features, works poorly for arbitrarily colored detailed label-segmentation masks, and can leak appearance information for textureless minimal guidance images.
Abstract
from arXiv · showhide
Large-scale text-to-image generative models have been a revolutionary breakthrough in the evolution of generative AI, allowing us to synthesize diverse images that convey highly complex visual concepts. However, a pivotal challenge in leveraging such models for real-world content creation tasks is providing users with control over the generated content. In this paper, we present a new framework that takes text-to-image synthesis to the realm of image-to-image translation -- given a guidance image and a target text prompt, our method harnesses the power of a pre-trained text-to-image diffusion model to generate a new image that complies with the target text, while preserving the semantic layout of the source image. Specifically, we observe and empirically demonstrate that fine-grained control over the generated structure can be achieved by manipulating spatial features and their self-attention inside the model. This results in a simple and effective approach, where features extracted from the guidance image are directly injected into the generation process of the target image, requiring no training or fine-tuning and applicable for both real or generated guidance images. We demonstrate high-quality results on versatile text-guided image translation tasks, including translating sketches, rough drawings and animations into realistic images, changing of the class and appearance of objects in a given image, and modifications of global qualities such as lighting and color.
1. Introduction
The paper addresses limited user control over text-to-image generation by targeting structural and semantic-layout control for versatile image-to-image translation. It introduces feature injection inside a fixed diffusion model to preserve guidance layout while following target prompts.
- Text-to-image foundation models provide expressive synthesis but largely restrict user control to the input text.
- Existing approaches that add guiding signals can require extensive compute, large-scale training tuples, and specific test-time input types.
- The framework extracts spatial features and self-attention from a guidance image and directly injects them into target-image generation.
- The method performs text-guided image-to-image translation without training or fine-tuning and supports both generated and real guidance images.
- The method achieves a significantly better balance between preserving guidance layout and deviating from guidance appearance than existing state-of-the-art baselines.
2. Related Work
Related work spans conventional image-to-image translation, text-guided image manipulation, and methods for controlling pretrained diffusion models. The paper distinguishes its feature-space approach through arbitrary target prompts, fine-grained layout control, and improved balance between layout preservation and text compliance.
- Image-to-image translation: Image-to-image translation maps images between domains while preserving domain-invariant characteristics such as object structure or scene layout.
- Text-guided image manipulation: Text-guided manipulation methods combine CLIP or pretrained generators to perform edits, with some approaches fine-tuning diffusion models.
- Diffusion-feature control: The framework uses a guidance image and text prompt, inverts the image to noise, and injects decoder spatial features and self-attention during generation.
- Diffusion-feature control: Unlike P2P, the method supports arbitrary target prompts rather than requiring word-to-word alignment between source and target prompts.
- Comparison with SDEdit: SDEdit exhibits a tradeoff between preserving guidance layout and fulfilling target text, whereas this method provides a better balance.
3. Preliminary
The method builds on text-conditioned latent diffusion, where a U-Net progressively denoises latent noise using residual, self-attention, and cross-attention blocks. Feature analysis indicates that intermediate decoder representations encode shared semantic regions, while deeper features contain higher-frequency information.
- Diffusion models: Diffusion models generate images by progressively removing noise from an initial Gaussian noise image through a learned backward process.
- Conditional diffusion: The denoising network can condition generation on signals including images, class labels, or text.
- Latent diffusion architecture: Stable Diffusion applies diffusion in the latent space of a pretrained autoencoder and uses a prompt-conditioned U-Net.
- Latent diffusion architecture: U-Net layers combine residual, self-attention, and cross-attention blocks; self-attention enables long-range interactions between image features.
- Diffusion-feature analysis: Intermediate decoder features encode localized semantic regions shared across objects despite domain and appearance variation.
- Diffusion-feature analysis: Deeper decoder features capture more high-frequency information that eventually forms the model’s predicted output noise.
4. Method
The method controls text-guided image translation by injecting guidance-image diffusion features and self-attention into a fixed text-to-image model. Intermediate decoder features preserve localized semantics, while self-attention helps retain layout and shape without excessive appearance leakage.
- Feature and attention injection: The framework extracts guidance-image features and self-attention during diffusion, then injects them into target-image generation without training or fine-tuning.It supports both real-world and text-generated guidance images, using DDIM inversion for real guidance images.
- Final configuration: The final configuration combines layer-4 feature injection with self-attention injection at higher-resolution layers to mitigate appearance leakage while preserving structure.Using self-attention alone restricts affinities without semantic association, producing misaligned structure; the combined configuration addresses this failure.
- Feature analysis: Intermediate decoder features encode localized semantic information shared across objects and appearance variations, motivating their use for structural guidance.Layer 4 features consistently represent semantic parts such as legs, torso, and head across generation time-steps.
- Feature and attention injection: Injecting features only at layer 4 weakly preserves structure, whereas deeper-layer injection improves structure but leaks guidance-image appearance.The leakage includes source colors such as red shirts and blue jeans, so deep spatial features are not modified in the final balance.
- Self-attention injection: Self-attention affinities align with semantic layout and progressively capture higher-frequency information, making them useful for retaining fine layout and shape details.Replacing self-attention matrices pulls features together according to the guidance-image affinities.
- Negative prompting: Negative prompting is especially beneficial for textureless primitive guidance images, while it has a minor role for natural-looking guidance images.The method uses classifier-free guidance to increase fidelity to the target prompt while allowing deviation from the unconditional prediction.
5. Results
The method is evaluated on diverse real and generated guidance images using new benchmarks and comparisons against multiple baselines. It achieves strong target-text fidelity while preserving guidance structure, outperforming baselines on this balance.
- Datasets: The evaluation uses Wild-TI2I with 148 pairs and ImageNet-R-TI2I with 150 image-text pairs, including real and generated guidance images.Wild-TI2I contains 53% real guidance images gathered from the Web; ImageNet-R-TI2I comprises varied renditions of ImageNet object classes.
- Qualitative results: Across diverse inputs, the method preserves guidance shape while complying with different target prompts, including artistic and textureless images.The reported samples include naturally looking, artistic, and textureless guidance images.
- Comparison to baselines: The method avoids SDEdit’s structure–appearance tradeoff, maintaining layout while achieving target-text fidelity across real and generated guidance images.SDEdit preserves structure at low noise but changes appearance little, whereas higher noise damages structure; DiffuseIT similarly makes little appearance change.
- Quantitative evaluation: The method achieves high structure preservation and target-text fidelity, matching low-noise and high-noise SDEdit respectively on the complementary evaluation metrics.CLIP cosine similarity measures text fidelity, while DINO-ViT self-similarity distance measures structure preservation.
- Comparison to P2P: Compared with P2P, the method preserves fine-grained layout while translating multiple traits, whereas P2P often deviates from structure during multiple prompt edits.The method obtains significantly lower self-similarity distance than P2P with cross-attention injected at all timesteps.
- Ablations: Ablations show that both spatial-feature injection and self-attention injection are critical for structure preservation.The features provide semantic association between original and translated content, while self-attention is essential for maintaining structure.
6. Discussion and Conclusion
The framework uses internal diffusion representations to translate diverse text-guided image-to-image inputs without training or fine-tuning. Its main limitation is reliance on semantic association in feature space, with additional appearance leakage possible for some textureless images.
- Conclusion: The framework uses simple feature manipulation in a fixed pre-trained diffusion model to balance preserving guidance layout with changing appearance.The authors describe this as a framework for diverse text-guided image-to-image translation founded on insights into internal representations.
- Limitations: The method does not work well on detailed label segmentation masks whose regions are colored arbitrarily because it relies on semantic association between source and translated content.The limitation is attributed to the absence of semantic association in the diffusion feature space.
- Limitations: For textureless minimal images, DDIM inversion may sometimes encode dominant low-frequency appearance information, causing appearance leakage into the results.This is reported as an additional limitation of the inversion procedure.
A. Ablations
The ablations identify spatial features and self-attention as complementary contributors to structure preservation, while negative prompting mainly affects appearance deviation on textureless guidance images.
- Negative prompting: Negative prompting slightly increases deviation from the guidance image while causing only a minor reduction in structure preservation.The comparison uses LPIPS distance between the guidance image and output across Wild-TI2I and ImageNet-R-TI2I.
- Feature analysis: The decoder layer used for feature injection captures localized semantic information, while encoder features progress from high-frequency noise toward lower-frequency content.The encoder trend is described as mirroring the decoder trend across layers.
B. Initial noise xT and spatial features
The translated image must share the guidance image’s initial noise because feature correlations depend more strongly on the seed than on the prompt. This supports injecting guidance features from the diffusion process.
- Initial noise: The translated image’s initial noise x_T must match the guidance image’s initial noise x_T^G for the method to work.Because decoder features are injected from the first backward step, the authors attribute this dependency to encoder features at t = T.
- Feature compatibility: The authors hypothesize that the target prompt has little effect on encoder features at t = T, allowing injected decoder features to comply with them.Changing the seed instead creates a mismatch between the encoder and injected decoder features.
- Feature correlation: Feature maps generated from the same initial noise under arbitrary prompts are significantly more correlated than those generated from different seeds under the same prompt.The analysis uses 10 prompts and 10 seeds to form 100 images and compares within-set feature-map variance.
C. Implementation Details
The experiments use Stable Diffusion with deterministic DDIM sampling and several ablations of feature, attention, prompting, and encoder injection.
- Model and sampling: Stable Diffusion v1-4 is used as the pre-trained text-to-image model.The checkpoint is provided through the official HuggingFace webpage.
- Model and sampling: All experiments use deterministic DDIM sampling with 50 steps, while real guidance images undergo deterministic inversion and backward sampling with 1000 steps each.Translation features are extracted only at the 50 sampling steps, with default injection thresholds τA = 25 and τf = 40.
- Prompting: Negative prompting interpolates between predicted noises with a hyperparameter α that is initialized and gradually decreased during sampling.Real guidance images use α0 = 1.0 with a linear scheduler, while generated guidance images use α0 = 0.75; primitive images use an exponential scheduler α(t) = e−6·t.
- Ablations and evaluation: The quantitative evaluation ablates feature injection, self-attention injection, negative prompting, and additional encoder-block feature injection.Scores cover structure preservation, target-text faithfulness, and deviation from the guidance image across three text-guided image-to-image benchmarks.
- Baselines: Competitors are run using official or publicly available implementations, including Prompt-to-Prompt, DiffuseIT, DiffusionCLIP, Text2LIVE, FlexIT, SDEdit, and VQGAN-CLIP.
D.1. ImageNet-R-TI2I benchmark.
The ImageNet-R-TI2I benchmark evaluates translations from diverse ImageNet-R renditions using manually selected guidance images and multiple rendition prompts.
- Benchmark construction: The benchmark uses 30 guidance images selected from 10 ImageNet-R classes, with 3 images per class.The classes include objects such as castles, cats, goldfish, hummingbirds, huskies, jeeps, pandas, penguins, pizzas, and violins.
- Benchmark construction: Each guidance image receives 5 automatically created target prompts using the template “≪rendition≫of a ≪class≫”.Renditions include art, cartoon, graphic, painting, sketch, graffiti, embroidery, origami, sculpture, tattoo, toy, video-game, photo, and image.
- Benchmark construction: The resulting ImageNet-R-TI2I benchmark contains 150 image-text pairs.This total comes from 30 guidance images paired with 5 target prompts each.
- Feature analysis: The supplementary material extends feature visualization to encoder and decoder blocks of the U-Net at diffusion time t=540.
D.2. Wild TI2I benchmark.
The Wild TI2I benchmark contains varied text-image pairs spanning object classes, visual renditions, and semantic detail, and the authors plan to release the benchmarks and code.
- Benchmark composition: The collected benchmark contains 148 text-image pairs covering people, animals, food, and landscapes.Examples span realistic images, drawings, solid masks, sketches, and illustrations with different levels of semantic detail.
- Benchmark composition: Fifty-three percent of the examples are real guidance images gathered from the Web, while the remainder are generated from text.
- Release: The authors will publicly release the benchmarks and code for academic use.