Source-linked AI summary
Blended Latent Diffusion
Omri Avrahami, Ohad Fried, Dani Lischinski
TL;DR
The paper addresses slow diffusion-based local text-driven editing of generic images and adapts a pretrained Latent Diffusion Model to this masked setting. It combines latent Blended Diffusion with latent optimization and a shrinking-mask strategy, reporting faster inference, better precision than baselines, and mitigation of some artifacts.
Problem
Diffusion models enable diverse text-guided image editing but are slow, while Latent Diffusion Models were not designed for local editing of real images.
Method
The method adapts Blended Diffusion to a pretrained LDM, uses latent optimization for reconstruction, and progressively shrinks a dilated mask for thin-mask edits.
Results
The method is faster than baselines, achieves better precision, and mitigates pixel-level and adversarial artifacts through latent-space operation.
Takeaways & Limitations
The system supports zero-shot local text-driven editing of real images using a text prompt and user-provided mask.
Takeaways & Limitations
Generating a ranked batch still takes over a minute on an A10 GPU, limiting applicability on lower-end devices.
Abstract
from arXiv · showhide
The tremendous progress in neural image generation, coupled with the emergence of seemingly omnipotent vision-language models has finally enabled text-based interfaces for creating and editing images. Handling generic images requires a diverse underlying generative model, hence the latest works utilize diffusion models, which were shown to surpass GANs in terms of diversity. One major drawback of diffusion models, however, is their relatively slow inference time. In this paper, we present an accelerated solution to the task of local text-driven editing of generic images, where the desired edits are confined to a user-provided mask. Our solution leverages a recent text-to-image Latent Diffusion Model (LDM), which speeds up diffusion by operating in a lower-dimensional latent space. We first convert the LDM into a local image editor by incorporating Blended Diffusion into it. Next we propose an optimization-based solution for the inherent inability of this LDM to accurately reconstruct images. Finally, we address the scenario of performing local edits using thin masks. We evaluate our method against the available baselines both qualitatively and quantitatively and demonstrate that in addition to being faster, our method achieves better precision than the baselines while mitigating some of their artifacts.
1 INTRODUCTION
The paper targets zero-shot local text-guided editing of generic images, adapting latent diffusion to improve speed while addressing reconstruction and thin-mask challenges.
- Local text-guided editing modifies only a masked part of a generic image, but has received less attention than image generation and global manipulation.
- Latent Diffusion Models offer competitive image quality with faster inference, but were designed for text-to-image generation rather than local editing.
- The method adapts Blended Diffusion to operate in the latent space of a pretrained LDM for zero-shot local editing from an image, text prompt, and mask.
- Latent optimization mitigates inaccurate reconstruction caused by lossy VAE-based latent encodings, especially in faces, text, and high-frequency details.
- For thin masks, the method starts with a dilated mask and gradually shrinks it during diffusion.
- The authors propose precision and diversity metrics and report that their method is faster than baselines while achieving better precision.
2 RELATED WORK
Related work spans text-guided generation, global editing, and local manipulation, with prior methods differing in input type, editing scope, and availability.
- Text-to-image synthesis and global editing: Text-guided image synthesis evolved from RNN and GAN approaches toward transformer-based methods and multimodal generative systems.
- Text-to-image synthesis and global editing: Diffusion-based global editing methods support tasks including translation, colorization, inpainting, uncropping, restoration, compositing, and stroke-based editing.
- Local text-guided image manipulation: Paint By Word introduced zero-shot local text-guided manipulation but operated only on generated images and used separate generative models for each domain.
- Local text-guided image manipulation: Text2LIVE edits object appearance without a pretrained generative model, focusing on colors, textures, fire, and smoke rather than general scene replacement or removal.
- Local text-guided image manipulation: GLIDE and DALL·E 2 are related diffusion-based approaches, while prompt-to-prompt edits generated images without masks and requires source and target prompts.
- Local text-guided image manipulation: At the stated time, Blended Diffusion and GLIDE-filtered were the only publicly available models addressing the paper's setting.
3 LATENT DIFFUSION AND BLENDED DIFFUSION
Latent diffusion accelerates diffusion by moving computation into a lower-dimensional latent space, while Blended Diffusion provides the basis for masked local editing.
- Diffusion models learn to reverse gradual noising, producing progressively less-noisy latents from standard normal noise.
- Latent Diffusion Models first compress images perceptually with an autoencoder, then perform diffusion in the resulting lower-dimensional latent space.
- Conditional latent diffusion can support image-to-image, semantic-map-to-image, and text-to-image transitions when combined with the autoencoder.
- Blended Diffusion combines an ImageNet-trained diffusion prior with CLIP guidance and spatially blends generated noisy images with corresponding noisy input regions.
- The paper incorporates Blended Diffusion's latent blending into an LDM and addresses artifacts arising in the latent space.
4 METHOD
Blended Latent Diffusion adapts latent diffusion for masked text-guided editing while preserving unmasked content. It addresses latent-space reconstruction errors and thin-mask failures through optimization and progressively shrinking masks.
- 4.1 Blended Latent Diffusion: The method encodes the source image into latent space, downsamples the mask, denoises a text-conditioned foreground, and blends it with a noised background at each step.The blended latent is decoded into the edited image after diffusion.
- 4.1 Blended Latent Diffusion: Latent-space diffusion accelerates editing by an order of magnitude and avoids pixel-level clipping artifacts and adversarial examples.The speedup comes from the lower-dimensional latent space and avoiding CLIP-loss gradients at every denoising step.
- 4.4 Evaluation: The method achieves better precision than baselines while also being faster, although progressive mask shrinking struggles with fine details such as a green bracelet.Evaluation is conducted qualitatively and quantitatively using proposed precision and diversity metrics.
- 4.2 Background Reconstruction: VAE-based lossy encoding causes imperfect reconstruction, especially for faces, text, and other high-frequency details outside the mask.Initial reconstruction may visibly differ from the input before diffusion begins.
- 4.3 Thin Masks: Thin masks can become ineffective after latent downsampling, so the method starts with a dilated mask and gradually shrinks it during diffusion.Only the final denoising steps use the thin latent mask, maintaining the text effect through the process.
- 4.2 Background Reconstruction: Per-image decoder-weight optimization preserves fine background details while matching the edited foreground and producing seamless blending.Pixel stitching creates seams, Poisson cloning shifts colors, and latent optimization alone over-smooths the result.
5 RESULTS
The method is compared qualitatively and quantitatively with local text-driven editing baselines, then demonstrated across several editing scenarios. It achieves stronger precision and faster batch inference while preserving backgrounds and reducing some artifacts.
- Qualitative comparison: Our method avoids Blended Diffusion’s pixel-level noise and generally produces better colors and textures, while competing methods can add artifacts or weaken text adherence.GLIDE sometimes gives better shadows but can add artifacts; GLIDE-masked improves masked-region generation at the expense of transitions, while DALL·E 2 often ignores or partially follows prompts.
- Quantitative comparison: Our method outperforms baselines in precision at both the batch and best-result levels.Diversity is second-best; Local CLIP-guided diffusion scores higher but substantially changes the entire image.
- User study: At least 50% of evaluators prefer our method for visual quality and text matching against each compared method.The study uses 50 random predictions and asks evaluators to compare visual quality and text matching.
- Inference time comparison: The method accelerates inference by ×10 versus Blended Diffusion and Local CLIP-guided diffusion at equal batch sizes, or ×20 at recommended batch sizes.It is also ×1.47 faster than PaintByWord++ and ×1.23 faster than GLIDE-filtered; latent-space diffusion and optimizing only the selected result provide the speed advantage.
- Use cases: The system supports adding, modifying, or replacing objects, injecting plausible text, replacing backgrounds, and transforming scribbles into natural objects.These use cases are demonstrated on generic real-world images.
6 LIMITATIONS & CONCLUSIONS
The method enables realistic local text-based image editing with strong background preservation, while remaining sensitive to inputs and limited by inference speed, ranking shortcomings, and text misinterpretation.
- Limitations: Over one minute on an A10 GPU is still required to generate a ranked prediction batch, limiting applicability on lower-end devices.The authors identify further inference acceleration as an important research avenue.
- Limitations: CLIP ranking considers only the generated masked area, so outputs can have realistic local regions without appearing realistic overall.The authors motivate a more holistic ranking system using an example from Figure 10.
- Limitations: The method is sensitive to small changes in the prompt, mask, or input image, and guiding text may be interpreted as a text-generation task.The latter failure can produce a movie poster named “big mountain” instead of a big mountain.
- Conclusions: The method enables local image editing using text, with results realistic enough for real-world editing scenarios.The authors report this conclusion despite several unresolved open problems.
- Conclusions: Near-perfect background preservation supports interactive editing across different image regions without altering areas edited earlier.The paper demonstrates this gradual editing workflow in Figure 17.
- Conclusions: Latent-space diffusion is faster than pixel-based diffusion and mitigates pixel-level artifacts observed in Blended Diffusion.Additional visual comparisons are provided in Figure 18.
B.3 Additional Comparison Against the Baselines
The method is compared qualitatively with multiple baselines and supports several local editing capabilities, including object addition and one-to-many prediction. Blended Diffusion preserves backgrounds and generally follows prompts, while still exhibiting noise-level artifacts.
- Additional baseline comparisons: Blended Diffusion preserves the input background and follows the guiding text in nearly all examples, except the graffiti example.The comparison includes Local CLIP-guided diffusion, PaintByWord++, GLIDE-filtered, and DALL·E 2.
- Additional editing capabilities: The supplementary examples cover adding objects, replacing backgrounds, and generating multiple predictions for one-to-many editing tasks.These figures provide additional demonstrations of the model’s editing capabilities.
- Background reconstruction: Background reconstruction is optimized on a per-image basis through the decoder’s weights.The reconstruction objective balances the edited foreground against preservation of the original background.
- Background reconstruction: Removing the foreground term from the reconstruction objective makes colors in the edited area less vivid.The experiments use λ = 100 to assign most weight to the background term.
C IMPLEMENTATION DETAILS
The experiments use released pretrained models and standard reconstruction optimizers on freely licensed or privately collected real images. DALL·E 2 is available only through a limited-credit online demo, preventing quantitative precision and diversity evaluation.
- Pretrained models: The experiments use pretrained LDM, CLIP ViT-B/16, Blended Diffusion, and GLIDE-filtered models implemented in PyTorch.All listed methods were released under MIT licenses.
- Model availability: DALL·E 2 was accessed through an online editing demo with limited credit tokens, so precision and diversity metrics could not be calculated for it.The model itself was unavailable to the authors.
- Input data: All input images were either freely released under Creative Commons licenses or drawn from the authors’ private collection.The inputs are real images rather than synthetic images.
- Optimization settings: Poisson image blending used the OpenCV implementation, while latent and weight optimization used Adam with learning rate 0.0001 for 75 steps per image.These choices specify the reconstruction procedure used in the experiments.
- Mask processing: Progressive mask shrinking begins with dilated latent masks using 3 × 3, 5 × 5, and 7 × 7 kernels across four diffusion stages.The first stage uses the most dilated mask.
C.1 Precision & Diversity Metrics
Precision and diversity are evaluated with an ImageNet classifier and masked foreground comparisons across randomized image-mask-prompt inputs. The protocol uses top-5 classification correctness and average pairwise LPIPS distance.
- Evaluation model: An EfficientNet ImageNet classifier replaces CLIP for evaluation because CLIP-based evaluation is not correlated with human judgments for CLIP-guided generators.The concern is susceptibility to adversarial examples.
- Evaluation setup: The benchmark samples 50 random images, rectangular masks spanning dimensions from [dim/5, dim/2], and random ImageNet classes as text prompts.Each baseline generates predictions using its recommended batch size.
- Precision: Precision counts a prediction as correct when the target ImageNet class appears among the classifier’s top-5 predictions after masking the edited result.Accuracy is averaged at the batch level for each input, with an additional top-ranked prediction measure.
- Diversity: Diversity is measured among correctly classified predictions by masking the foreground and averaging pairwise LPIPS distances.Only images classified correctly are retained because they are considered relevant to end users.
- Thin masks: Thin masks can become even thinner after downscaling into latent space, limiting or eliminating the edit effect.The paper identifies this as a specific limitation of local latent editing.
C.2 User Study
A user study compares the method with available baselines on visual quality and text matching using paired judgments from Mechanical Turk. The reported binomial test indicates statistically significant results, while unavailable models are excluded.
- Study design: The study compares paired images from the method and a baseline for visual quality and correspondence to the guiding text.Evaluators answer separate questions for image quality and prompt matching.
- Study design: Each question receives five ratings per task, producing 250 ratings per task, with one hour allotted for each image-pair task.Raters can select equal quality or equal text matching when neither image is better.
- Study scope: The user study includes only freely available models and excludes GLIDE-full and DALL·E 2 because they are unavailable.The exclusion limits the set of models directly compared by human raters.
- Statistical analysis: p-value < 5%: A binomial statistical test suggests that the user-study results are statistically significant.The analysis is reported in Table 4.
- Ranking effectiveness: The top 20% of CLIP-ranked predictions are consistently better than the bottom 20%, although individual adjacent rankings are not strictly ordered.The ranking is useful at the group level rather than reliably distinguishing every single image.
- Ranking effectiveness: CLIP ranking is essential for the Blended Diffusion baseline, making its full-batch inference time the relevant comparison.This affects how inference times should be compared with the proposed method.
D SENSITIVITY ANALYSIS
Small changes to the input prompt, mask, or image can produce small changes in the method’s output.
- Small prompt changes may produce small output changes.
- Making the input mask larger or smaller may change the output result.
- Rotating the image by 5° or blurring it with a Gaussian kernel can change the output.
E SOCIETAL IMPACT
The paper identifies both beneficial and harmful societal implications of easier image manipulation, alongside method-specific artifacts, sensitivities, and dataset-related risks.
- Easier content manipulation can democratize creation, enhance creativity, and enable new applications, but can also facilitate harmful uses.Mentioned harms include fake news, harassment, bullying, and adverse psychological or sociological effects.
- The model’s noncurated LAION-400M training data may contain disturbing content that the model could repeat.The dataset contains 400M text-image pairs collected from the internet.
- Text-guided editing may inherit biases present in the training data.
- Blended Diffusion produces noticeable pixel-level noise artifacts compared with the paper’s method.
- Removing the foreground term in Equation (3) results in slightly less vivid colors.
- CLIP ranking separates the top and bottom 20% of 24 predictions, but does not reliably order individual images.