PaperlayerView exact revision on arXiv

Source-linked AI summary

Blended Diffusion for Text-driven Editing of Natural Images

Omri Avrahami, Dani Lischinski, Ohad Fried

arXiv:2111.14818v2cs.CVcs.GRcs.LG

TL;DR

The paper addresses local text-guided editing of generic real-world images while preserving unedited regions. It combines CLIP-guided diffusion with spatial blending and augmentations, and reports stronger realism, background preservation, and text correspondence than baselines. The method supports multiple text-driven editing applications but is limited by slow inference and inherited CLIP weaknesses.

  • Problem

    Existing text-driven manipulation methods do not provide a general solution for editing specified regions of diverse real images while preserving the rest.

  • Method

    The method combines pretrained CLIP for text guidance with DDPM generation, blending noised input images with local guided diffusion latents across noise levels.

  • Results

    The method outperforms three baselines in realism, background preservation, and correspondence to the text prompt.

  • Takeaways & Limitations

    The approach provides general-purpose region-based editing for real, diverse images and supports multiple outputs and applications such as object replacement and background editing.

  • Takeaways & Limitations

    Sequential DDPM inference takes about 30 seconds per image, limiting real-time use and applicability on weak devices.

Abstract

from arXiv · show

Natural language offers a highly intuitive interface for image editing. In this paper, we introduce the first solution for performing local (region-based) edits in generic natural images, based on a natural language description along with an ROI mask. We achieve our goal by leveraging and combining a pretrained language-image model (CLIP), to steer the edit towards a user-provided text prompt, with a denoising diffusion probabilistic model (DDPM) to generate natural-looking results. To seamlessly fuse the edited region with the unchanged parts of the image, we spatially blend noised versions of the input image with the local text-guided diffusion latent at a progression of noise levels. In addition, we show that adding augmentations to the diffusion process mitigates adversarial results. We compare against several baselines and related methods, both qualitatively and quantitatively, and show that our method outperforms these solutions in terms of overall realism, ability to preserve the background and matching the text. Finally, we show several text-driven editing applications, including adding a new object to an image, removing/replacing/altering existing objects, background replacement, and image extrapolation. Code is available at: https://omriavrahami.com/blended-diffusion-page/

1. Introduction

The paper targets text-guided local editing of generic real-world images, combining CLIP and DDPM while preserving unmasked regions. Its blending scheme and augmentations address background coherence and adversarial results.

  • Existing GAN-based editing methods are often domain-restricted and require inversion for real images, with a trade-off between reconstruction accuracy and editability.
  • The proposed approach performs region-based text-guided editing on generic natural images while preserving the user-specified unedited region.
  • DDPM supplies the generative backbone for natural-looking results, while CLIP guides edits toward the user-provided text prompt.
  • The method blends CLIP-guided diffusion latents with noised input images at each diffusion step to produce coherent edits and preserve surrounding content.
  • Augmentations during diffusion reduce adversarial results, enabling gradient-based diffusion guidance.

2. Related Work

Prior text-guided manipulation methods commonly generate abstract images, rely on restricted-domain generators, or lack spatial control and background preservation. Diffusion-based alternatives broaden the approach, but some require task-specific training.

  • Text-to-image synthesis: CLIP-guided generative methods mainly create abstract artworks and cannot edit parts of real images while preserving the remainder.
  • Text-driven image manipulation: This paper instead focuses on text-driven manipulation restricted to a user-specified region rather than general text-to-image synthesis.
  • Text-driven image manipulation: StyleCLIP edits real images through StyleGAN2 inversion but remains restricted to supported domains and lacks spatial control.
  • Text-driven image manipulation: Bau et al.'s GAN-based method may alter backgrounds, focuses mainly on bedrooms and color or texture edits, and operates only on generated images.
  • Text-driven image manipulation: Concurrent diffusion approaches address global manipulation or text-to-image synthesis, while GLIDE supports local editing using a designated trained diffusion model.

3. Denoising Diffusion Probabilistic Models

DDPMs generate images by reversing a learned diffusion process: data are progressively noised toward Gaussian noise, then iteratively denoised with a learned posterior. The framework supports direct noisy-state sampling and high-quality generation.

  • A forward Markovian process adds Gaussian noise with variance β_t to produce latent states x_1 through x_T from data x_0.
  • When T is sufficiently large, the terminal latent x_T is nearly isotropic Gaussian noise.
  • Any noisy state x_t can be sampled directly from x_0 without generating intermediate states.
  • The reverse process starts from Gaussian noise and iteratively removes noise to produce samples from the training distribution.
  • A neural network p_θ approximates the unknown reverse posterior by predicting the mean and covariance of x_{t-1} given x_t.
  • Ho et al.'s formulation predicts the added noise ε_θ(x_t,t), from which the reverse-process mean can be derived using Bayes' theorem.
  • Diffusion models have been shown to achieve high-quality image generation, including sample quality superior to contemporary generative models in reported FID comparisons.

4. Method

The method combines local CLIP guidance with DDPM denoising and progressively blends edited foreground latents with noised input backgrounds. This design addresses the trade-off between text alignment and background preservation while supporting coherent, multiple outputs.

  • The task seeks an edited region consistent with text while keeping the complementary image area close to the source and making the boundary seamless.
  • 4.1. Local CLIP-guided diffusion: The initial local CLIP-guided diffusion method combines masked text guidance with a background-preservation loss, but the two objectives create an inherent trade-off.The loss is DCLIP(bx0, d, m) + λDbg(x, bx0, m).
  • 4.1. Local CLIP-guided diffusion: The method estimates clean images from noisy diffusion latents and uses CLIP cosine distance between masked image and text embeddings for local guidance.
  • 4.2. Text-driven blended diffusion: Extending augmentations average CLIP gradients across transformed copies, while multiple generated results can be ranked with CLIP for selection.
  • 4.2. Text-driven blended diffusion: Text-driven blended diffusion performs a CLIP-guided foreground step, obtains a correspondingly noised input background, and combines them using the mask at every diffusion stage.The blended latent is xt−1 = xt−1,fg ⊙m + xt−1,bg ⊙(1−m).
  • 4.2. Text-driven blended diffusion: The final step replaces the area outside the mask with the input image, strictly preserving the background.

5. Results

The method is evaluated against multiple baselines on synthetic and real images, showing stronger realism, background preservation, and coherence. Ablations and applications demonstrate the effects and breadth of text-guided editing.

  • Comparisons: The method preserves the background perfectly and produces natural, coherent edits, whereas competing methods introduce global changes or unrealistic results.
  • Comparisons: A user study found that the method outperforms three baselines in realism, background preservation, and correspondence to the text prompt.Participants rated each result on a 1–5 Likert scale across the three criteria.
  • Ablation: Extending augmentations makes generated images more visually plausible and more coherent with the background under the same random seed.
  • Applications: The method supports object addition, removal, replacement, and partial alteration, including multiple plausible outcomes for the same guiding text.
  • Applications: Text guidance enables background replacement, scribble-guided object insertion, and image extrapolation with different prompts controlling different directions.

6. Limitations and Future Work

The method is limited by slow diffusion-based inference, imperfect result ranking, and inherited CLIP biases. Future work considers noise-robust CLIP training and extending the task to other modalities.

  • Limitations: Generating one image takes about 30 seconds on a modern GPU, limiting real-time use and applicability on weak end-user devices.The sequential DDPM process and generation of multiple samples contribute to the inference-time cost.
  • Limitations: Ranking considers only the edited area, so partial objects can receive high scores despite poor overall coherence.
  • Limitations: Because the method relies on CLIP, it can inherit typographic biases and generate text-bearing signs instead of the requested object.
  • Future Work: Future work proposes training CLIP with noisy images and extending the task to general-purpose editing of 3D objects or videos.

7. Societal Impact

The method has legitimate creative uses but also lowers the technical barrier to manipulating photographs. This creates risks including fake-news credibility, harassment, and reduced trust in photographic evidence.

  • The method can support legitimate uses while also enabling photo-based deception, bullying, harassment, extortion, and psychological harm.
  • By making manipulation easier for users with limited technical capabilities, the work may increase misuse and undermine trust in photographic evidence.

8. Conclusions

The paper presents text-driven editing for natural images and supports iterative refinement across diverse editing applications. It also documents artifacts, failure cases, and implementation boundaries.

  • Applications: The method supports adding, changing, and removing objects and backgrounds, plus scribble- and clip-art-guided edits in mixed editing sessions.The supplemental examples describe compositional editing without modifications for mixed sessions.
  • Iterative editing: Users can progressively refine results by adding masks, correcting rough regions, and omitting text prompts for conventional inpainting.An example replaces a background, removes unwanted objects, and refines the initial mask in successive steps.
  • Iterative editing: The raw results shown in the paper generally omit incremental user refinements, despite the method supporting iterative editing.The supplemental document states that main-paper and supplemental results are usually presented without further interaction.
  • Failure cases: The method can produce artifacts, typographically induced failures, and objects whose apparent size does not fit the surrounding photograph.These failure cases are explicitly documented in the supplemental material.
  • Method boundary: Naive blending of the input and diffusion result inside the mask produces unnatural results, motivating the paper’s progressive blending approach.The supplemental material illustrates this failure directly.
  • Implementation boundary: Most results use 256 × 256 unconditional DDPM outputs, although the method can also use a 512 × 512 model and is not limited to one resolution.The paper reports demonstrations beyond 256 × 256 and describes use of OpenAI’s unconditional 512 × 512 model.

B. Implementation Details

The implementation combines pretrained CLIP and diffusion components with fixed model choices, task-specific diffusion settings, and augmentation-based gradient computation. It also generates multiple candidates for selection.

  • Models: The experiments use a pretrained OpenAI CLIP ViT-B/16 model and an unconditional 256 × 256 diffusion model.Both pretrained components are used without parameter changes to CLIP or prompt engineering.
  • Diffusion settings: The standard diffusion schedule uses k = 75 steps, with k = 60 for scribble editing and k = 67 for background editing.These settings allow the model to alter the input image while varying by application.
  • Augmentations: The method uses N = 16 extending augmentations to mitigate adversarial example phenomena.The implementation details identify this value as sufficient for mitigation.
  • Sampling: The system generates 64 samples and uses CLIP-based ranking to select the best results.Multiple stochastic outcomes are produced for each input before selection.
  • Augmentations: Each augmented copy receives a projective transformation of the image and mask, and the resulting CLIP gradients are averaged.The procedure resizes inputs for CLIP, creates transformed copies, computes gradients, and averages them.
  • Ablation: The augmentation ablation finds that augmented outputs are more visually plausible and coherent than outputs without augmentations.The comparison uses the same random seed for runs with and without augmentation.

B.3. Inference Time

Inference is accelerated through batching and parallel GPU generation, while the applications use diffusion-based procedures for object editing, scribble transformation, and image extrapolation.

  • Inference time: The method and Local CLIP-guided diffusion each require 27 seconds per image, compared with 78 seconds for PaintByWord++.These are the reported single-image synthesis times under the stated setup.
  • Acceleration: Batch generation and multi-GPU parallelism accelerate production of multiple stochastic results.The experiments concurrently use four NVIDIA A10 GPUs.
  • Inference time: Using these accelerations, the system generates 64 synthesis results in about 6 minutes, or less than 6 seconds per image.The reported throughput uses four NVIDIA A10 GPUs.
  • Scribble-guided editing: Scribble-guided editing masks the scribble area, supplies a text prompt, and applies the object-altering algorithm.Longer diffusion preserves mainly the scribble’s color while increasing realism and diversity; shorter diffusion modifies it less.
  • Image extrapolation: Image extrapolation sequentially inpaints translated missing regions, repeats the process in both directions, and denoises the chained result.The procedure supports arbitrarily large output resolution, with gradual diffusion steps improving the process.

B.6. Ranking Implementation Details

The ranking implementation generates many candidates and selects from the highest-ranked results, but the ranking is imperfect because it evaluates only the generated region.

  • Candidate selection: The implementation ranks 64 synthesis results and manually chooses from the top 10 ranked images.The top 20% are almost always better than the bottom 20%, although adjacent rankings are unreliable.
  • Ranking limitation: The ranking is imperfect because it considers only the generated area, so the top-ranked image is not always better than the second-ranked image.The authors nevertheless report a consistent quality separation between the top and bottom 20%.

C. User Study

A 35-participant user study evaluated predictions for realism, background preservation, and correspondence to the guiding text. The method significantly outperformed all other tested conditions across every measured criterion.

  • The study involved 35 participants who rated model predictions from input image, mask, and text description.
  • Participants scored overall realism, preservation of the unedited background, and correspondence to the guiding text on a 1–5 scale.
  • Questions were randomly ordered, and participants could revise ratings before submission.
  • p < 10^-130: differences between conditions were statistically significant under the Kruskal-Wallis test.Tukey’s honestly significant difference analysis found the method significantly better than every other condition.
  • The method was statistically better than all other methods on every measured condition in the user study.
Loading 2111.14818v2…