PaperlayerView exact revision on arXiv

Source-linked AI summary

An Edit Friendly DDPM Noise Space: Inversion and Manipulations

Inbar Huberman-Spiegelglas, Vladimir Kulikov, Tomer Michaeli

arXiv:2304.06140v3cs.CVcs.LG

TL;DR

Diffusion editing of real images requires inversion, but native DDPM noise is difficult to use because it lacks an edit-friendly structure. The paper introduces an alternative inversion that extracts edit-friendly noise maps, enabling perfect reconstruction, structure-preserving semantic edits, and diverse outputs without optimization.

  • Problem

    Real-image diffusion editing requires inversion, while DDPM inversion is difficult because many consistent noise sequences exist and not all are edit friendly.

  • Method

    The paper proposes an alternative, optimization-free DDPM inversion that extracts noise maps with stronger image-structure encoding for editing.

  • Results

    The edit-friendly maps enable perfect reconstruction, preserve structure during text-conditioned semantic edits, and support diverse outputs and integration with existing editing methods.

  • Takeaways & Limitations

    Fixing the extracted maps while changing the text condition provides a simple route to structure-preserving and diverse editing of real images.

  • Takeaways & Limitations

    Auxiliary image sequences that are far from the network’s training-input distribution may produce poor results when the text condition changes.

Abstract

from arXiv · show

Denoising diffusion probabilistic models (DDPMs) employ a sequence of white Gaussian noise samples to generate an image. In analogy with GANs, those noise maps could be considered as the latent code associated with the generated image. However, this native noise space does not possess a convenient structure, and is thus challenging to work with in editing tasks. Here, we propose an alternative latent noise space for DDPM that enables a wide range of editing operations via simple means, and present an inversion method for extracting these edit-friendly noise maps for any given image (real or synthetically generated). As opposed to the native DDPM noise space, the edit-friendly noise maps do not have a standard normal distribution and are not statistically independent across timesteps. However, they allow perfect reconstruction of any desired image, and simple transformations on them translate into meaningful manipulations of the output image (e.g. shifting, color edits). Moreover, in text-conditional models, fixing those noise maps while changing the text prompt, modifies semantics while retaining structure. We illustrate how this property enables text-based editing of real images via the diverse DDPM sampling scheme (in contrast to the popular non-diverse DDIM inversion). We also show how it can be used within existing diffusion-based editing methods to improve their quality and diversity. Webpage: https://inbarhub.github.io/DDPM_inversion

1. Introduction

Diffusion editing real content requires inverting the generation process, but DDPM inversion is difficult because many consistent noise sequences are possible and not all support editing. The paper introduces edit-friendly noise maps that preserve structure during prompt changes and support fast, diverse editing.

  • Editing real content requires extracting noise vectors that reconstruct the image through the reverse diffusion process.
  • DDPM inversion remains challenging, while common DDIM inversion is approximate, needs many timesteps for accuracy, and can produce sub-optimal text-guided edits.
  • DDPM has infinitely many perfectly reconstructing noise sequences, but only some are suitable for editing, such as preserving structure when the text condition changes.
  • The proposed inversion imprints images more strongly onto noise maps through higher variances, improving structure preservation without optimization or model fine-tuning.
  • The edit-friendly space supports semantic edits while preserving structure, including text guidance and hand-drawn colored-stroke manipulations.
  • Integrating the inversion with existing editing methods improves fidelity and enables diverse edited outputs through stochastic noise-vector extraction.

2. Related work

Related work uses approximate DDIM inversion, forward propagation, optimization, attention manipulation, model fine-tuning, and task-specific conditional diffusion for image editing. DDPM-based real-image editing is less common and often trades realism against faithfulness.

  • DDIM inversion maps one initial noise vector deterministically to an image, but approximation errors can accumulate under classifier-free guidance.
  • Null-text optimization and regularization improve DDIM reconstruction, while forward propagation offers similar accuracy without optimization and faster editing.
  • DDPM real-image editing commonly adds noise to an image before backward diffusion, creating a tradeoff between generated realism and fidelity to original contents.
  • Text-driven editing methods manipulate spatial features, self-attention, or attention maps, and DiffEdit derives edit masks from the source and target prompts.
  • Other approaches optimize text embeddings or fine-tune diffusion models, while Palette and InstructPix2Pix learn conditional models for specific editing tasks.
  • In DDPM, the latent code comprises T + 1 noise maps used across the reverse diffusion process to synthesize x0.

3. The DDPM noise space

DDPM’s stochastic generation uses a sequence of timestep-specific noise maps, but consistent inversions are not necessarily suitable for editing. The proposed inversion imprints image structure more strongly, enabling reconstruction and meaningful latent-space manipulations.

  • DDPM generation: DDPM reverses a process that gradually transforms a clean image into white Gaussian noise, using x_T and stochastic maps z_t across timesteps.The reverse process iteratively applies x_{t-1} = μ̂_t(x_t) + σ_t z_t, with η controlling stochasticity between DDPM and DDIM.
  • DDPM latent codes: The sequence {x_T, z_T, …, z_1} uniquely determines the generated image, but a given image has infinitely many perfectly reconstructing noise sequences.This overcomplete latent space creates flexibility, while leaving open which consistent inversion is useful for editing.
  • Inversion challenge: Naive auxiliary trajectories can fall outside the network’s training distribution, causing prompt changes to produce poor results with fixed extracted noise maps.CycleDiffusion keeps extracted noise statistically similar to the generative process, but this native-like space does not preserve global structure under editing.
  • Edit-friendly inversion: The proposed inversion constructs auxiliary images directly from x_0, producing higher-variance, negatively correlated noise maps that encode input structure more strongly.The method extracts z_t from the reverse recursion, compensates for numerical error accumulation, and can be applied to conditional or guided diffusion processes.
  • Reconstruction and diversity: The inversion reconstructs the input up to machine precision, while stochastic construction yields multiple edited variants from different consistent noise sequences.This diversity is available for text-based editing and is not naturally provided by DDIM inversion.
  • Latent manipulations: Simple transformations of the edit-friendly maps preserve image structure during shifts and support masked color edits without modifying textures.Shifting the native code loses structure, whereas shifting the proposed code causes only minor degradation; color edits use a strength parameter and timestep range.

4. Text-Guided Image Editing

The edit-friendly latent code enables text-guided changes to real images while preserving structure, and supports strong color edits without modifying textures or structures.

  • Text-guided editing: Fixing extracted noise maps while changing from source to target prompts modifies image semantics while preserving structure and enables diverse outputs.The same inversion can also be integrated into methods that rely on DDIM inversion to improve preservation of fine textures and global structure.
  • Text-guided editing: Integrating the inversion into existing diffusion editing methods improves preservation of fine textures such as fur, flowers, and leaves.The comparison specifically contrasts the integrated approach with methods using DDIM inversion.
  • Color manipulation: The method produces strong color edits without modifying textures or structures, whereas SDEdit trades off mask integration and structural preservation as noise changes.The example uses T2 = 70, T1 = 20, and s = 0.05 with an unconditional ImageNet model and 100 inference steps.

5. Experiments

Experiments evaluate the method on real-image text-guided editing using complementary fidelity, text-compliance, timing, and qualitative comparisons across datasets and baselines.

  • Experimental setup: The evaluation uses real images with source and target prompts, testing the extracted latent code alone and combined with existing DDIM-based editing methods.The implementation uses Stable Diffusion at 512 × 512 × 3 image resolution and 64 × 64 × 4 latent resolution.
  • Datasets: The two datasets contain 212 and 60 image-text pairs, respectively, covering modified ImageNet-R-TI2I and modified Zero-Shot I2IT.The second dataset contains 15 images per category across Cat, Dog, Horse, and Zebra.
  • Metrics: LPIPS measures structure preservation, CLIP-based scores measure text compliance, and editing time measures computational cost.Lower LPIPS is better, while higher CLIP-based scores are better; diversity is reported in the supplementary material.
  • Quantitative comparison: Figure 10 compares LPIPS and CLIP scores across methods and parameter settings on the modified ImageNet-R-TI2I dataset.The plotted methods include the proposed inversion, P2P with the proposed inversion, CycleDiffusion, and mid-way DDIM inversion.
  • Comparisons: Compared with baselines including PnP, EDICT, null-text inversion, CycleDiffusion, P2P, and DDIM inversion, the method balances fidelity to the input with target-text adherence.PnP and null-text inversion often preserve structure but require more than 2.5 minutes per image, while CycleDiffusion applies weaker edits when preserving structure.
  • Zero-Shot I2I translation: Both Zero-Shot and Zero-Shot with the proposed inversion achieve CLIP accuracy of 0.88, while LPIPS improves from 0.35 to 0.27 with the proposed inversion.The lower LPIPS score indicates greater similarity to the input image, while the accompanying images retain added detail such as fur and flowers.

6. Conclusion

The paper concludes that its DDPM inversion produces noise maps better suited to image editing because they encode image structure more strongly than regular-sampling noise maps.

  • Conclusion: The proposed DDPM inversion encodes image structure more strongly than regular sampling, making its noise maps better suited for image editing.The conclusion highlights text-based editing both with the method alone and in combination with other editing methods.
  • Latent manipulations: Shifting the extracted latent code can shift the input image by inserting boundary columns or rows copied from another region of the noise map.For a shift of d pixels, the experiments copy columns or rows indexed {50, . . . , 50 + d − 1}.
  • Latent manipulations: The shifting experiment plots mean squared error over valid pixels, with colored regions showing one standard error of the mean in each direction.This supplementary figure evaluates the effect of shifting the latent code during image generation.

B. The effect of the numerical error

The numerical-error correction step supports perfect reconstruction, especially when reconstruction hyperparameters are matched, while its effect is negligible for editing.

  • Error correction: The correction step is designed to avoid numerical drifting and assist perfect reconstruction.Reconstruction requires identical prompt, Tskip, and strength parameters during inversion and sampling.
  • Error correction: With strength = 30, reconstruction PSNR without correction can fall below 17dB.This quantifies the reconstruction degradation associated with omitting the correction step in the example from Fig. S2.
  • Error correction: For editing, the correction changes results little: edited-image PSNR between corrected and uncorrected outputs is 67.4dB.The correction is therefore important for reconstruction fidelity but not needed for editing in the reported example.
  • Error correction: Figure S2 compares reconstructions and edited outputs across strength settings, with prompts shown above the image columns and Tskip fixed at 36.The displayed examples include images with and without error correction.

C. CycleDiffusion

CycleDiffusion noise maps retain sampling-like statistics, limiting simple artifact-free pixel manipulations and weakening text-editing fidelity. The paper also examines parameter effects and integration with Prompt-to-Prompt.

  • CycleDiffusion noise maps have statistical properties resembling regular DDPM sampling, including per-pixel standard deviations and timestep correlations.These statistics are illustrated for CycleDiffusion, regular sampling, and the proposed approach.
  • Simple flips and shifts on CycleDiffusion noise maps cannot produce artifact-free pixel-space effects while preserving structure.The comparison considers a horizontal flip and a 30-pixel rightward horizontal shift.
  • At matched CLIP similarity, the proposed inversion achieves better input-image preservation than CycleDiffusion in text-based editing.CycleDiffusion’s LPIPS loss increases significantly when tuned for higher CLIP similarity.
  • Tskip controls the generation start timestep, while strength controls classifier-free guidance during text-guided editing.These parameters determine how generation proceeds after extracting latent noise maps with the source text.
  • Prompt-to-Prompt integration is evaluated with reduced cross-attention control because the proposed latent space already strongly encodes structure.The cross-attention hyper-parameter is decreased from 0.8 to 0.6.

F. Additional details on experiments and further numerical evaluation

The experiments use Stable Diffusion v1-4 on an RTX A6000 and evaluate editing with perceptual, text-alignment, runtime, and diversity measures.

  • All text-based editing experiments use the StableDiffusion-v-1-4 checkpoint as the pretrained text-to-image model.Experiments were run on an RTX A6000 GPU.
  • Evaluation measures CLIP-based scores, LPIPS scores, running time, and diversity among generated outputs.Diversity is computed from average LPIPS distance across eight outputs per image and source text.

F.1. Experiments on the modified ImageNet-R-TI2I

On the modified ImageNet-R-TI2I evaluation, the proposed inversion balances source-image fidelity, text compliance, and short edit times, while improving Prompt-to-Prompt integration.

  • The modified ImageNet-R-TI2I dataset contains 44 images, including 30 from PnP and 14 from external and existing-method sources.The added images were checked for reasonable source and target prompts.
  • Prompt-to-Prompt uses the replace mode for rendering or class changes and the refine mode for added tokens or different prompt lengths.All methods use 100 forward and backward steps except PnP, whose supplied code does not support changing this parameter.
  • The proposed inversion achieves a good LPIPS–CLIP balance while requiring short edit times.Hyper-parameters were selected to obtain the best CLIP loss under an LPIPS distance constraint of 0.3 for several methods.
  • Integrating the proposed inversion into Prompt-to-Prompt improves performance on both evaluation metrics.The comparison is summarized in Table S2 and Figure S8.
  • The proposed method, CycleDiffusion, and null-text inversion support diversity among generated outputs.This comparison is included in the numerical evaluation of all methods.

F.2. Experiments on the modified zero-shot I2IT dataset

On the modified Zero-Shot I2IT dataset, integrating the proposed inversion improves similarity to the input while retaining high CLIP accuracy and producing non-negligible diversity.

  • The modified Zero-Shot I2IT dataset contains four categories: cat, dog, horse, and zebra.Ten images per category came from prior work, with five additional Internet images per category.
  • Zero-Shot I2I uses predefined source-target classes rather than source-target prompt pairs.When combined with the proposed inversion, the method uses Tskip = 0 and an empty source prompt.
  • Integrating the proposed inversion improves similarity to the input image while keeping CLIP accuracy high.The comparison is summarized in Table S4.
  • The combined method exhibits non-negligible diversity among generated outputs.The reported comparison uses the hyper-parameters summarized for the modified Zero-Shot I2IT evaluation.

G. Additional results

The supplementary figures show that the method produces varied text-based edits while preserving input structure and satisfying target prompts, alongside qualitative comparisons with other methods.

  • Repeated inversions with identical source and target prompts produce non-negligibly variable edits that remain structurally consistent and prompt-compliant.S9 illustrates variation in a sculpture cat’s eyes and mouth and in rainbow placement and angle.
  • Additional examples show that edited details such as an origami dog’s eyes and nose and a zebra’s stripes change across samples.
  • Figures S11 and S12 provide qualitative comparisons between all tested methods on the ImageNet-R-TI2I dataset.
Loading 2304.06140v3…