PaperlayerView exact revision on arXiv

Source-linked AI summary

EDICT: Exact Diffusion Inversion via Coupled Transformations

Bram Wallace, Akash Gokul, Nikhil Naik

arXiv:2211.12446v2cs.CVcs.AIcs.LG

TL;DR

Real-image inversion is needed for text-guided editing, but DDIM inversion can be unstable because its local linearization errors accumulate. EDICT uses coupled noise vectors to make inversion mathematically exact, enabling high-fidelity reconstructions and complex edits without training or prompt tuning.

  • Problem

    DDIM inversion for real images is unstable, causing incorrect reconstruction and content loss during inversion-based editing.

  • Method

    EDICT reformulates DDIM using two coupled noise vectors and intermediate mixing layers to enable mathematically exact inversion.

  • Results

    EDICT reconstructs real images with high fidelity and supports local, global, and stylization edits while preserving original image structure.

  • Takeaways & Limitations

    EDICT can be combined with any pretrained DDM without model finetuning, prompt tuning, or extra images.

Abstract

from arXiv · show

Finding an initial noise vector that produces an input image when fed into the diffusion process (known as inversion) is an important problem in denoising diffusion models (DDMs), with applications for real image editing. The state-of-the-art approach for real image editing with inversion uses denoising diffusion implicit models (DDIMs) to deterministically noise the image to the intermediate state along the path that the denoising would follow given the original conditioning. However, DDIM inversion for real images is unstable as it relies on local linearization assumptions, which result in the propagation of errors, leading to incorrect image reconstruction and loss of content. To alleviate these problems, we propose Exact Diffusion Inversion via Coupled Transformations (EDICT), an inversion method that draws inspiration from affine coupling layers. EDICT enables mathematically exact inversion of real and model-generated images by maintaining two coupled noise vectors which are used to invert each other in an alternating fashion. Using Stable Diffusion, a state-of-the-art latent diffusion model, we demonstrate that EDICT successfully reconstructs real images with high fidelity. On complex image datasets like MS-COCO, EDICT reconstruction significantly outperforms DDIM, improving the mean square error of reconstruction by a factor of two. Using noise vectors inverted from real images, EDICT enables a wide range of image edits--from local and global semantic edits to image stylization--while maintaining fidelity to the original image structure. EDICT requires no model training/finetuning, prompt tuning, or extra data and can be combined with any pretrained DDM. Code is available at https://github.com/salesforce/EDICT.

1. Introduction

Diffusion-model image editing requires inversion, but DDIM inversion can distort real images. EDICT introduces coupled transformations for exact inversion and high-fidelity editing without training or tuning.

  • Inversion finds the initial noise vector that generates a given input image through the diffusion process.
  • DDIM provides deterministic inversion, but local approximation errors can destabilize reconstruction and limit text-editing strength.
  • EDICT maintains two coupled noise vectors, enabling mathematically exact inversion for model-generated and real images.
  • EDICT can combine with any pretrained DDM without computationally expensive finetuning, prompt tuning, or multiple images.
  • EDICT supports complex real-image edits, including background, object, breed, and global transformations while preserving fine-grained content.
  • EDICT reconstructs complex scenes more faithfully than DDIM in ImageNet and COCO examples using 50 steps.

2. Related Work

Related work spans diffusion sampling, normalizing flows, and diffusion-based image editing. EDICT draws on invertible coupling layers while addressing inversion-based editing limitations.

  • DDMs generate images from noise using thermodynamics-inspired training and include stochastic DDPM and deterministic DDIM sampling methods.
  • Normalizing flows learn invertible mappings between latent Gaussian distributions and image space, with coupling layers inspiring EDICT.
  • SDEdit, Prompt-to-Prompt, and DiffEdit demonstrate guidance-based editing from noisy or intermediate diffusion representations.

3. Background

DDMs learn to predict noise for conditional image denoising, while DDIM provides a deterministic sampling and inversion path whose reverse remains approximate for discrete steps. EDICT adapts affine coupling layers by tracking two quantities that can be alternately transformed and exactly recovered.

  • Denoising Diffusion Models: DDMs train a time-aware model to predict noise added to images or autoencoded latents under a monotonic noising schedule.The training objective is MSE between predicted and sampled noise, conditioned on signals such as text embeddings.
  • Denoising Diffusion Implicit Model (DDIM): DDIM deterministically noises an image from x0 to xT, enabling inversion along the reverse of its denoising process.This deterministic process differs from the stochastic DDPM sampling formulation.
  • Denoising Diffusion Models: Classifier-free guidance strengthens conditioning by adding a weighted difference between conditional and unconditional noise predictions.The weighting parameter G controls the contribution of the conditioning signal.
  • Denoising Diffusion Implicit Model (DDIM): DDIM inversion is approximate because it replaces the noise prediction at xt with the prediction at xt−1, reversing an Euler integration step.More sophisticated solvers can stabilize inversion with fewer steps, but reconstruction remains non-exact.
  • Denoising Diffusion Implicit Model (DDIM): Conditional reconstructions can become extremely distorted at high noise levels, forcing guidance downweighting and limiting edit strength.This instability is especially relevant for real-world image editing.
  • Affine Coupling Layers: Affine coupling layers split an input into two halves and alternately transform them, allowing exact recovery of the original input.EDICT parallels this construction by tracking two quantities modified through affine transformations and nonlinear transformations of their counterparts.
  • Affine Coupling Layers: EDICT’s information flow computes adjacent coupled states in either direction, so all steps are invertible rather than approximate.The process can operate while holding only one member of each sequence in memory at a time.

4. Exact Diffusion Inversion via Coupled Transformations (EDICT)

EDICT reformulates diffusion with two coupled latent sequences and invertible mixing, preserving exact reversibility while stabilizing practical generation and inversion.

  • 4.1. Making an Invertible Diffusion Process: EDICT initializes both reverse-process sequences identically and applies alternating coupling updates so each sequence supplies the other’s noise prediction.Only one member from each sequence must be held in memory at a time.
  • 4.3. Complete Summary of the Method: With identical prompts and seeds, full EDICT produces identical images matching DDIM in quality and composition, unlike the no-mixing variant.Without mixing, x and y diverge and produce distorted, inconsistent images.
  • 4.1. Making an Invertible Diffusion Process: EDICT tracks two latent sequences that alternate in updating one another, allowing the sequences to be recovered exactly from adjacent values.This coupling reuses DDIM’s linearization assumption for computation but not for invertibility, which is guaranteed up to machine precision.
  • 4.2. Stabilization: Intermediate invertible averaging layers contract the two sequences after each diffusion step, preventing divergence that otherwise distorts generations.During deterministic noising, the averaging operation becomes a dilating layer.
  • 4.2. Stabilization: p = 0.97 suffices for generative convergence, while sufficiently large mixing coefficients are needed to keep latents aligned throughout inversion.The method typically uses p = 0.93, with [0.9, 0.97] generally effective for 50 steps; low p can cause unstable exponential dilation.
  • 4.3. Complete Summary of the Method: EDICT alternates x and y update order and uses double precision to symmetrize computation and mitigate roundoff errors.The conditioning is implicitly included in the noise-prediction terms.
  • 4.3. Complete Summary of the Method: For editing, EDICT encodes an image into an autoencoder latent, initializes both sequences identically, and deterministically noises them under base-text conditioning.The process runs for s · S steps, where s is editing strength and S is the number of global timesteps.

5. Experiments

Experiments show that EDICT supports diverse, large-scale edits while preserving original image details and unedited components, outperforming sampling-based baselines in visual comparisons.

  • Image Editing: EDICT reconstructs and edits real images using Stable Diffusion, with experiments covering complex object, context, deformation, and style changes.The experiments use Stable Diffusion 1.4 and include qualitative and baseline comparisons.
  • Image Editing: EDICT adds objects to scenes while preserving auxiliary details such as cloud patterns, tree colors, water patterns, and reflections.The examples include interactions between introduced and original objects, including reflected additions.
  • Image Editing: EDICT performs object-preserving context changes, maintaining near-perfect identity and detail even when occlusions and ground changes are introduced.Chair examples retain realistic footing across settings including grass and snow.
  • Image Editing: EDICT performs challenging object deformations and novel-view edits while preserving fine-grained details such as faces, clothing, foliage, and paths.The deformation examples change broad poses and spatial arrangements without requiring the model finetuning used by Imagic.
  • Image Editing: EDICT produces global style changes while generally maintaining layout and details, including preserved cloud patterns and styles such as cubism.Layout can also change when appropriate, such as removing trees in the Mars example.
  • Baseline Comparison: EDICT preserves unedited components better than conditional and unconditional DDIM inversion, Prompt-to-Prompt, and SDEdit while performing requested semantic edits.The baseline comparison uses 50 sampling steps; finetuning- or prompt-tuning-based methods are treated as complementary rather than directly compared.

6. Discussion

The discussion identifies practical trade-offs and ethical risks alongside EDICT’s image-editing capabilities, including determinism, computational cost, input sensitivity, and misuse concerns.

  • Limitations and Future Work: EDICT is deterministic and outputs only one generation per image-prompt pair, unlike stochastic methods such as SDEdit.The paper notes that controllable randomness could enable multiple candidate generations in future work.
  • Limitations and Future Work: EDICT requires approximately twice the computational time of a baseline DDIM process.This is presented as a computational trade-off of the method.
  • Limitations and Future Work: Editing performance can vary unpredictably across inputs and may require careful prompt selection.The paper identifies formalizing prompt selection as a possible direction for improvement.
  • Ethics: Like other image-generation and editing models, EDICT may reflect socioeconomic training-data biases or produce inappropriate images.The paper also warns that image-editing methods can be used for harassment and misinformation.

A. Quantitative Experiment Details

The quantitative editing benchmark evaluates species swaps, contextual changes, and stylistic transformations across five ImageNet mammal classes using semantic and visual similarity metrics.

  • Benchmark Setup: The benchmark uses five ImageNet mammal classes: African Elephant, Ram, Egyptian Cat, Brown Bear, and Norfolk Terrier.The images come from the validation set.
  • Benchmark Setup: Species editing swaps each pictured animal to every other class, producing 20 species-editing pairs.The species-edit prompt is simply “A [animal].”
  • Benchmark Setup: The benchmark includes contextual prompts for an animal in the snow and an animal in a parking lot.Additional listed contextual prompts include wild, shopping-mall, ocean, and football-field settings.
  • Benchmark Setup: The benchmark includes stylistic editing with the prompt “A impressionistic painting of a [animal].”The listed style prompts also include pencil drawing, photography, crayon drawing, and digital rendering.
  • Evaluation: LPIPS measures visual similarity to the original image, while CLIP measures semantic similarity to text queries.For species edits, the five CLIP text queries share the form “A [animal].”
  • Evaluation: Mean and median metrics are reported for each benchmark experiment, with mean-average and median-average values across experiments in Figure S9.The aggregation is described for baseline comparisons.

B. Misalignment of Pseudo-Gradient

The pseudo-gradient of classifier-free guidance is inconsistent across time steps, and the paper identifies this misalignment as driving vanilla DDIM inversion and reconstruction instability.

  • Misalignment of Pseudo-Gradient: The classifier-free guidance pseudo-gradient is inconsistent across time steps, driving instability in vanilla DDIM inversion and reconstruction.The analysis reports similar behavior at higher diffusion step counts.

C.1. Additional Edit Results

Additional figures extend EDICT’s real-image editing demonstrations and test its behavior with substantially more diffusion steps. Results remain largely stable, with some instability appearing in the final row.

  • With 100 and 250 global steps, EDICT shows minimal changes relative to the default 50-step experiments.The experiments use p = 0.9350/S to preserve the original aggregate dilation/contraction factor.
  • Some instability appears in the final row when the experiments are rerun with 100 and 250 global steps.

C.3. Dog Breeds: Extended Results

Extended dog-breed experiments show that EDICT matches target breed prompts while preserving background details that baseline methods erase or alter.

  • EDICT consistently matches the desired dog-breed output while preserving background details that baseline methods erase or alter.The base prompt is “A dog,” and the target prompt is “A [target dog breed].”
  • The two EDICT sequence outputs, x and y, are visually identical, demonstrating convergence of the coupled process.

D. Reconstruction Results

The reconstruction results examine visual metrics, inversion stability, and additional examples across step counts and image-editing settings. They report exact-reconstruction behavior and strong preservation of image content, while conditional DDIM reconstructions remain unstable.

  • EDICT’s reconstruction evaluation includes higher-precision MSEs and errors measured over 1000 steps on the COCO validation set.
  • EDICT achieves Pareto-optimality across all visual-metric experiments, improving on DDIM UC in both metrics in three of four experiments.It also outperforms DDIM UC P2P in CLIP score and DDIM C in LPIPS for those experiments.
  • Conditional reconstructions become increasingly distorted at high noise levels, whereas unconditional reconstructions have relatively insignificant errors.The accompanying analysis attributes DDIM instability to inconsistent classifier-free-guidance pseudo-gradients across timesteps.
Loading 2211.12446v2…