PaperlayerView exact revision on arXiv

Source-linked AI summary

ReNoise: Real Image Inversion Through Iterative Noising

Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Elor, Daniel Cohen-Or

arXiv:2403.14602v1cs.CVcs.GRcs.LGeess.IV

TL;DR

Faithful inversion of real images is challenging because diffusion steps are not invertible, particularly for few-step models. ReNoise iteratively refines and averages inversion estimates; the authors report improved reconstruction quality and editability for fixed computation, alongside text-driven editing on real images.

  • Problem

    Real-image inversion is challenging because diffusion denoising steps are not invertible, and existing methods can degrade reconstruction accuracy or editability for few-step models.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    ReNoise uses fixed-point renoising iterations to refine inversion estimates and averages iterations to mitigate convergence errors.

    Source: §1 · p. 2, Inversion Process as Backward Euler · p. 6

  • Results

    For a fixed amount of computation, ReNoise reports higher reconstruction quality and editability, with effectiveness across diffusion models and sampling algorithms.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The method provides an effective inversion for few-step diffusion models and facilitates editing on those models.

    Source: §6 · p. 10

    Edit Enhancement and Noise Correction require hyperparameter tuning that may vary across models to maintain high reconstruction quality and editability.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent advancements in text-guided diffusion models have unlocked powerful image manipulation capabilities. However, applying these methods to real images necessitates the inversion of the images into the domain of the pretrained diffusion model. Achieving faithful inversion remains a challenge, particularly for more recent models trained to generate images with a small number of denoising steps. In this work, we introduce an inversion method with a high quality-to-operation ratio, enhancing reconstruction accuracy without increasing the number of operations. Building on reversing the diffusion sampling process, our method employs an iterative renoising mechanism at each inversion sampling step. This mechanism refines the approximation of a predicted point along the forward diffusion trajectory, by iteratively applying the pretrained diffusion model, and averaging these predictions. We evaluate the performance of our ReNoise technique using various sampling algorithms and models, including recent accelerated diffusion models. Through comprehensive evaluations and comparisons, we show its effectiveness in terms of both accuracy and speed. Furthermore, we confirm that our method preserves editability by demonstrating text-driven image editing on real images.

LCM-LoRA

Figure 1 presents ReNoise with SDXL Turbo and LCM, comparing it with DDIM inversion and showing prompt-driven image edits.

LCM-LoRA

  • The figure applies ReNoise to SDXL Turbo and LCM, including recent few-step models.It presents the method as applicable across diffusion models.Source: Figure 1 · p. 1
  • The examples compare ReNoise inversion with DDIM inversion.The caption identifies SDXL Turbo and LCM as the models shown.Source: Figure 1 · p. 1
  • The right side demonstrates prompt-driven image edits using the inversions.Source: Figure 1 · p. 1

1. Introduction

Real-image editing requires finding noise that reconstructs the input, but diffusion denoising steps are not invertible and real images may lie outside the model distribution. ReNoise iteratively refines inversion estimates and averages predictions, improving reconstruction and supporting editing across models and samplers.

1. Introduction

  • Inversion seeks Gaussian noise whose denoising reconstructs a real image, but denoising steps are not invertible and real images may fall outside the model distribution.The model predicts the preceding latent at each step, not the inverse step needed for reconstruction.Source: §1 · p. 1
  • ReNoise refines the estimate of each forward-trajectory point by repeatedly renoising and averaging predictions.It starts from the usual reversed-sampling approximation and applies fixed-point iteration.Source: §1 · p. 2
  • Longer inversion strides improve reconstruction and can make the method effective for models trained with few denoising steps.The longer strides yield a more favorable UNet-operation tradeoff despite repeated renoising.Source: §1 · p. 2
  • Experiments cover different samplers and models, including time-distilled models such as SDXL-Turbo, and demonstrate text-driven image manipulation on real images.The authors report effectiveness in reconstruction and inversion speed.Source: §1 · p. 2

2. Related Work

Prior real-image editing methods commonly rely on inversion, while deterministic and non-deterministic approaches face different reconstruction or editability constraints. ReNoise builds on DDIM inversion and fixed-point iteration to address the amplified challenges of few-step models.

2. Related Work

  • Real-image editing commonly uses inversion to obtain a latent whose denoising returns the input, while a parallel process injects features to preserve selected properties during editing.Source: §2 · p. 2
  • Deterministic inversion commonly builds on DDIM; guidance can magnify its accumulated error, motivating methods that optimize or solve for a replacement null-text embedding.The cited methods include, with requiring lengthy optimization.Source: §2 · p. 2
  • DDPM inversion methods construct a noise series that reconstructs the input, but insufficient steps can encode too much information and limit editing, especially for few-step models.Source: §2 · p. 3
  • ReNoise improves DDIM reconstruction and extends fixed-point inversion work toward few-step models, with reported operation across different models and samplers.Earlier fixed-point methods targeted Stable Diffusion without significant computational overhead.Source: §2 · p. 3
  • Few-step diffusion models generate high-quality images in 1–4 steps, but existing inversion methods can degrade reconstruction accuracy or editability in this setting.Standard diffusion models typically require 50 denoising steps.Source: §2 · p. 3

3. Method

ReNoise reverses sampler steps and refines each latent estimate through repeated denoising-based updates, then averages the estimates. To preserve editability alongside reconstruction fidelity, it also regularizes and optimizes noise predictions.

3.1. ReNoise Inversion

  • Because few-step strides make the linearity assumption inaccurate, ReNoise uses the reversed denoising direction only to initialize an estimate and then refines it.Estimates may converge non-monotonically, but averaging them improves estimation of the target.Source: Figure 5 · p. 4, Figure 6 · p. 4
  • Sampler reversal iteratively reconstructs the trajectory from the image; for non-deterministic samplers, the sampled noises are reused during inversion and image generation.Sampler parameters govern the contributions of the previous latent, predicted noise, and additional noise.Source: §3.1 · pp. 3–4
  • The prior approximation assumes enough inversion steps for a near-linear trajectory, limiting interactive use with few-step models and potentially weakening reconstruction for detailed or smooth images and certain prompts.Source: §3.1 · p. 4
  • ReNoise repeatedly applies the denoising model to refine each inversion estimate while keeping z_{t−1} as the step's starting point, then defines z_t as a weighted average of estimates.The iterative estimates are intended to provide a more precise UNet input than z_{t−1} alone.Source: §3.1 · p. 4

3.2. Reconstruction-Editability Tradeoff

  • To enhance editability, ReNoise combines patch-wise KL divergence with a pixel-pair correlation penalty and uses the losses to regularize UNet predictions during renoising.The regularization is applied for renoising iterations with positive weights.Source: §3.2 · p. 5
  • For non-deterministic samplers, directly replacing the sampled noise with a trajectory-bridging value affects editability, so ReNoise instead optimizes that noise using the stated objective.The optimization improves reconstruction fidelity while preserving the noisy-latent distribution.Source: §3.2 · p. 5

4. Convergence Discussion

The section frames ReNoise as fixed-point renoising for the implicit inversion step, then examines convergence analytically and empirically. Its examples and SDXL measurements support convergence in the studied setting, while showing slower convergence at some early noise levels.

Method framing

  • The method treats inversion as backward Euler and uses fixed-point renoising iterations to refine its implicit next-state estimate; averaging mitigates convergence errors and improves reconstruction.The paper identifies the inversion as lacking a closed-form solution.Source: §4 · p. 5, Inversion Process as Backward Euler · pp. 5–6

Renoising toy example

  • In a shifted-Gaussian toy example, one renoising iteration lets the backward Euler step recover the exact pre-image, though this convergence is not guaranteed generally.The paper then discusses sufficient conditions and empirical verification for image diffusion.Source: Inversion Process as Backward Euler · p. 6

Convergence analysis

  • The target noise estimate is a stationary point of mapping G; exponentially decreasing successive-difference norms have a finite sum, supporting convergence of the iteration sequence.The convergence argument assumes G is C2-smooth.Source: Inversion Process as Backward Euler · p. 6
  • For SDXL, scaled Jacobian norms stay below 1 across ReNoise indices; the highest values occur at smaller t (except the first step) and during the initial iteration.The authors use this to support convergence and the strategy of not applying ReNoise in early steps.Source: Inversion Process as Backward Euler · p. 6
  • Figure 7 shows consecutive-estimate distances decreasing exponentially toward G’s stationary point, and the averaging strategy also converges; averages span 32 images and 10 timesteps.Vertical bars indicate standard deviation.Source: Inversion Process as Backward Euler · p. 6, Figure 7 · p. 6

5. Experiments

The experiments evaluate reconstruction quality and editability across four diffusion models and deterministic and non-deterministic samplers, including few-step models.

5.1. Reconstruction and Speed

  • With 100 UNet operations, allocating work to renoising improves reconstruction despite longer strides; fewer denoising steps can also speed repeated editing.The comparison varies inversion, denoising, and renoising steps while holding total operations fixed.Source: Table 1 · p. 8, §5.1 · p. 8
  • At equal UNet-operation counts, additional renoising improves reconstruction more than adding inversion steps, particularly with Ancestral-Euler and LCM samplers.These samplers can degrade when inversion steps exceed denoising steps.Source: §5.1 · p. 8

5.2. Reconstruction and Editability

  • LCM LoRA edits retain original-image details, including background details when replacing an object such as a cat with a koala.The edits replace object words or add descriptive adjectives to the original prompt.Source: §5.2 · p. 8

5.3. Ablation Studies

  • On SDXL Turbo, nine renoising iterations recover finer details, averaging strengthens reconstruction, and Noise Correction mitigates edit-loss degradation in the full method.The ablation also reports minor artifacts from regularizing noise prediction with edit loss.Source: §5.3 · pp. 8–9, Table 2 · p. 9

5.4. Comparisons

  • Against edit-friendly DDPM inversion, ReNoise better reconstructs fine details and can edit substantially without the same difficulty preserving the original image.The comparison highlights shower details and a ginger-cat edit that omits a corner decoration.Source: §5.4 · pp. 9–10
  • With Stable Diffusion, ReNoise achieves results comparable to Null-Text Inversion in 13 seconds, versus 3 minutes, while plain DDIM inversion and NPI each take 9 seconds.The comparison uses one renoising iteration; NPI performs similarly to plain DDIM inversion.Source: §5.4 · p. 10

6. Conclusion

The conclusion presents ReNoise as a trajectory-warping meta-algorithm that improves reconstruction and editability for diffusion inversion, while noting limits in hyperparameter portability and few-step editing evidence.

6. Conclusion

  • ReNoise is a meta-algorithm that warps iterative diffusion inversion trajectories and improves reconstruction quality and editability for a fixed computation budget.The conclusion says averaging the last few renoising iterations significantly enhances reconstruction quality.Source: §6 · p. 10
  • The authors report effectiveness across diffusion models and sampling algorithms, numerical stability, and convergence to an inversion trajectory that eases hyperparameter adjustment.They identify effective inversion for few-step models as a contribution that facilitates editing on those models.Source: §6 · p. 10
  • Edit Enhancement and Noise Correction require model-specific hyperparameter tuning, and advanced editing methods need more extensive testing for few-step models.The hyperparameters may vary across models; the conclusion says no advanced editing has been demonstrated for few-step models.Source: §6 · p. 10

A.1. ReNoise As Contraction mapping

The appendix examines convergence of ReNoise using a scaled Jacobian norm and an averaging argument.

A.1. ReNoise As Contraction mapping

  • The estimated scaled norm stays below 1 across ReNoise indices, supporting practical convergence of the algorithm.The estimates cover noise levels and iteration indices and average over 32 images.Source: §A.1 · p. 13
  • The largest scaled norms occur at smaller noise levels and the first renoising iteration, supporting the strategy of skipping ReNoise in early steps.The initial noise-level norm approaches 0, where ReNoise is nearly an identical operation.Source: §A.1 · p. 13
  • The averaging argument shows that when the renoising sequence converges, its averages converge to the same stationary point, including with unequal weights.The appendix presents the argument using standard calculus.Source: §A.1 · p. 13

B. Implementation Details

The implementation section specifies prompt generation and per-model hyperparameters, and describes configurations used in sampler comparisons.

B. Implementation Details

  • BLIP-2 generates captions for input images, which are then used as diffusion-model prompts.Source: B. Implementation Details · p. 13
  • Table 3 lists per-model ReNoise hyperparameters optimized for the reconstruction-editability trade-off across Stable Diffusion, SDXL, SDXL Turbo, and LCM LoRA.Source: B. Implementation Details · p. 13, Table 3 · p. 14
  • Sampler comparisons hold denoising steps fixed while varying inversion steps, renoising iterations, edit enhancement loss, and Noise Correction configurations.Source: Figure 16 · p. 14

C. Additional Experiments

Additional experiments examine ReNoise editing, its component trade-offs, and inversion stability across image-editing methods and diffusion samplers.

Editing Results With SDXL Turbo

  • SDXL Turbo examples edit images by inverting with a source prompt and changing only a few words in the target prompt during denoising.Figure 15 provides additional image-editing examples.Source: C. Additional Experiments · p. 14

Reconstruction and Speed

  • LPIPS scores follow trends similar to the previously reported PSNR results for the same reconstruction-speed configuration.The additional evaluation is presented in Figure 16.Source: C. Additional Experiments · p. 14

Image Editing Ablation

  • The full SDXL Turbo configuration combines Lpatch-KL, Lpair, and optimization-based noise correction for good reconstruction and editability; overriding noise affects editability.Ledit improves editing but reduces reconstruction accuracy, while the full method uses optimization-based noise correction.Source: C. Additional Experiments · pp. 14–15, Figure 18 · p. 15

Improving DDIM Instabilities

Inversion for Non-deterministic Samplers

  • Against edit-friendly DDPM inversion on SDXL Turbo, ReNoise comparisons highlight cases where DDPM struggles to alter content, preserve details, and maintain edit quality.Examples include difficulty turning a cat ginger while retaining a corner decoration, and lower-quality edits in the dog example.Source: C. Additional Experiments · p. 15, Figure 19 · p. 15

Editing With ReNoise

Loading 2403.14602v1…