PaperlayerView exact revision on arXiv

Source-linked AI summary

Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models

Daiki Miyake, Akihiro Iohara, Yu Saito, Toshiyuki Tanaka

arXiv:2305.16807v2cs.CV

TL;DR

Diffusion-based image editing must preserve fidelity to the original image, but existing optimization-based approaches are computationally expensive. This paper proposes negative-prompt inversion, which replaces optimization with forward computation and achieves comparable reconstruction quality with over 30-fold faster processing. The method also supports ultrafast text-guided real-image editing and benefits from increased sampling steps.

  • Problem

    Existing high-fidelity image reconstruction methods optimize parameters such as model weights, text embeddings, or latent variables, causing high computational costs and memory usage.

  • Method

    Negative-prompt inversion obtains reconstruction latent variables and text embeddings using only forward computations, without optimization or backpropagation.

  • Results

    The method achieves visually comparable reconstruction quality to existing optimization-based methods while providing more than 30-fold faster processing.

  • Takeaways & Limitations

    Combining negative-prompt inversion with prompt-to-prompt enables ultrafast real-image editing, including practical use in computationally costly scenarios such as video editing.

  • Takeaways & Limitations

    Average reconstruction quality does not reach null-text inversion’s level, and some images exhibit significant reconstruction failures.

Abstract

from arXiv · show

In image editing employing diffusion models, it is crucial to preserve the reconstruction fidelity to the original image while changing its style. Although existing methods ensure reconstruction fidelity through optimization, a drawback of these is the significant amount of time required for optimization. In this paper, we propose negative-prompt inversion, a method capable of achieving equivalent reconstruction solely through forward propagation without optimization, thereby enabling ultrafast editing processes. We experimentally demonstrate that the reconstruction fidelity of our method is comparable to that of existing methods, allowing for inversion at a resolution of 512 pixels and with 50 sampling steps within approximately 5 seconds, which is more than 30 times faster than null-text inversion. Reduction of the computation time by the proposed method further allows us to use a larger number of sampling steps in diffusion models to improve the reconstruction fidelity with a moderate increase in computation time.

1 The University of Tokyo, Japan 2 DATAGRID Inc., Japan 3 Kyoto University, Japan

Figure 1 compares negative-prompt inversion with DDIM and null-text inversion in reconstruction fidelity and time, and shows editing results using prompt-to-prompt.

  • Figure 1 compares reconstruction fidelity and computation time across negative-prompt inversion, DDIM inversion, and null-text inversion.
  • The proposed method is labeled “Ours” in the comparison.
  • The rightmost column shows image-editing results produced with prompt-to-prompt using the proposed reconstruction.

1. Introduction

The introduction identifies the computational cost of optimization-based high-fidelity reconstruction and presents negative-prompt inversion as a forward-only alternative. The method is reported to match existing reconstruction quality closely while enabling substantially faster reconstruction and editing.

  • Motivation: High-fidelity image editing requires reproducing original-image details before applying text-guided changes.
  • Motivation: Existing approaches optimize model weights, text embeddings, or latent variables, creating high computational costs and memory usage.
  • Proposed method: Negative-prompt inversion obtains latent variables and text embeddings using only forward computations, without optimization or backpropagation.
  • Proposed method: The method approximates null-text inversion’s optimized embedding with the conditioning text prompt’s embedding and can use a source-prompt embedding for editing.
  • Results: More than 30-fold faster processing is achieved while maintaining visually equivalent reconstruction quality to existing methods.
  • Results: Combining negative-prompt inversion with prompt-to-prompt enables ultrafast real-image editing.

2. Related work

Prior diffusion-model editing and reconstruction methods commonly use optimization or additional guidance to preserve fidelity. Negative-prompt inversion instead offers a model-independent inversion component that can be combined with editing methods while retaining image structure.

  • Image editing by diffusion models: Imagic, UniTune, and SINE edit compositional structures, object states, and styles while preserving fidelity through model or text-embedding finetuning.
  • Image editing by diffusion models: Prompt-to-prompt uses null-text inversion, whose per-step null-text optimization typically makes editing take several minutes.
  • Image editing by diffusion models: Plug-and-Play avoids optimization by using DDIM inversion plus attention and feature maps to preserve image structures during editing.
  • Negative-prompt inversion: Its inversion is independent of the editing method, permitting different editing methods while maintaining high-quality image structure.
  • Image reconstruction by diffusion models: Textual Inversion and DreamBooth reconstruct common concepts by finetuning models, whereas ELITE and E4T seek reconstructive text embeddings with encoders.
  • Image reconstruction by diffusion models: Noise map guidance and ReNoise improve non-optimized reconstruction through gradient-based path guidance and backward Euler inversion, respectively.
  • Negative-prompt inversion: Negative-prompt inversion achieves nearly the same reconstruction as null-text inversion using only forward computation, enabling editing in a few seconds.
  • Novelty: Although a similar implementation exists, this work claims the first theoretical and experimental justification of the idea.

3. Method

The method reconstructs real images without optimization by combining DDIM inversion with classifier-free guidance and replacing optimized null-text embeddings with the prompt embedding. This forward-only substitution also supports image editing with source and edited prompts.

  • Overview: Given a real image and prompt, the method computes reverse-process latent variables for high-fidelity reconstruction without optimization.The stated goal is to obtain latent variables that reconstruct the original image using only forward computations.
  • Classifier-free guidance: Classifier-free guidance combines conditional and unconditional predictions, with guidance scale w controlling conditioning strength.The unconditional prediction uses the null-text embedding ∅, while the conditional prediction uses prompt embedding C.
  • DDIM inversion: DDIM inversion reverses the reverse-diffusion update to iteratively obtain latent variables from the original representation.The forward process degrades z0 toward noise, while DDIM inversion expresses that process using the model’s predicted noise field.
  • Null-text inversion: Null-text inversion optimizes a separate null-text embedding at every diffusion step to match DDIM-inverted latent variables, improving reconstruction under CFG.This sequential optimization requires repeated updates at t = T, . . . , 1 and is time-consuming.
  • Image editing: For editing, the edited prompt supplies the text condition while the source prompt replaces the null-text embedding during sampling.This substitution can be combined with an image-editing method such as prompt-to-prompt.
  • Negative-prompt inversion: Negative-prompt inversion replaces each optimized null-text embedding with the prompt embedding, enabling reconstruction through forward computation alone.The method is motivated by approximating the optimized null-text embedding with C when conditional and unconditional predicted velocity fields are approximately equal.

4. Experiments

Experiments show that negative-prompt inversion provides reconstruction quality comparable to null-text inversion while substantially reducing inference time, and supports practical real-image editing with prompt-to-prompt. Increasing sampling steps further improves reconstruction quality while retaining a speed advantage.

  • 4.1. Experimental Setup: Table 1 evaluates reconstruction with PSNR and LPIPS, editing with CLIP score, and inference speed across DDIM inversion, null-text inversion, and the proposed method.The experiments used 100 square COCO validation images resized to 512×512 and compared methods under standardized sampling settings.
  • 4.2. Reconstruction: Negative-prompt inversion achieved reconstruction quality slightly below null-text inversion but far above DDIM inversion, while running 30 times faster than null-text inversion.The speedup results from avoiding null-text inversion’s iterative optimization and backpropagation.
  • 4.2. Reconstruction: Negative-prompt inversion reconstructed images nearly identically to the inputs, with quality comparable to null-text inversion and visibly better fidelity than DDIM inversion.DDIM inversion produced noticeable differences in object position and shape.
  • 4.3. Editing: The proposed method and null-text inversion achieved almost identical CLIP scores for prompt-to-prompt editing, making the proposed editing quality comparable when reconstruction fidelity is considered.The method is compatible with CFG-based editing approaches and was also evaluated with prompt-to-prompt and Imagic.
  • 4.3. Editing: Prompt-to-prompt combined with negative-prompt inversion enabled ultrafast real-image editing while preserving input-image structure.Figure 3 shows reconstruction and editing examples in which the edited images preserved the structure of the input images.
  • 4.4. Number of sampling steps: Increasing sampling steps from 50 toward 500 further improved reconstruction quality toward null-text inversion, while 500 steps still took 46 seconds and remained approximately three times faster than null-text inversion with 50 steps.The total execution time scales roughly with execution time per sampling step multiplied by the number of steps.
  • 4.4. Number of sampling steps: Reconstruction became more detailed as sampling steps increased, but differences were generally imperceptible at first glance, making 20 and 50 steps sufficiently satisfactory for practical use.Objects and composition were recovered even with 20 steps, while finer details improved with additional steps.
  • 4.4. Number of sampling steps: Editing quality was evaluated with LPIPS for structure preservation and CLIP scores for agreement with editing prompts as sampling steps varied.Figure 6 reports both metrics with 95% confidence-interval shadings.

5. Limitations

The proposed method’s average reconstruction quality does not reach null-text inversion’s level, and some images fail substantially at 50 sampling steps. Post-reconstruction editing failures are outside the paper’s scope.

  • 5. Limitations: The proposed method’s average reconstruction quality remains below that of null-text inversion, despite generally imperceptible differences at first glance.Some individual reconstructions nevertheless failed significantly.
  • 5. Limitations: At 50 sampling steps, failure cases included missing or altered people, color gradients reconstructed as separate objects, and disappearance of tiny objects.Increasing sampling steps can improve reconstruction in at least some cases, while human failures may relate to Stable Diffusion’s AutoEncoder.
  • 5. Limitations: Failures in post-reconstruction image editing are outside the paper’s scope because the inversion method is independent of the editing methods.

6. Conclusions

Negative-prompt inversion enables optimization-free real-image inversion with high-fidelity reconstruction and more than 30-fold faster processing. More sampling steps can further improve reconstruction while retaining faster computation than existing methods.

  • 6. Conclusions: Negative-prompt inversion produced visually high-fidelity reconstructions comparable to optimization-based methods while achieving a speed-up of over 30 times.
  • 6. Conclusions: Increasing the number of sampling steps further improved reconstruction quality while maintaining faster computational time than existing methods.
  • 6. Conclusions: The method provides a practical approach for real-image reconstruction, especially in high-computational-cost scenarios such as video editing.Parallelizing multiple GPUs and optimizing the program could potentially increase throughput and reduce latency.

Supplementary Material: Negative-prompt Inversion: Fast Image

The appendix develops a continuous-time account of DDPM and DDIM, then uses it to explain the velocity field and the proposed conditioning-based inversion argument.

  • A.1. Theoretical consideration: The generalized Ornstein–Uhlenbeck stochastic differential equation defines the continuous-time diffusion process and its Fokker–Planck equation.The decay parameter γ_t is positive, deterministic, and integrable; constant γ_t gives the ordinary OU process.
  • A.1. Theoretical consideration: The conditional particle distribution induced by the continuous-time process is Gaussian with mean √(α_t/α_s)z_s and covariance (1−α_t/α_s)I.Discretizing this process in time yields the DDPM formulation.
  • A.1. Theoretical consideration: An alternative process with the same marginal distribution yields DDIM when discretized, using a prescribed terminal Gaussian and an explicit trajectory.Its intermediate position is determined from z_T and z_0 through the α_t schedule.
  • A.1. Theoretical consideration: DDIM paths are straight halflines z_t=√α_t z_0+√(1−α_t)d because the normalized noise component d_t remains constant across time.The random velocity d follows the standard Gaussian distribution N(0,I).
  • A.1. Theoretical consideration: The model estimates the velocity from z_t, obtains an estimate of z_0, and uses the time-invariant normalized noise to estimate z_s.The resulting general update reduces to the standard DDIM recurrence when s=t±1.
  • A.1. Theoretical consideration: With z_0 drawn from p(z_0), the learned velocity field is the posterior mean of the normalized noise under the induced prior and likelihood.Proposition 1 expresses this posterior mean as a Gaussian-density-weighted expectation over z_0.
  • A.1. Theoretical consideration: The posterior-mean expression follows from the joint distribution of z_0 and z and proves the stated velocity-field formula.The joint density is obtained by integrating the deterministic likelihood against p(z_0) and the standard Gaussian noise density.
  • A.1. Theoretical consideration: The velocity field is deterministic and continuously differentiable, so nearby positions and times have approximately similar velocity predictions.Conditioning is incorporated by replacing the unconditional model with ϵ_θ(z_t,t,C); the null-text inversion condition is exact when conditional and unconditional predictions match.

A.2. Empirical evaluations

Empirical evaluations test whether the optimized null-text embedding can be approximated by the input prompt embedding, supporting the proposed inversion method.

  • A.2. Empirical evaluations: The practical analysis notes that perfect model learning is too strong, so the theoretical equality is expected to hold only approximately.The experiments use 100 COCO image-prompt pairs to compare optimized null-text and prompt embeddings.
  • A.2. Empirical evaluations: The velocity comparison measures distances between predictions from optimized null-text, input-prompt, and other-prompt embeddings.The comparison is performed along the latent sequences produced by null-text inversion.
  • A.2. Empirical evaluations: Optimized-null-text velocity predictions were closer to input-prompt predictions than to other-prompt predictions.The input-versus-other distance was smaller than the optimized-null-text-versus-input distance near t=0, where errors accumulate but updates have small scales.
  • A.2. Empirical evaluations: Figure 8 compares mean velocity L1 distances on the left and embedding similarities on the right, with separate curves for input and other prompts.The solid, dashed, and dotted curves encode optimized-versus-input, optimized-versus-other, and input-versus-other similarities.
  • A.2. Empirical evaluations: The optimized embedding becomes semantically similar to the input prompt as optimization progresses, indicating that negative-prompt inversion approximates null-text inversion.Similarity is higher away from t=T, excluding the region near t=0, where detailed image information is considered important for reconstruction.

B. Implementation details

The implementation comparison fixes null-text inversion at 50 sampling steps and evaluates reconstruction quality under time constraints.

  • B. Implementation details: Null-text inversion uses 50 sampling steps, Adam optimization, scheduled learning rates, and early stopping.The learning-rate and stopping-threshold schedules are specified as functions of sampling steps.

C. Additional experimental results

Additional experiments show that the proposed method reconstructs images and supports diverse prompt-to-prompt edits while preserving image details.

  • C.1. Comparison of reconstructed images: Under a 30-second limit, the proposed method outperformed null-text inversion in LPIPS when null-text inversion used fewer than 10 sampling steps.Null-text inversion quality degraded below 10 sampling steps.
  • C.1. Comparison of reconstructed images: Additional reconstructions from the proposed method were close to null-text inversion, while DDIM inversion was not similar to the input images.Null-text inversion almost perfectly reconstructed the inputs in these examples.
  • C.2. Additional edited images: Null-text inversion and the proposed method maintained original-image details across object replacement, style, size, and pose edits.DDIM inversion failed to preserve details during editing in the reported examples.

C.3. Comparison of edited images using other editing methods

The proposed method was evaluated in combination with SDEdit and Plug-and-Play for image editing. It reconstructed input details for SDEdit and improved some Plug-and-Play results, though it was not generally better there.

  • Comparison with other editing methods: The proposal was combined with SDEdit and Plug-and-Play to test compatibility with other image-editing methods.The experiments specifically examined whether the proposed inversion method could support these two editing pipelines.
  • SDEdit: With SDEdit, negative-prompt inversion and the proposed method reconstructed input details while editing images according to the prompts.SDEdit alone could not reconstruct the input images in the reported comparison.
  • Plug-and-Play: With Plug-and-Play, the proposed method was not generally better, but it produced better reconstruction quality and editing results in the first, second, and fourth rows.The comparison is shown in the additional Plug-and-Play editing results.
  • Additional visual comparisons: The paper includes additional reconstructed-image results for the three methods and additional prompt-to-prompt editing results combined with those methods.These supplementary figures extend the visual comparisons across reconstruction and prompt-to-prompt editing settings.
  • Additional visual comparisons: Additional SDEdit and Plug-and-Play visual results show comparisons involving the proposed method.The SDEdit figure compares null-text inversion or the proposed method, while the Plug-and-Play figure uses the proposed method.
Loading 2305.16807v2…