PaperlayerView exact revision on arXiv

Source-linked AI summary

Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models

Daiki Miyake, Akihiro Iohara, Yu Saito, Toshiyuki Tanaka

arXiv:2305.16807v2cs.CV

TL;DR

Diffusion-based image editing needs faithful reconstructions, but optimization-based approaches can be computationally costly. Negative-prompt inversion reconstructs images using forward computations without optimization, achieving visually comparable quality with a speed-up of over 30 times.

  • Problem

    High-fidelity image editing requires reproducing original-image details, while existing optimization-based approaches incur high computational costs and memory usage.

    Source: §1 · p. 2

  • Method

    Negative-prompt inversion obtains latent variables and text embeddings for real-image reconstruction using forward computations without optimization or backpropagation.

    Source: §1 · p. 2

  • Results

    Over 30 times speed-up, with visually high-fidelity reconstructions comparable to optimization-based inversion methods.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The method provides a practical approach to real-image reconstruction, particularly in high-computational-cost scenarios such as video editing.

    Source: §6 · p. 8

    Average reconstruction quality does not reach null-text inversion, and some cases show significant reconstruction failures.

    Source: §5 · p. 8

Abstract

from arXiv · show

In image editing employing diffusion models, it is crucial to preserve the reconstruction fidelity to the original image while changing its style. Although existing methods ensure reconstruction fidelity through optimization, a drawback of these is the significant amount of time required for optimization. In this paper, we propose negative-prompt inversion, a method capable of achieving equivalent reconstruction solely through forward propagation without optimization, thereby enabling ultrafast editing processes. We experimentally demonstrate that the reconstruction fidelity of our method is comparable to that of existing methods, allowing for inversion at a resolution of 512 pixels and with 50 sampling steps within approximately 5 seconds, which is more than 30 times faster than null-text inversion. Reduction of the computation time by the proposed method further allows us to use a larger number of sampling steps in diffusion models to improve the reconstruction fidelity with a moderate increase in computation time.

1. Introduction

Text-guided image editing depends on faithful reconstruction, but optimization-based methods are costly. Negative-prompt inversion uses forward computations to achieve comparable reconstruction quality at DDIM-inversion speed, enabling fast editing.

1. Introduction

  • Existing image-editing methods often optimize model weights, text embeddings, or latent variables to preserve original-image details, incurring high computational and memory costs.Faithful reproduction of image details is needed before prompt-guided editing can be performed.Source: §1 · p. 2
  • The method obtains reconstruction latents and text embeddings using forward computations alone, without optimization or backpropagation, reducing memory usage.It approximates null-text inversion’s optimized embedding with the conditioning prompt embedding, and editing also works with a source-prompt embedding.Source: §1 · p. 2
  • More than 30-fold increase in processing speed accompanies visually equivalent reconstruction quality; negative-prompt inversion runs at DDIM-inversion speed with accuracy comparable to null-text inversion.The comparison also reports noticeably lower reconstruction accuracy for DDIM inversion and much longer computation time for null-text inversion.Source: §1 · p. 2
  • Combining negative-prompt inversion with methods such as prompt-to-prompt enables ultrafast editing of real images.Text-guided diffusion supports high-quality, diverse changes to image elements such as objects and styles.Source: §1 · p. 2

2. Related work

Prior diffusion-based editing and reconstruction methods trade off fidelity, computation, and editing flexibility; the proposed inversion instead offers near-null-text reconstruction using forward computation and can pair with different editing methods.

Image editing by diffusion models

  • Imagic [15], UniTune [30], and SINE [34] edit image composition, object states, or styles while preserving fidelity through model and/or text-embedding finetuning; prompt-to-prompt [10] uses null-text inversion optimization at each prediction step.These optimization-based approaches typically take several minutes to edit one image.Source: §2 · p. 2
  • Plug-and-Play [29] avoids optimization by using DDIM inversion and inserting attention and feature maps to preserve image structure during prompt-based editing.Source: §2 · p. 2

Image reconstruction by diffusion models

  • Textual Inversion [6] and DreamBooth [24] finetune models for concept acquisition, while ELITE [32] and E4T [7] use encoders to find embeddings but limit editing because corresponding text is not explicitly obtained.The finetuning methods are difficult to use for high-fidelity reconstruction of the original image; encoder methods require less computation.Source: §2 · p. 2
  • Noise map guidance [4] aligns reverse and forward diffusion paths using gradients, while ReNoise [8] improves reconstruction with backward (implicit) Euler inversion; both avoid inference-stage optimization.Source: §2 · p. 2

Contribution

  • The proposed method achieves nearly the same reconstruction as null-text inversion using only forward computation, enabling editing in a few seconds and flexible text-guided editing with methods such as prompt-to-prompt.It is independent of the editing method, so it can be combined with different methods while maintaining high-quality image structure.Source: §2 · pp. 2–3

Novelty

  • The authors identify a similar existing implementation [20] but claim their work is the first to justify the method theoretically and experimentally.Source: §2 · p. 3

3. Method

Negative-prompt inversion reconstructs images without optimization by replacing null-text embeddings with prompt embeddings, enabling forward-only computation; for editing, it uses the source prompt as the negative prompt and the edited prompt as the condition.

3.1. Overview

  • The method targets latent variables that reconstruct a given real image from its prompt, without optimization.The latents are computed along the reverse-diffusion process.Source: §3.1 · p. 3

3.2. DDIM inversion

  • DDIM inversion represents the forward process in terms of εθ(z_t,t,C) by inverting the reverse diffusion process [5] [26], while DDIM inference iteratively calculates z_t to z_{t−1}.Source: §3.2 · p. 3

3.3. Null-text inversion

  • Null-text inversion optimizes a null-text embedding at every diffusion step to preserve reconstruction fidelity under classifier-free guidance, but this optimization is time-consuming.It first computes a latent sequence by DDIM inversion, then sequentially optimizes each step’s embedding to match the target latent.Source: §3.3 · pp. 3–4

3.4. Negative-prompt inversion

  • Negative-prompt inversion replaces optimized null-text embeddings with prompt embeddings, enabling reconstruction using only forward computation without optimization.The approximation is motivated by the expected continuity of the predicted velocity field across nearby diffusion steps.Source: §3.4 · pp. 4–5
  • For editing, negative-prompt inversion uses the source prompt embedding as the negative prompt and the edited prompt embedding as the text condition.This replaces the per-step null-text optimization used by null-text inversion.Source: §3.4 · p. 5

4. Experiments

Experiments show that negative-prompt inversion approaches null-text inversion’s reconstruction quality at over 30× speed, supports real-image editing, and can improve reconstruction further by using more sampling steps at increased runtime.

4.1. Setting

  • Experiments used Stable Diffusion v1.5 on 100 COCO validation images resized to 512×512, usually with 50 sampling steps, evaluating reconstruction by PSNR and LPIPS and editing by CLIP score.Inference speed was measured on an NVIDIA RTX A6000 with an AMD EPYC 7343 CPU.Source: §4.1 · p. 5

4.2. Reconstruction

  • Reconstruction quality was slightly below null-text inversion but far better than DDIM inversion, while negative-prompt inversion ran 30× faster than null-text inversion.It also used approximately half as much memory as null-text inversion.Source: §4.2 · p. 5, Table 1 · p. 6, Figure 3 · p. 6

4.3. Editing

  • Prompt-to-prompt editing with negative-prompt inversion achieved CLIP scores nearly matching null-text inversion and better editing quality than Imagic; overall, it was comparable to null-text inversion.Examples preserved image composition while replacing objects or changing backgrounds, and the inversion method is compatible with CFG-based editing approaches.Source: §4.3 · pp. 5–6, Table 1 · p. 6, Figure 3 · p. 6

4.4. Number of sampling steps

  • 46 seconds at 500 sampling steps remained approximately 3× faster than null-text inversion at 50 steps (130 seconds), while reconstruction quality improved toward null-text inversion.Reconstruction quality also improved with increasing steps, though the improvement was generally imperceptible at first glance and 20 or 50 steps were satisfactory for practical use.Source: §4.4 · pp. 6–7, Figure 4 · p. 7, Figure 5 · p. 7
  • Above 50 sampling steps, negative-prompt inversion better preserved image structure than null-text inversion while achieving comparable CLIP scores; DDIM’s higher CLIP score did not indicate better overall editing.Source: Figure 6 · p. 7, §4.4 · p. 8

5. Limitations

The method’s average reconstruction quality remains below null-text inversion, and some 50-step cases fail markedly despite differences usually being imperceptible at first glance. Editing failures are outside the paper’s scope because the inversion method is independent of editing methods.

5. Limitations

  • The proposed method’s average reconstruction quality does not reach null-text inversion, and some failures are significant despite usually being imperceptible at first glance.The failures shown in Figure 7 occurred at 50 sampling steps, while null-text inversion successfully reconstructed those images.Source: §5 · p. 8
  • Figure 7 shows failures involving people, color gradients reconstructed as separate objects, and a missing tiny object; more sampling steps may improve reconstruction.The authors attribute human reconstruction failures potentially to Stable Diffusion’s AutoEncoder and suggest a more effective encoder-decoder pair could help.Source: §5 · p. 8
  • Post-reconstruction editing failures are beyond the paper’s scope because the inversion method is independent of editing methods.Source: §5 · p. 8

6. Conclusions

Negative-prompt inversion enables optimization-free real-image inversion with high-fidelity reconstruction comparable to optimization-based methods and a speed-up of over 30 times. More sampling steps further improve reconstruction quality while remaining faster than existing methods.

6. Conclusions

  • Over 30 times faster than optimization-based inversion methods, negative-prompt inversion achieves visually high-fidelity reconstruction without optimization, and more sampling steps further improve quality while retaining faster computation than existing methods.The paper presents the method as a practical approach to real-image reconstruction.Source: §6 · p. 8
  • The method is especially beneficial in high-computational-cost applications such as video editing.The conclusion also states that it provides a practical approach for real-image reconstruction.Source: §6 · p. 8
  • Parallelizing multiple GPUs and optimizing the program could increase throughput and reduce latency, potentially enabling real-time processing.The paper states that the approach is available to any user and does not encourage socially inappropriate use.Source: §6 · p. 8

Supplementary Material: Negative-prompt Inversion: Fast Image · A. Justifying arguments · A.1. Theoretical consideration

The theoretical appendix derives DDPM and DDIM from continuous-time diffusion and explains why null-text inversion can ideally reconstruct the forward trajectory by matching unconditional and conditional predictions. It also characterizes the learned DDIM velocity and the role of classifier-free guidance.

A.1. Theoretical consideration

  • The appendix models continuous-time diffusion as a generalized Ornstein–Uhlenbeck process and gives its particle distribution’s evolution through the Fokker–Planck equation.The process uses a positive, integrable time-dependent decay parameter γ_t.Source: §A.1 · p. 11
  • Discretizing the generalized Ornstein–Uhlenbeck process yields DDPM [11, Section 2], with Gaussian states whose mean and variance are determined by α_t.For an initial state z_0, the marginal is N(√α_t z_0, (1−α_t)I).Source: §A.1 · p. 11
  • A different process with the same Fokker–Planck distribution yields DDIM [26] when discretized, with paths that keep the normalized noise component constant over time.The appendix describes these paths as straight halflines starting at z_0 with random velocity d ∼ N(0, I).Source: §A.1 · p. 11
  • Under a prior over z_0, the DDIM velocity learned at position z and time t is the posterior mean of normalized noise and is deterministic and continuously differentiable.Its continuity implies nearby positions and times have approximately equal velocity fields.Source: §A.1 · p. 12
  • For reverse sampling, classifier-free guidance combines conditional and null-text predictions, while null-text inversion optimizes the null embedding at each step to minimize reconstruction error.The procedure first applies conditional DDIM inversion, then runs the guided reverse process while optimizing each step’s null-text embedding.Source: §A.1 · pp. 12–13
  • The appendix extends its DDIM inversion and sampling formulas to prompt conditioning by replacing the unconditioned model prediction with a conditional prediction.The prompt embedding is denoted C.Source: §A.1 · p. 12

Supplementary Material: Negative-prompt Inversion: Fast Image

  • With one sample, guidance scale w ≠ 1, a perfect conditional velocity model, and matched forward and reverse states, reconstruction error is zero iff unconditional and conditional predictions agree.Thus, under these ideal conditions, setting the null-text embedding equal to the prompt embedding achieves exact reconstruction at each step.Source: §A.1 · p. 13

A.2. Empirical evaluations

Across 100 COCO image-prompt pairs, optimized null-text embeddings’ velocity predictions were well approximated by the input prompt embedding, and the optimized embeddings became more semantically similar to that prompt as optimization progressed.

Evaluation setup

  • The evaluation used 100 COCO image-prompt pairs to compare prompt-text embedding C with optimized null-text embedding ∅t under the practical assumption that model learning is only approximate.The paper notes that approximate learning makes the proposition’s perfect-learning assumption too strong for practical situations.Source: §A.2 · p. 13

Velocity prediction comparison

  • Optimized null-text embeddings produced velocity predictions closer to the input prompt than to other prompts, supporting the claim that the input prompt approximates null-text inversion.The gap grew as t became smaller, which the authors attribute to accumulated optimization errors; predictions near t = 0 have little impact because they are added at very small scales.Source: §A.2 · pp. 13–14, Figure 8 · p. 14

Embedding-space comparison

  • As optimization progressed, optimized null-text embeddings became more semantically similar to the input prompt than to other prompts, confirming that the method approximates null-text inversion.Similarity was higher in the region where the model generates image details, while large-t similarity remained near zero because optimization barely progressed after initialization and early stopping.Source: §A.2 · p. 14, Figure 8 · p. 14

B. Implementation details

The implementation uses the published null-text inversion settings as its 50-step baseline, with Adam optimization and step-dependent learning-rate and early-stopping schedules. Scheduling both by diffusion steps was observed to worsen reconstruction quality.

B. Implementation details

  • At 50 sampling steps, null-text inversion used the settings from the GitHub implementation [19], with Adam and a learning rate reaching 5 × 10^-3 at the final step and changing linearly by a factor of 10^-4 with sampling steps.Source: B. Implementation details · p. 14
  • The early-stopping threshold rose linearly with sampling steps from 10^-5 by a factor of 2 × 10^-5, while scheduling the learning rate and threshold by diffusion steps worsened reconstruction quality.Source: B. Implementation details · p. 15

C. Additional experimental results · C.1. Comparison of reconstructed images · C.2. Comparison of edited images using prompt-of-prompt

Under a feasible processing-time constraint, the proposed method reconstructs images better than null-text inversion at fewer than 10 sampling steps, and its reconstructions and prompt-to-prompt edits preserve details close to null-text inversion.

C.1. Comparison of reconstructed images

  • With processing time limited to less than 30 seconds, our method outperformed null-text inversion in LPIPS when null-text inversion used fewer than 10 sampling steps.The compared null-text inversion settings used 2, 5, and 10 sampling steps; the passage reports degraded LPIPS below 10 steps and better reconstruction quality for our method under feasible processing times.Source: §C.1 · p. 15
  • DDIM inversion produced reconstructions that were not similar to the input images, whereas null-text inversion almost perfectly reconstructed them.These qualitative comparisons are shown in Figure 10.Source: §C.1 · p. 15
  • Our method also produced reconstructions close to those from null-text inversion.The comparison of additional reconstructed images is shown in Figure 10.Source: §C.1 · p. 15

C.2. Comparison of edited images using prompt-of-prompt

  • In prompt-to-prompt editing, DDIM inversion failed to preserve original-image details while performing edits.The reported examples are shown in Figures 11 and 12.Source: §C.2 · p. 15
  • Null-text inversion and the proposed method both edited images while maintaining their original details.This held for object replacement, style changes, size changes, and pose changes.Source: §C.2 · p. 15
  • The reported prompt-to-prompt edits included object replacement, style changes, size changes, and pose changes.Both null-text inversion and the proposed method maintained original-image details across these edit types.Source: §C.2 · p. 15

C.3. Comparison of edited images using other editing methods · DDIM Inv

The proposed negative-prompt inversion can be combined with SDEdit and Plug-and-Play; results vary by editing method, with stronger reconstruction reported for SDEdit and in some Plug-and-Play examples. Additional reconstruction and Prompt-to-Prompt results are shown under “DDIM Inv.”

C.3. Comparison of edited images using other editing methods

  • For SDEdit, the methods start from DDIM-inverted noisy samples; for Plug-and-Play, the proposed combination uses the original-image prompt instead of null-text during DDIM inversion.SDEdit uses a noise ratio t0; the sample at t = t0 · T is reconstructed through inverse diffusion, with an edited prompt used for editing.Source: §C.3 · p. 15
  • With SDEdit, negative-prompt inversion and the proposed method recovered input details and edited as prompted, unlike SDEdit’s failure to reconstruct the inputs.Figure 13 shows the SDEdit comparisons.Source: §C.3 · p. 15, Figure 13 · p. 19
  • With Plug-and-Play, the proposed method was not generally better, but improved reconstruction and editing in the first, second, and fourth rows.Figure 14 shows the Plug-and-Play examples.Source: §C.3 · p. 15, Figure 14 · p. 20

DDIM Inv

  • Additional reconstructed-image results for three methods are presented under “DDIM Inv.”These examples appear in Figure 10.Source: Figure 10 · p. 16
  • Captions for Figures 11 and 12 present additional results of images edited by combining Prompt-to-Prompt with three methods.Source: Figure 11 · p. 17, Figure 12 · p. 18
  • The additional SDEdit results are captioned with the prompt “A yellow bird stands perched on a tree branch.”The prompt accompanies the SDEdit examples.Source: DDIM Inv · p. 17, Figure 13 · p. 19

Related papers

Loading 2305.16807v2…