PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code

Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, Qiang Xu

arXiv:2310.01506v2cs.CV

TL;DR

Diffusion-based editing requires inversion that preserves essential source content while maintaining fidelity to a target prompt, yet prior methods jointly optimize both branches under constrained iterations and distribution mismatch. The paper introduces Direct Inversion, which disentangles the branches and directly rectifies the source path while leaving the target branch unchanged. Across eight editing approaches, it improves preservation and fidelity over baselines and achieves nearly an order of speedup over optimization-based inversion methods.

  • Problem

    Prior inversion methods seek a shared latent for source and target branches, but limited optimization can leave distribution gaps that compromise both branches.

  • Method

    Direct Inversion disentangles source and target branches, directly rectifies the source deviation path, and leaves the target branch unchanged.

  • Results

    Across 8 editing approaches, Direct Inversion improves content preservation by up to 20.2% and edit fidelity by up to 2.5% over baseline configurations, with nearly an order of editing speedup over optimization-based methods.

  • Takeaways & Limitations

    PIE-Bench standardizes evaluation with 700 images across 10 editing categories and seven metrics covering essential content preservation and edit fidelity.

  • Takeaways & Limitations

    Prior optimization-based methods often restrict target-latent optimization to only a few iterations because of the generation network’s size and prolonged optimization time.

Abstract

from arXiv · show

Text-guided diffusion models have revolutionized image generation and editing, offering exceptional realism and diversity. Specifically, in the context of diffusion-based editing, where a source image is edited according to a target prompt, the process commences by acquiring a noisy latent vector corresponding to the source image via the diffusion model. This vector is subsequently fed into separate source and target diffusion branches for editing. The accuracy of this inversion process significantly impacts the final editing outcome, influencing both essential content preservation of the source image and edit fidelity according to the target prompt. Prior inversion techniques aimed at finding a unified solution in both the source and target diffusion branches. However, our theoretical and empirical analyses reveal that disentangling these branches leads to a distinct separation of responsibilities for preserving essential content and ensuring edit fidelity. Building on this insight, we introduce "Direct Inversion," a novel technique achieving optimal performance of both branches with just three lines of code. To assess image editing performance, we present PIE-Bench, an editing benchmark with 700 images showcasing diverse scenes and editing types, accompanied by versatile annotations and comprehensive evaluation metrics. Compared to state-of-the-art optimization-based inversion techniques, our solution not only yields superior performance across 8 editing methods but also achieves nearly an order of speed-up.

1 INTRODUCTION

Diffusion-based editing must balance essential content preservation with edit fidelity, but prior unified inversion approaches face optimization, distribution, and runtime problems. Direct Inversion disentangles source and target branches, preserves the target branch, directly rectifies the source path, and improves benchmark performance with substantially lower editing time.

  • 1 INTRODUCTION: Prior optimization-based methods jointly adjust a latent for source and target branches, but limited iterations can leave a gap from the original latent and force inputs outside the pretrained distribution.These issues can compromise the generative capacity of both branches.
  • 1 INTRODUCTION: Direct Inversion disentangles source and target branches: the source branch preserves essential content while the unchanged target branch maintains edit fidelity.The source branch directly rectifies the deviation path, whereas the target branch remains aligned with the target prompt.
  • 1 INTRODUCTION: Direct Inversion outperforms five inversion methods with Prompt-to-Prompt, improving preservation by up to 83.2% in Structure Distance and edit fidelity by up to 8.8% in Edit Region Clip Similarity.It also achieves nearly an order of editing speedup over optimization-based inversion methods.
  • 1 INTRODUCTION: Direct Inversion improves eight editing approaches by up to 20.2% in content preservation and 2.5% in edit fidelity relative to their baseline configurations.PIE-Bench evaluates these dimensions using seven metrics.
  • 1 INTRODUCTION: PIE-Bench provides a standardized benchmark of 700 images across 10 editing categories, with prompts, directives, subjects, masks, and seven evaluation metrics.The benchmark emphasizes both essential content preservation and edit fidelity.

2 RELATED WORK

Diffusion-based editing methods address two objectives: following the editing instruction and preserving essential source content. Related approaches achieve these objectives through end-to-end models, latent integration, inversion variants, overfitting, attention preservation, and source embeddings, while PIE-Bench addresses missing standardized evaluation.

  • 2 RELATED WORK: Diffusion-based editing balances edit fidelity with essential content preservation, using source and target diffusion branches to maintain content and apply instructions.The source branch maintains essential source content while the target branch incorporates the edit.
  • 2 RELATED WORK: Edit fidelity methods include end-to-end editing models and latent integration, which inserts editing instructions through noisy latent features in the target branch.These represent two of the related-work strategies for introducing edits.
  • 2 RELATED WORK: Content-preservation methods include editing-image overfitting, DDPM/DDIM inversion variants, attention preservation, and source embeddings that retain source-image information.These approaches strengthen source influence or reconstruct essential source content during editing.

3 PRELIMINARIES

The preliminaries describe diffusion sampling, DDIM inversion, and classifier-free guidance, then explain why conditioning and existing inversion strategies create reconstruction and editing difficulties.

  • Diffusion Models: Diffusion models map random noise zT through timestep-dependent denoising to an output image or latent z0.Training samples noisy latents from real images, and a denoiser learns to predict the added Gaussian noise.
  • DDIM Inversion: DDIM inversion reverses the diffusion process to obtain a noisy latent for an input image, but finite-step reversal introduces perturbations.These deviations propagate when sampling an image from the inverted latent.
  • Classifier-free Guidance: Classifier-free guidance mixes conditional and unconditional noise predictions using a guidance scale and text condition.The condition is the text-prompt embedding, while the null condition represents an absent prompt.
  • Inversion Challenges: Conditional inputs worsen DDIM inversion quality, motivating techniques that reduce the resulting deviation during editing.Existing approaches include optimization-based methods such as Null-Text Inversion and StyleDiffusion, alongside alternatives with unstable editing outcomes.

4 METHOD

The method argues that optimizing one latent for both diffusion branches is slow and compromises preservation and model integrity. Direct Inversion instead separates the branches, correcting the source path while leaving target generation unchanged, and is integrated into a benchmarkable editing pipeline.

  • Motivation: Unified latent optimization can require 148.48 seconds per image, limits target-latent optimization, and uses inputs outside the pretrained diffusion distribution.These issues can reduce essential content preservation and affect diffusion-model integrity.
  • 4.2 METHOD: Direct Inversion disentangles source and target branches, correcting the source path while leaving the target branch unaltered.This assigns reconstruction to the source branch and preserves the target branch’s editing behavior.
  • 4.2 METHOD: Direct Inversion requires no optimization, adds minimal time overhead, and is designed as a plug-and-play correction for existing editing methods.The algorithm inserts the method into the forward editing process after inversion.
  • T with Direct Inversion: The method updates the latent using the source-prompt difference, retaining the editability of the target prompt’s latent space.The procedure rectifies the deviation path during DDIM forward processing.
  • Benchmark: PIE-Bench provides a standardized evaluation set of 700 images across 10 editing types with prompts, instructions, subjects, and editing masks.The benchmark spans natural and artificial scenes across animal, human, indoor, and outdoor categories.

5 EXPERIMENTS

Experiments evaluate Direct Inversion across inversion and editing methods, efficiency, essential-content preservation, and ablations. Direct Inversion improves the balance between preservation and editability while requiring less inference time than key optimization-based methods.

  • 5.2 COMPARISON WITH INVERSION-BASED EDITING: Direct Inversion improves background and structure retention while improving or maintaining editability across the evaluated editing methods.Table 1 covers Prompt-to-Prompt, MasaCtrl, pix2pix-zero, and Plug-and-Play comparisons.
  • 5.2 COMPARISON WITH INVERSION-BASED EDITING: Direct Inversion achieves better editing results with far less inference time than Null-Text Inversion and StyleDiffusion.Negative-Prompt Inversion and Edit Friendly DDPM are faster, but their editing results are reported as much less acceptable.
  • 5.3 COMPARISON WITH ESSENTIAL CONTENT PRESERVATION METHODS: Direct Inversion simultaneously improves structure and background preservation and editability, unlike preservation methods whose clip similarity decreases.The comparison includes Negative-Prompt Inversion, Proximal Guidance, Edit Friendly DDPM, and EDICT.
  • 5.4.1 COMPARING Direct Inversion AND NULL-TEXT INVERSION: Direct Inversion’s improvement over Null-Text Inversion follows three steps: branch disentanglement, removal of forced null-text assignment, and removal of the distance gap.The ablation varies branch assignment and the added distance to examine their effects on CLIP similarity, structure, and background preservation.
  • 5.4.2 INFLUENCE OF ADDING THE DIFFERENCE TO TARGET LATENT: Adding source distance to the target latent decreases both structure/background preservation and CLIP similarity.This comparison supports adding the source-prompt distance to the source latent instead.

6 CONCLUSION

The paper introduces Direct Inversion to disentangle source and target diffusion branches and separate preservation from edit fidelity. It also introduces PIE-Bench and reports superior edit quality and inference speed across the evaluated methods.

  • 6 CONCLUSION: Direct Inversion separates source-content preservation from target edit fidelity by disentangling the two diffusion branches.The conclusion presents this separation as the technique’s central design principle.
  • 6 CONCLUSION: PIE-Bench contains 700 images in natural and artificial scenes with ten distinct editing types.The benchmark addresses the lack of standardized performance criteria for inversion and editing.
  • 6 CONCLUSION: Direct Inversion outperforms eight editing methods across five inversion techniques in both edit quality and inference speed.The conclusion states this evaluation outcome without reporting individual metric values.
  • 6 CONCLUSION: Existing background-preservation methods suffer from unstable, time-consuming optimization and persistent error-propagation inversion.The paper attributes these issues to prior approaches and contrasts them with Direct Inversion.

B BENCHMARK CONSTRUCTION

PIE-Bench is constructed to provide standardized evaluation for diffusion-based image editing. It combines diverse scenes and editing categories with structured prompts, directives, subjects, and manually annotated masks.

  • B BENCHMARK CONSTRUCTION: PIE-Bench addresses the absence of standardized criteria caused by prior datasets’ lack of manual labels and fine-grained classification.Earlier resources included a 55-image benchmark and a large automatically constructed dataset, but did not support comprehensive metric evaluation.
  • B BENCHMARK CONSTRUCTION: Each benchmark entry includes a source prompt, target prompt, editing directive, edit subjects, and a hand-annotated editing mask.These annotations support more systematic evaluation than subjective visualization alone.
  • B BENCHMARK CONSTRUCTION: PIE-Bench contains 700 images spanning ten editing types across natural and artificial scenes.Editing types include object changes, additions, deletions, pose, color, material, background, style, and volunteer-written random edits.
  • B BENCHMARK CONSTRUCTION: For editing types one through nine, images are evenly distributed across natural and artificial scenes and four scene categories.The listed scene categories include animal, human, indoor environment, and an additional category described in the benchmark construction.

C EVALUAION METRICS

The evaluation measures structure, background preservation, text-image consistency, and inference efficiency. Metrics are computed globally or outside editing masks where appropriate to separate preservation from editing quality.

  • C EVALUAION METRICS: The evaluation covers structure distance, background preservation, edit prompt-image consistency, and inference time.Together, these aspects assess both editing quality and efficiency.
  • C EVALUAION METRICS: Structure distance uses cosine similarity between DINO-ViT self-similarity features to capture structure while ignoring appearance.This metric is suited to edits that should not cause large structural changes.
  • C EVALUAION METRICS: Background preservation uses PSNR, LPIPS, MSE, and SSIM outside manually annotated PIE-Bench editing masks.Computing these metrics outside the masks targets unchanged image regions.
  • C EVALUAION METRICS: CLIP similarity measures consistency between edited images and target prompts for both the whole image and the editing mask.Mask-based scoring blacks out content outside the edited region.
  • C EVALUAION METRICS: Inference time is measured per image for inversion techniques with Prompt-to-Prompt on one NVIDIA A800 80G and averaged over 20 random runs.This protocol evaluates efficiency under a fixed hardware and repeated-run setting.

D IMPLEMENTATION DETAILS

The experiments use a common inference setting and evaluate Direct Inversion through reconstruction, background-preservation comparisons, and model-based editing comparisons.

  • The evaluation fixes the model, sampling steps, guidance scales, hardware, and recommended method-specific hyperparameters to support fair comparison.Experiments use Stable Diffusion v1.4, 50 steps, RTX3090, inverse guidance 1, and forward guidance 0 unless clarified.
  • Reconstruction experiments compare DDIM, Null-Text, Negative-Prompt, StyleDiffusion, and Direct Inversion using the source prompt as model input.
  • Direct Inversion simultaneously improves structure/background preservation and editability, unlike methods whose preservation gains reduce CLIP similarity.The comparison is reported in the full background-preservation table.
  • Direct Inversion with Prompt-to-Prompt achieves better structure and background preservation and higher CLIP similarity than InstructPix2Pix and InstructDiffusion.Against Blended Diffusion, ground-truth masks improve preservation and CLIP similarity but produce editing–background incompatibility and larger Structure Distance.

E.4 INFLUENCE OF GUIDANCE SCALE

Guidance scales create a trade-off between essential-content preservation and edit fidelity, with 2.5 inverse and 7.5 forward guidance providing the best reported balance.

  • 2.5 inverse and 7.5 forward guidance achieve the best balance between essential-content preservation and edit fidelity.This result is reported in the guidance-scale ablation and associated analysis.
  • Increasing forward guidance first improves background preservation and then reduces it, while CLIP similarity increases consistently.The preservation inflection point is approximately where inverse and forward guidance scales are equal.
  • The guidance-scale results expose a trade-off because current editing methods do not accurately distinguish regions requiring modification from regions requiring preservation.
  • Direct Inversion remains robust across 20, 50, 100, and 500 diffusion steps, with smaller step counts favoring structure and background preservation.The supplied passage also indicates larger step counts favor another reported criterion, but the excerpt is truncated.
  • Performance is evaluated across different Direct Inversion add-back intervals, including 1, 2, 5, 10, 24, and 49 steps.

F QUALITATIVE RESULTS

Qualitative experiments compare Direct Inversion across inversion-based, essential-content-preservation, and model-based editing methods to further verify quantitative results.

  • Comparison with different inversion-based editing: Visualization comparisons cover different inversion methods combined with Prompt-to-Prompt and Direct Inversion added to MasaCtrl, Pix2Pix-Zero, and Plug-and-Play.
  • Comparison with essential content preservation methods: Qualitative essential-content comparisons include Proximal Guidance, Edit Friendly DDPM, and EDICT.
  • Comparison with model-based editing: Qualitative model-based comparisons include InstructPix2Pix, InstructDiffusion, and Blended Latent Diffusion.

G LIMITATIONS AND FUTURE WORKS

Direct Inversion inherits limitations from the editing methods it enhances: editing remains unstable, success is not guaranteed, and broader evaluation and applications remain future work.

  • Direct Inversion improves existing methods on average but cannot fundamentally change their unstable editing performance or guarantee success in every case.A selected example shows Blended Latent Diffusion succeeding with a ground-truth mask while other methods fail.
  • Existing diffusion editing algorithms may fail to realize edits without explicit masks, despite the underlying diffusion models having the capability to perform them.
  • Potential misuse includes arbitrary modification of private photos, while training data may transmit personally identifiable information, social biases, and violent content into editing results.
  • Future work includes extending diffusion-based editing to video, improving success rates and scene coverage, and developing more comprehensive evaluation metrics.
Loading 2310.01506v2…