PaperlayerView exact revision on arXiv

Source-linked AI summary

DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing

Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent Y. F. Tan, Song Bai

arXiv:2306.14435v6cs.CVcs.LG

TL;DR

Existing drag-based editing offers precise control but is limited by GAN generality, while diffusion editing often lacks comparable spatial precision. DRAGDIFFUSION optimizes diffusion latents using UNet features, adds identity-preserving techniques, and introduces DRAGBENCH; experiments demonstrate versatility and generality, with robustness remaining a limitation.

  • Problem

    GAN-based drag editing is constrained by pretrained GAN capacity, while many diffusion editing methods lack precise spatial control because they manipulate text embeddings.

  • Method

    DRAGDIFFUSION optimizes a diffusion latent at a selected step using UNet feature supervision, with identity-preserving fine-tuning and reference-latent control.

  • Results

    Qualitative and quantitative experiments across diverse examples and DRAGBENCH demonstrate DRAGDIFFUSION’s versatility and generality.

  • Takeaways & Limitations

    DRAGDIFFUSION extends interactive point-based editing to large-scale pretrained diffusion models and provides DRAGBENCH for evaluating such methods.

  • Takeaways & Limitations

    The authors leave making drag-based editing on diffusion models more robust and reliable as future work, because handle points can sometimes miss their targets.

Abstract

from arXiv · show

Accurate and controllable image editing is a challenging task that has attracted significant attention recently. Notably, DragGAN is an interactive point-based image editing framework that achieves impressive editing results with pixel-level precision. However, due to its reliance on generative adversarial networks (GANs), its generality is limited by the capacity of pretrained GAN models. In this work, we extend this editing framework to diffusion models and propose a novel approach DragDiffusion. By harnessing large-scale pretrained diffusion models, we greatly enhance the applicability of interactive point-based editing on both real and diffusion-generated images. Our approach involves optimizing the diffusion latents to achieve precise spatial control. The supervision signal of this optimization process is from the diffusion model's UNet features, which are known to contain rich semantic and geometric information. Moreover, we introduce two additional techniques, namely LoRA fine-tuning and latent-MasaCtrl, to further preserve the identity of the original image. Lastly, we present a challenging benchmark dataset called DragBench -- the first benchmark to evaluate the performance of interactive point-based image editing methods. Experiments across a wide range of challenging cases (e.g., images with multiple objects, diverse object categories, various styles, etc.) demonstrate the versatility and generality of DragDiffusion. Code: https://github.com/Yujun-Shi/DragDiffusion.

1. Introduction

DRAGDIFFUSION extends interactive point-based editing to diffusion models to address limited GAN generality and the lack of precise spatial control in many diffusion-based methods. The paper also introduces DRAGBENCH and reports versatility across diverse editing cases.

  • GAN capacity limits drag-based editing, while prompt-controlled diffusion methods generally lack precise pixel-level spatial control.
  • DRAGDIFFUSION is presented as the first interactive point-based image editing method using diffusion models, improving spatial control and generalizability.
  • The method optimizes a diffusion latent at one appropriately selected step, using UNet features that encode semantic and geometric information for drag-based editing.
  • DRAGBENCH is introduced as the first benchmark dataset for drag-based editing, covering diverse categories, scenes, styles, point instructions, and editable masks.
  • Extensive qualitative, quantitative, and ablation experiments demonstrate versatility and generality while examining identity-preserving fine-tuning, reference-latent-control, and key optimization factors.

2. Related Work

Related work spans GAN-based, prompt-based diffusion, and point-based image editing. DRAGDIFFUSION extends point-based editing to diffusion models, while LoRA provides an efficient fine-tuning strategy used in this setting.

  • Generative Image Editing: GAN-based editing methods face constrained generality because of limited model capacity and difficulty in inverting real images into GAN latents.
  • Generative Image Editing: Most diffusion editing methods manipulate prompts, which usually changes high-level semantics or styles rather than providing precise pixel-level spatial control.
  • Point-based editing: DRAGGAN combines latent-code optimization with point tracking, whereas DRAGDIFFUSION extends the framework to diffusion models across different domains.
  • LoRA in Diffusion Models: LoRA freezes original model weights and injects trainable rank-decomposition matrices into layers for parameter-efficient fine-tuning.

3. Methodology

DragDiffusion edits images by optimizing diffusion latents under user-provided dragging instructions, then denoising them with reference-latent control to preserve coherence with the original image. Its pipeline combines identity-preserving fine-tuning, latent optimization, and guided denoising.

  • Overview: The pipeline has three stages: identity-preserving fine-tuning, latent optimization from dragging instructions, and denoising with reference-latent control.The fine-tuning stage uses LoRA, while latent optimization and guided denoising operate on diffusion latents.
  • Identity-preserving fine-tuning: LoRA fine-tuning adapts the diffusion UNet so its features encode the input image more accurately during editing.The original UNet parameters remain represented by θ, while LoRA parameters ∆θ are optimized by gradient descent using the diffusion noise-prediction objective.
  • Identity-preserving fine-tuning: 80 LoRA optimization steps take around 25 seconds on an A100 GPU and are reported as sufficient for identity-preserving fine-tuning.The paper contrasts this with 1000 steps required for subject-driven image generation.
  • Diffusion latent optimization: DDIM inversion supplies an initial latent at a selected diffusion step, which is repeatedly updated through motion supervision and point tracking.The process stops when all handle points reach their targets or the maximum iteration count is reached.
  • Diffusion latent optimization: Motion supervision uses UNet feature maps to move handle-point features toward corresponding target locations while a mask restricts the editable region and preserves the unmasked area.The unmasked-area term operates on diffusion latents rather than UNet features.
  • Diffusion latent optimization: Point tracking updates handle locations after each latent update by nearest-neighbor search within a local square patch using UNet features.Tracking compensates for handle-point movement caused by motion supervision.
  • Reference-latent-control: Naive DDIM denoising can cause identity shift or quality degradation, so reference-latent control uses the original latent’s denoising process to guide the optimized latent.Replacing attention components with those generated from the original latent improves coherence between original and edited results.

4. Experiments

Experiments evaluate DragDiffusion qualitatively and quantitatively on real and diffusion-generated images, using DRAGBENCH, Image Fidelity, and Mean Distance. Results show broad applicability, strong editing performance, and identifiable effects from inversion steps, identity-preserving fine-tuning, and UNet feature selection.

  • Experimental Setup: Experiments use Stable Diffusion 1.5, DDIM inversion and denoising, latent optimization, and reference-latent-control, with LoRA fine-tuning for identity preservation.The default setup uses 50 DDIM steps, optimizes the latent at step 35, and applies reference-latent-control in UNet upsampling blocks.
  • Benchmark and Metrics: DRAGBENCH provides diverse images and drag instructions containing handle-target point pairs plus masks for editable regions.The benchmark spans varied object categories, indoor and outdoor scenes, and realistic and aesthetic styles.
  • Benchmark and Metrics: Image Fidelity measures similarity to the original image, while Mean Distance measures how accurately semantic content reaches target points.Image Fidelity is computed as 1 minus mean LPIPS; Mean Distance uses DIFT-identified points.
  • Qualitative Evaluation: On real images, DragDiffusion produces reasonable pose manipulation, local deformation, and content filling, while DRAGGAN often struggles across domains.The comparison uses the same user edits, and the pool-filling example illustrates an editing type beyond pose manipulation and local deformation.
  • Qualitative Evaluation: DragDiffusion edits diffusion-generated images across Stable-Diffusion-1.5, Counterfeit-V2.5, Majicmix Realistic, and Interior Design Supermix, handling small and large edits.The results support compatibility with multiple pretrained diffusion-model variants and drag instructions of different magnitudes.
  • Quantitative Analysis: DRAGDIFFUSION significantly outperforms DRAGGAN on both Image Fidelity and Mean Distance, while removing fine-tuning sharply worsens MD and removing reference-latent-control lowers IF.The quantitative comparison averages results over DRAGBENCH; better performance appears toward the upper-left of the MD-IF plane.
  • Ablation Study: Inversion steps show a critical effective range of t ∈[30, 40], with IF decreasing as t increases and MD first decreasing then increasing.Small t limits latent flexibility, whereas overly large t values make the latent unstable.
  • Ablation Study: Identity-preserving fine-tuning initially improves both MD and IF, then plateaus; 80 steps are adopted by default to balance effectiveness and efficiency.The plateau indicates that lengthy LoRA fine-tuning no longer significantly improves performance.

5. Conclusion and Future Works

The paper extends interactive point-based editing to large-scale pretrained diffusion models and introduces DRAGBENCH for systematic evaluation. Its experiments report versatility and generality, while future work targets more robust and reliable drag-based editing on diffusion models.

  • Conclusion: DRAGDIFFUSION extends interactive point-based image editing to large-scale pretrained diffusion models.The method is presented as the first interactive point-based editing method with diffusion models.
  • Conclusion: DRAGBENCH is introduced as a benchmark dataset to facilitate evaluation of interactive point-based image editing methods.The dataset addresses the absence of dedicated benchmarks for this recently introduced task.
  • Conclusion: Comprehensive qualitative and quantitative results demonstrate the reported versatility and generality of DragDiffusion.The conclusion summarizes the experimental evidence supporting the method's scope across evaluated cases.
  • Future Work: Future work will make drag-based editing more robust and reliable on diffusion models.The paper identifies robustness and reliability as remaining objectives.

A. Details About DRAGBENCH Dataset

DRAGBENCH includes diverse images and point-based editing instructions, while generated-image editing uses model-specific handling for latent optimization and classifier-free guidance.

  • Dataset composition: DRAGBENCH contains 205 images and 349 handle-target point pairs across 10 categories, including animals, buildings, humans, interiors, landscapes, and other objects.
  • Generated-image editing: For diffusion-generated images, DRAGDIFFUSION omits LoRA fine-tuning and DDIM inversion because image features and diffusion latents are already available.
  • Generated-image editing: With classifier-free guidance, positive- and negative-prompt UNet feature maps are concatenated along channels to supervise latent optimization.

D. Execution Time

On an A100 GPU, DRAGDIFFUSION completes its main stages within seconds for a 512 × 512 real image, with latent optimization varying by drag magnitude.

  • Stage timing: 25 seconds are required for LoRA fine-tuning on a 512 × 512 real image using an A100 GPU.
  • Stage timing: 10 to 30 seconds are required for latent optimization, depending on the magnitude of the drag instruction.
  • Stage timing: Final Latent-MasaCtrl-guided denoising takes about 1 to 2 seconds and is negligible compared with earlier stages.

E. Limitations

DRAGDIFFUSION occasionally fails to move some handle points precisely to their requested targets, especially in challenging multi-point settings.

  • Observed limitation: Some handle points occasionally cannot precisely reach their desired targets.
  • Possible causes: The authors suggest inaccurate point tracking or latent optimization difficulties with multiple handle-target pairs as possible causes.
  • Future direction: Improving robustness and reliability for drag-based diffusion editing remains future work.

F. More Qualitative Results

Additional experiments report greater versatility than DRAGGAN, generality across additional diffusion models, and applicability beyond 512 × 512 resolution.

  • Comparison with DRAGGAN: Qualitative comparisons in Fig. 13 consistently show DRAGDIFFUSION has better versatility than DRAGGAN.
  • Different diffusion models: Results on Realistic-Vision and DVarch images corroborate DRAGDIFFUSION’s generality across different diffusion models.
  • Higher resolutions: Experiments beyond 512 × 512 resolution further demonstrate the versatility of DRAGDIFFUSION.

G. Visual Ablation on the Number of Identity-preserving fine-tuning steps

The visual ablation shows that identity-preserving fine-tuning with at least 80 steps generally avoids artifacts in dragging results.

  • n ≥80 generally produces reasonable dragging results without artifacts.Fewer than 80 steps can cause distorted faces and scenes, unexpected hands, and other artifacts.

H. Visual Ablation on the UNet Feature Maps

UNet decoder feature-map depth affects the balance between local-detail preservation and editing effectiveness. The third block provides the strongest supported choice among the tested blocks, while additional figures show broader comparisons and resolution results.

  • UNet feature-map ablation: The first UNet block poorly preserves local details because it lacks fine-grained information.This corresponds to low Image Fidelity and high Mean Distance.
  • UNet feature-map ablation: The fourth UNet block produces ineffective editing because it contains only low-level information.Results remain almost the same as the original real image, corresponding to high Image Fidelity and high Mean Distance.
  • UNet feature-map ablation: The second and third UNet blocks yield reasonable editing, with the third preserving local details slightly better.Examples include a more reasonable headwrap and better building details; the third block also achieves lower Mean Distance and higher Image Fidelity.
  • Visual comparisons: Additional comparisons evaluate DRAGGAN and DRAGDIFFUSION under the same drag-based editing instruction on DRAGBENCH images.The figure recommends zooming in to inspect details.
  • Resolution results: DRAGDIFFUSION produces editing results beyond 512 × 512 resolution, including 768 × 512 and 512 × 1024 images.These results use images generated by Counterfeit-V2.5.
  • DRAGBENCH comparisons: DRAGDIFFUSION significantly outperforms DRAGGAN in Image Fidelity and Mean Distance across every DRAGBENCH category.The comparisons average Image Fidelity and Mean Distance over DRAGBENCH samples and also report category-level results.
Loading 2306.14435v6…