PaperlayerView exact revision on arXiv

Source-linked AI summary

DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models

Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, Jian Zhang

arXiv:2307.02421v2cs.CV

TL;DR

Existing T2I diffusion models generate high-quality images but lack precise editing control. DragonDiffusion uses intermediate image-feature correspondence to create gradient and multi-scale guidance, with visual cross-attention for consistency. It supports multiple drag-style editing tasks without fine-tuning or additional modules.

  • Problem

    Existing large-scale T2I diffusion models generate high-quality images but often lack precise editing ability for generated or real images.

  • Method

    DragonDiffusion transforms image-feature correspondence changes into gradient guidance, adds multi-scale semantic and geometric guidance, and uses visual cross-attention for consistency.

  • Results

    DragonDiffusion performs object moving, resizing, appearance replacement, object pasting, and content dragging without specific training.

  • Takeaways & Limitations

    The method transfers a pre-trained diffusion model's generation ability to diverse image-editing tasks without extra fine-tuning or new blocks.

Abstract

from arXiv · show

Despite the ability of existing large-scale text-to-image (T2I) models to generate high-quality images from detailed textual descriptions, they often lack the ability to precisely edit the generated or real images. In this paper, we propose a novel image editing method, DragonDiffusion, enabling Drag-style manipulation on Diffusion models. Specifically, we construct classifier guidance based on the strong correspondence of intermediate features in the diffusion model. It can transform the editing signals into gradients via feature correspondence loss to modify the intermediate representation of the diffusion model. Based on this guidance strategy, we also build a multi-scale guidance to consider both semantic and geometric alignment. Moreover, a cross-branch self-attention is added to maintain the consistency between the original image and the editing result. Our method, through an efficient design, achieves various editing modes for the generated or real images, such as object moving, object resizing, object appearance replacement, and content dragging. It is worth noting that all editing and content preservation signals come from the image itself, and the model does not require fine-tuning or additional modules. Our source code will be available at https://github.com/MC-E/DragonDiffusion.

1 INTRODUCTION

Large-scale T2I diffusion models generate high-quality images but remain difficult to control precisely for fine-grained editing. DragonDiffusion uses image-feature correspondence and gradient guidance to support diverse drag-style edits without extra fine-tuning or blocks.

  • T2I diffusion models generate images from prompts, but designing prompts that match user intent and enable fine-grained editing remains challenging.
  • Existing editing methods often use GAN latent spaces, while diffusion-based approaches commonly rely on text-image correspondence that can be weak and prompt-dependent.
  • Intermediate diffusion features exhibit strong high-level correspondence across image content, enabling point-to-point correspondence for relevant regions.
  • DragonDiffusion converts changes in image-feature correspondence into gradient guidance through energy functions and uses visual cross-attention to preserve consistency.
  • The method supports object moving, resizing, appearance replacement, object pasting, and content dragging without extra model fine-tuning or new blocks.

2 RELATED WORK

Diffusion models generate images through noise addition and iterative denoising, while conditional scores incorporate external guidance. Related image-editing methods manipulate latent spaces or use text controls, motivating alternative energy-based guidance.

  • 2.1 DIFFUSION MODELS: Diffusion models transform an image into Gaussian noise through repeated perturbations and recover it through reverse denoising steps.
  • 2.1 DIFFUSION MODELS: Text-conditioned diffusion models inject prompts into the denoiser through cross-attention and can be viewed as score functions for sampling.
  • 2.2 ENERGY FUNCTION IN DIFFUSION MODEL: Conditional diffusion scores combine an unconditional denoiser with a conditional gradient derived from an energy function.
  • 2.2 ENERGY FUNCTION IN DIFFUSION MODEL: Energy functions measure compatibility with a target condition and have supported classifier-, sketch-, mask-, universal-guidance, and image-editing applications.
  • 2 RELATED WORK: Prior image-editing methods invert images into StyleGAN latent spaces or use text as control in diffusion-based systems.

3 METHOD

DragonDiffusion edits images by converting feature-correspondence objectives into gradient guidance during diffusion sampling, while using inversion features and attention to preserve unedited content. Its multi-scale design supports semantic and geometric alignment without extra training or modules.

  • Gradient-guidance-based editing design: The method addresses energy-function construction by using high-level correspondence in diffusion UNet features to measure editing targets without training an additional distance model.This converts image-editing objectives into conditional gradients for score-based diffusion.
  • DDIM inversion with memory bank: DDIM inversion converts the input image into zT and stores intermediate latents and attention keys in a memory bank for later editing guidance.Reference-image information is also stored for cross-image tasks such as appearance replacement and object pasting.
  • Energy function via feature correspondence: Editing guidance constrains correspondence between guided and target regions using cosine similarity over intermediate UNet decoder features and separate spatial and appearance terms.Binary masks locate the original and target regions, while feature means provide global appearance representations.
  • Guidance design and sampling: The total guidance combines editing, content-consistency, and optional task-specific energies, with gradient guidance applied only during the first 30 of 50 sampling steps.The content-consistency term compares editing results with the original image in the unedited mask region.
  • Multi-scale guidance: Features from different decoder layers provide complementary guidance: the second layer carries more semantic information, whereas the third carries more geometric information.The method studies these layer roles when reconstructing content from consistency guidance.
  • Application-specific implementation: In object moving, optional inpainting guidance constrains the region where the object was initially located, preventing residual object content there.The reported design uses wi = 2.5 and effectively achieves the editing goal without noticeable artifacts.

4 EXPERIMENTS

Experiments evaluate DragonDiffusion across diverse editing tasks, face manipulation, efficiency, and component ablations. The results show broad editing capability, robustness in complex scenarios, and the importance of the inversion, consistency, and cross-attention components.

  • Applications: DragonDiffusion performs object moving, resizing, pasting, appearance replacement, and content dragging without task-specific training.Objects blend with surrounding content, appearance replacement preserves the original outline, and continuous editing remains stable.
  • Efficiency: 3.62s is required to prepare zT and build the memory bank for one-point dragging at 512 × 512 resolution.The experiment uses an NVIDIA A100 GPU with Float32 precision; inference complexity is reported as acceptable for diffusion generation.
  • Comparisons: On face manipulation, DragonDiffusion provides promising editing accuracy and better robustness and generalization than the compared diffusion and GAN-based methods.DragGAN can be more accurate but shows weaker content consistency and robustness outside faces, while requiring aligned inputs.
  • Comparisons: DragonDiffusion handles complex and unaligned body and face scenarios effectively, whereas DragGAN quality decreases in complex scenarios and depends on image alignment.DragonDiffusion also supports arbitrary image resolutions, unlike the fixed size used by GANs.
  • Ablation Study: Removing inversion, content consistency guidance, or visual cross-attention causes distinct degradation in object-moving results.Random initialization changes the result substantially, removing content guidance causes local distortion, and removing cross-attention weakens consistency.

5 CONCLUSION

DragonDiffusion transfers intermediate feature correspondence in a pretrained diffusion model into gradient guidance for drag-style image editing. Multi-scale guidance and memory-bank-based visual cross-attention support semantic and geometric alignment while preserving consistency across several editing applications.

  • Conclusion: DragonDiffusion models editing as changing feature correspondence and converts editing operations into gradient guidance.The method uses a pretrained diffusion model rather than adding task-specific training or modules.
  • Conclusion: Multi-scale guidance considers both semantic and geometric alignment, while visual cross-attention enhances consistency between original and edited images.The cross-attention strategy reuses intermediate information from inversion with almost no additional cost.
  • Conclusion: DragonDiffusion supports object moving, resizing, appearance replacing, object pasting, and content dragging.These applications are demonstrated for image editing with the proposed diffusion-based pipeline.

A.1 ALGORITHM PIPELINE OF DRAGONDIFFUSION

Algorithm 1 encodes the image, selects an editing task, optionally encodes a reference image, and computes an inversion prior with a memory bank before iterative guided denoising. Each denoising step updates the predicted noise using the energy gradient and then computes the previous latent.

  • Algorithm Pipeline: The pipeline takes a text condition, UNet denoiser, diffusion schedule, editable image, masks, learning rate, and guidance-step count.The masks identify generated, guidance, and shared regions.
  • Algorithm Pipeline: The input image is encoded into z0, an editing task is selected, and a reference image is encoded when the task uses one.Supported task labels include resizing and moving, dragging, pasting, and replacing.
  • Algorithm Pipeline: The method computes the inversion prior zgen_T and builds a memory bank by processing the image through the inversion sequence.Intermediate information is stored for later editing guidance.
  • Algorithm Pipeline: At each denoising step, gradient guidance modifies the predicted noise before the diffusion update produces zt−1.The update uses learning rate η and the energy gradient ∇ztE.

A.2 EFFICIENCY OF THE MEMORY BANK DESIGN

The memory bank stores intermediate inversion information to provide editing guidance and is compared with separate-branch and null-text inversion strategies. The reported design achieves accurate results while maintaining a 3.62-second preparation time.

  • Memory Bank Design: The comparison covers DDIM inversion with a separate branch, null-text inversion with a separate branch, and the proposed memory-bank design.These strategies generate the inversion prior and guidance information in different ways.
  • Memory Bank Design: The memory bank stores intermediate inversion information for subsequent image-editing guidance.It is evaluated against separate generation branches and null-text optimization.
  • Efficiency: 3.62 seconds is the reported time complexity for preparing zT and building the memory bank while maintaining accurate results.Null-text optimization can produce more accurate results but has higher time complexity.

A.3 MORE RESULTS OF DRAGONDIFFUSION ON DIFFERENT APPLICATIONS

DragonDiffusion supports diverse image-editing applications without training or additional modules, preserving content while moving, dragging, pasting, and replacing objects.

  • Object moving: DragonDiffusion performs object moving with good content consistency, including complex scenarios and continuous movement edits.The continuous moving results are described as stable.
  • Content dragging: The method enables natural point-drag editing of image content across different scenarios using several points.
  • Object pasting: DragonDiffusion supports cross-image object pasting by fine-tuning an object in one image before naturally pasting it into another.
  • Appearance replacement: The method replaces appearances effectively for both small objects such as ice-cream and large objects such as cakes.
  • Overall capability: DragonDiffusion performs well across these editing tasks without any training or additional modules.

A.4 MORE QUALITATIVE COMPARISONS BETWEEN OUR DRAGONDIFFUSION AND OTHER METHODS ON CONTENT DRAGGING

Across multiple object and face categories, DragonDiffusion uses a single generalized model and is reported to balance editing accuracy, generation quality, and consistency with the original images.

  • Cross-category generalization: DragonDiffusion, like DragDiffusion, uses Stable Diffusion’s generalization to edit dogs, horses, cars, churches, elephants, and faces with one model.DragGAN requires switching between different models for different categories.
  • Content consistency: Qualitative comparisons report that DragonDiffusion produces better consistency with the original images across these categories.
  • Editing quality: The visual results report that DragonDiffusion balances editing accuracy and generation quality.

A.5 USER STUDY

The user study compares DragGAN, DragDiffusion, and DragonDiffusion across generation quality, editing accuracy, and content consistency using diverse image categories and volunteer votes.

  • Evaluation design: The study evaluates generation quality, editing accuracy, and content consistency across dog, horse, car, elephant, church, and face samples.
  • Evaluation design: Twenty volunteers selected the best-performing method in each of 16 image groups, with votes compiled in Figure 18.
  • Reported outcome: The supplied study passage states that DragonDiffusion has better subjective performance, but does not provide the completed vote breakdown.
  • Methods and categories: Figure 18 compares DragGAN, DragDiffusion, and DragonDiffusion across the six listed categories.
Loading 2307.02421v2…