PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation

Gwanghyun Kim, Taesung Kwon, Jong Chul Ye

arXiv:2110.02711v6cs.CVcs.AIcs.LG

TL;DR

Existing GAN inversion methods paired with CLIP struggle to faithfully manipulate diverse real images. DiffusionCLIP uses diffusion-model inversion and CLIP-guided fine-tuning to manipulate images across seen and unseen domains, with reported robust performance and multi-attribute editing. The paper also identifies societal risks and control-dependent identity-preservation requirements.

  • Problem

    GAN inversion methods combined with CLIP struggle to reconstruct and faithfully manipulate diverse real images, especially those with novel poses, views, details, or high content variance.

  • Method

    DiffusionCLIP converts input images into latent noise and fine-tunes the reverse-diffusion score function with directional CLIP loss for text-guided manipulation.

  • Results

    DiffusionCLIP performs robust manipulation in trained and unseen domains, including widely varying ImageNet images, and outperforms state-of-the-art baselines in qualitative and human evaluations.

  • Takeaways & Limitations

    Noise combination enables straightforward simultaneous multi-attribute manipulation, while diffusion inversion supports faithful editing across varied image content.

  • Takeaways & Limitations

    The paper notes societal risks and states that identity and pixel-preservation losses depend on the type of control.

Abstract

from arXiv · show

Recently, GAN inversion methods combined with Contrastive Language-Image Pretraining (CLIP) enables zero-shot image manipulation guided by text prompts. However, their applications to diverse real images are still difficult due to the limited GAN inversion capability. Specifically, these approaches often have difficulties in reconstructing images with novel poses, views, and highly variable contents compared to the training data, altering object identity, or producing unwanted image artifacts. To mitigate these problems and enable faithful manipulation of real images, we propose a novel method, dubbed DiffusionCLIP, that performs text-driven image manipulation using diffusion models. Based on full inversion capability and high-quality image generation power of recent diffusion models, our method performs zero-shot image manipulation successfully even between unseen domains and takes another step towards general application by manipulating images from a widely varying ImageNet dataset. Furthermore, we propose a novel noise combination method that allows straightforward multi-attribute manipulation. Extensive experiments and human evaluation confirmed robust and superior manipulation performance of our methods compared to the existing baselines. Code is available at https://github.com/gwang-kim/DiffusionCLIP.git.

1. Introduction

Existing CLIP-guided GAN inversion methods enable zero-shot text-driven manipulation but struggle with faithful reconstruction of diverse real images. DiffusionCLIP addresses this with diffusion-based inversion and CLIP guidance, supporting unseen-domain manipulation and multi-attribute editing.

  • GAN inversion combined with CLIP enables zero-shot image manipulation guided by text prompts.
  • Encoder-based GAN inversion often fails on novel poses, views, details, and highly variable datasets, causing unintended content changes.The paper illustrates failures such as reconstructing a face without an unexpected hand on the cheek.
  • DiffusionCLIP fine-tunes the reverse-diffusion score function with a CLIP loss after converting the input image into latent noise.The approach relies on diffusion models’ inversion and image-generation capabilities.
  • DiffusionCLIP supports manipulation across trained and unseen domains, including translations between two unseen domains and manipulation of widely varying ImageNet images.
  • Combining noise predicted by several fine-tuned models enables simultaneous multi-attribute manipulation in one sampling process.
  • Qualitative comparisons and human evaluation report robust, accurate manipulation that outperforms state-of-the-art baselines.

2. Related Works

Diffusion models use forward noising and reverse denoising processes, while DDIM provides deterministic sampling that enables near-full inversion. CLIP guidance supplies text-driven objectives, including global target matching and local directional alignment.

  • Diffusion Models: Diffusion probabilistic models consist of a forward process that adds noise and a reverse process that generates data from noisy latent variables.
  • Diffusion Models: The forward process gradually adds Gaussian noise, while the reverse model learns to predict noise for denoising transitions.
  • Diffusion Models: DDIM changes the sampling process by controlling noise variance; setting σ_t to 0 makes sampling deterministic and enables full latent inversion with fewer steps.
  • CLIP Guidance: CLIP uses pretrained image and text encoders to measure compatibility between visual content and natural-language prompts.
  • CLIP Guidance: Global CLIP loss minimizes cosine distance between a generated image and a target text, whereas local directional loss aligns image and text embedding changes.
  • CLIP Guidance: Directional CLIP guidance uses reference and target texts to encourage distinct manipulated images and is described as more robust to mode collapse and adversarial attacks.

3. DiffusionCLIP

DiffusionCLIP inverts an input image with diffusion models, fine-tunes the reverse process using directional CLIP and identity losses, and generates text-guided manipulations. Its deterministic sampling and noise-combination strategies support fast manipulation, unseen-domain translation, stroke-conditioned synthesis, and multi-attribute transfer.

  • DiffusionCLIP Fine-tuning: DiffusionCLIP converts an input image into a latent with a pretrained diffusion model, fine-tunes the reverse path using directional CLIP loss, and generates a target-text sample.The pipeline uses deterministic forward-reverse DDIM processes for the core manipulation workflow.
  • DiffusionCLIP Fine-tuning: The fine-tuning objective combines directional CLIP loss with identity loss to guide target attributes while limiting unwanted changes.The identity term can include pixel-level and face-identity losses, depending on the manipulation type.
  • Image Translation between Unseen Domains: The method supports translation between two unseen domains and stroke-conditioned image synthesis by bridging unseen images through a pretrained domain.Source images are perturbed until domain-related components blur while object identity or semantics are preserved, then sampled through the pretrained model.
  • Noise Combination: Combining noise predictions from multiple fine-tuned models changes multiple attributes in one sampling process without fine-tuning a new model for every attribute combination.The same noise-combination strategy also interpolates smoothly between original and manipulated images by mixing pretrained and fine-tuned models.

4. Experiments

Experiments evaluate reconstruction, text-driven manipulation, unseen-domain translation, multi-attribute transfer, and sampling choices. DiffusionCLIP shows strong reconstruction and manipulation results across diverse images and comparisons.

  • Reconstruction: DiffusionCLIP achieves higher face reconstruction quality than pSp, e4e, ReStyle, and HFGI on MAE, SSIM, and LPIPS.The comparison uses quantitative reconstruction metrics against state-of-the-art GAN inversion baselines.
  • Qualitative comparison: DiffusionCLIP is compared with TediGAN, StyleCLIP, and StyleGAN-NADA for text-driven manipulation without target-control images.StyleCLIP-LO and StyleCLIP-GD denote StyleCLIP’s latent-optimization and global-direction variants.
  • Unseen-domain manipulation: DiffusionCLIP successfully manipulates images into unseen domains, whereas StyleCLIP and TediGAN fail in the reported comparison.The method also supports manipulation in pretrained domains.
  • User study: All DiffusionCLIP results were preferred over baselines in general and hard CelebA-HQ cases, with approximately 90% preference against StyleCLIP for out-of-domain manipulation.The study collected 6000 votes from 50 people and used both in-domain and out-of-domain attributes.
  • Diverse real images: DiffusionCLIP manipulates dog-face, bedroom, and ImageNet images, with nearly flawless reconstruction and high-resolution manipulation beyond trained-domain boundaries.ImageNet diversity makes GAN-based inversion and latent-space manipulation limited in the reported setting.
  • Multi-attribute and continuous manipulation: Combining predicted noises enables simultaneous multi-attribute transfer and continuous control of a target attribute’s change degree.The change degree is controlled by γ through mixing noises from original and fine-tuned models.
  • Sampling analysis: Sampling conditions affect reconstruction and manipulation: severe shape changes require larger t0, while skin-color changes can use t0 = 300.The method studies forward steps Sfor, generative steps Sgen, and return step t0 to balance quality and speed.

5. Discussion and Conclusion

The paper concludes that DiffusionCLIP combines pretrained diffusion models with CLIP loss for strong in-domain and out-of-domain manipulation. It also acknowledges limitations and societal risks requiring careful use.

  • Conclusion: DiffusionCLIP uses pretrained diffusion models and CLIP loss to achieve excellent in-domain and out-of-domain image manipulation.The conclusion attributes this performance to the near-perfect inversion property.
  • Conclusion: Combining fine-tuned models with sampling strategies supports several additional image-manipulation applications.The conclusion presents these combinations as novel applications of the method.
  • Limitations and risks: The authors identify limitations and societal risks and advise careful use for proper purposes.Further details are deferred to supplementary sections G and H.

Supplementary Material

The supplementary material describes diffusion-model forward and reverse processes, DDIM sampling, and an ODE interpretation. It also gives the associated noise-prediction and sampling relationships.

  • DDPM: DDPMs model data with forward and reverse processes over latent variables, gradually adding noise forward and denoising from xT toward x0 in reverse.The forward process uses Gaussian transitions under a variance schedule, while the reverse process is parameterized by pθ.
  • DDPM: The forward DDPM process represents xt as a noisy combination of x0 and Gaussian noise under cumulative coefficients ᾱt.The closed-form marginal is given as q(xt|x0) = N(xt; √αtx0, (1 − αt)I).
  • DDIM: The denoising network predicts noise εθ(xt,t), which is used in the reverse update and in estimating the clean sample through fθ.The reverse process includes model-predicted terms and optional Gaussian noise z.
  • DDIM: DDIM provides a non-Markovian process with the same forward marginals as DDPM and a corresponding reverse sampler.Setting reverse noise σt to zero makes sampling deterministic and permits fewer steps.
  • ODE approximation: DDIM updates can be rewritten as an Euler approximation to an ODE, with forward and reverse formulations expressed through εθ.The supplementary derivation connects the discrete update equations to the ODE limit and its reversal.

A.2. Additional Related Works

Diffusion-based manipulation addresses limitations of GAN inversion by supporting text-guided edits across pretrained and unseen domains while preserving image details. DiffusionCLIP also supports multi-attribute manipulation through noise combination.

  • Diffusion models provide image manipulation advantages including broad mode coverage and stable training.
  • Prior diffusion-manipulation studies mainly addressed local editing or translation from unseen domains into trained domains.
  • DiffusionCLIP enables text-guided manipulation across infinitely many text-driven attributes, including translations between unseen domains.
  • GAN-based conditional methods require additional training and manually collected datasets whenever new controls are introduced.
  • GAN inversion methods combine latent manipulation or generator fine-tuning with CLIP loss for text-guided editing without target-distribution training data.
  • Limited GAN inversion causes failures on high-variance datasets and images with novel poses, views, or details, producing unintended changes.
  • DiffusionCLIP uses near-perfect inversion to preserve important details during zero-shot manipulation of high-variance images.
  • It can translate between unseen domains or generate unseen-domain images from strokes.

B. Details on Methods

DiffusionCLIP inverts images with deterministic diffusion, fine-tunes the reverse-process score function using CLIP and identity losses, and then manipulates images through reverse sampling. Additional procedures support unseen-domain translation, multi-attribute editing, and GPU-efficient training.

  • DiffusionCLIP fine-tuning: DiffusionCLIP first inverts real or pretrained-domain images into latent variables before updating a copy of the diffusion model.
  • DiffusionCLIP fine-tuning: The fine-tuned model is updated using CLIP direction and identity losses so reverse-DDIM samples represent the target text while retaining identity.
  • GPU-efficient fine-tuning: GPU-efficient fine-tuning performs loss calculation and gradient updates at each time step, reducing VRAM usage by about half but doubling time.
  • Image manipulation: Manipulation inverts the input with the pretrained model and applies reverse DDIM with the fine-tuned model to produce the manipulated image.
  • Unseen-domain translation: Combining SDEdit with DiffusionCLIP translates images from one unseen domain into another through an intermediate pretrained domain.
  • Multi-attribute transfer: Noise predicted by multiple fine-tuned models can be combined during one sampling process to change several attributes without training a multi-attribute model.
  • Network architecture: The diffusion model uses a U-Net shared across time steps, receiving x_t and t and outputting ϵ_θ(x_t, t).
  • Network architecture: The 256 × 256 architecture contains encoder, middle, decoder, time-embedding, residual, and self-attention components.

D.2. Human Evaluation

The evaluation compares DiffusionCLIP with GAN inversion and diffusion-based baselines using human judgments, quality metrics, and qualitative examples. Results emphasize manipulation of unseen domains, preservation of object identity and background details, and robustness to difficult poses.

  • Human evaluation: The human study collected 6,000 votes from 50 people across general and hard CelebA-HQ cases.
  • Human evaluation: Evaluation covered four in-domain and two out-of-domain attributes using official pretrained checkpoints and implementations.
  • Quantitative evaluation: The quantitative metrics were directional CLIP similarity, segmentation consistency, and face identity similarity.
  • Quantitative evaluation: Segmentation consistency measures pixel accuracy when pretrained-model segmentation of the reference is treated as the label.
  • Quantitative evaluation: The evaluation goal is high attribute correspondence alongside preservation of identity and avoidance of unintended changes.
  • Diffusion-based comparison: Latent optimization and conditional sampling failed on unseen-domain manipulation, whereas DiffusionCLIP was designed to overcome the pretrained distribution restriction.
  • ImageNet comparison: Compared with VQGAN-CLIP on ImageNet, DiffusionCLIP translated images into target styles while preserving object identity and background details.
  • GAN inversion comparison: StyleSpace and InterfaceGAN failed on images with hand poses, indicating practical limitations of their GAN inversion bases.

E. Additional Results

Additional experiments extend DiffusionCLIP to 512 × 512 ImageNet images and translations between unseen domains. The paper also identifies failures associated with limitations of the CLIP encoder.

  • ImageNet results: DiffusionCLIP manipulates 512 × 512 ImageNet images using GPU-efficient fine-tuning with 40 inversion and 12 generation steps.
  • Unseen-domain translation: Unseen-domain translations include animation to Pixar, portrait art to Gogh paintings, and strokes to Neanderthal men without curated source or target datasets.
  • Failure cases: DiffusionCLIP sometimes fails because its manipulations depend on the CLIP encoder’s representations.
  • Failure cases: Face-to-object edits and recently popular target controls may fail when their representations are absent from CLIP.

F. Hyperparameter and Ablation Study

The ablations show that reconstruction quality depends on diffusion-step settings, while identity loss and precomputed-latent count should be selected according to the target manipulation.

  • Dependency on Sfor, Sgen and t0: Reconstruction quality improves as Sfor and Sgen increase, but mismatched settings can cause artifacts.Artifacts occur for (Sfor, Sgen) = (6, 40) and (200, 6).
  • Dependency on Sfor, Sgen and t0: Increasing the return step t0 decreases quality because the intervals between diffusion steps become larger.
  • Identity loss: Face identity loss is recommended for identity-sensitive controls such as Makeup, but it can hinder artistic transfer.
  • Precomputed images: Increasing the number of precomputed images N from 5 to 100 enables stronger image manipulation.
  • Stochastic manipulation: Stochastic DDPM sampling produces varied modifications that can support artistic transfer.

G. Running Time and Resources

DiffusionCLIP trades preparation time and sampling settings against manipulation speed, with fine-tuning taking minutes and larger images requiring substantially more computation.

  • Resources: The latent precomputing procedure requires about 6GB of VRAM, while original and GPU-efficient updating require 23GB and 12GB, respectively.
  • Unseen-domain translation: Unseen-domain translation adds forward DDPM and reverse DDIM processing, taking 2.272–5.098 seconds or 10.446–13.272 seconds depending on sampling settings.
  • Inference: Multi-attribute transfer takes 2.602 seconds with (Sfor, Sgen) = (40, 6) and 14.744 seconds with (Sfor, Sgen) = (200, 40).
  • Running time: Fine-tuning takes 1–7 minutes, compared with 10–12 hours for StyleCLIP's latent mapper and a few minutes for StyleGAN-NADA.
  • Resources: Increasing image size from 256 × 256 to 512 × 512 increases running time fourfold and GPU usage twofold.

H. Societal Impacts

The paper highlights both the accessibility and risks of text-guided manipulation, including misuse, model bias, and qualitative demonstrations across difficult image settings.

  • Societal risks: Simple text prompts enable high-quality image manipulation without professional artistic skills, but realistic results can be used maliciously to confuse people.
  • Model bias: CLIP and diffusion-model biases can introduce biased manipulations and generated results.
  • Model bias: CelebA-HQ-trained generative models tend to produce attractive faces mostly aged 20–40, reflecting dataset-related bias.
  • Human evaluation: Human evaluations compare DiffusionCLIP with StyleCLIP and StyleGAN-NADA on hard and general CelebA-HQ cases.
  • Applications: Additional demonstrations cover church manipulation, ImageNet images including tennis balls and frogs, and translation between unseen domains.
Loading 2110.02711v6…