PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation

Gwanghyun Kim, Taesung Kwon, Jong Chul Ye

arXiv:2110.02711v6cs.CVcs.AIcs.LG

TL;DR

GAN-inversion approaches can struggle to reconstruct real images with novel poses, views, and details, making faithful text-guided manipulation difficult. DiffusionCLIP uses pretrained diffusion models and CLIP loss for text-guided edits, with reported manipulation across trained and unseen domains and general images.

  • Problem

    Encoder-based GAN inversion approaches can fail to reconstruct images with novel poses, views, and details, a challenge for faithful manipulation of diverse real images.

    Source: §1 · p. 2

  • Method

    DiffusionCLIP fine-tunes pretrained diffusion models using CLIP loss for text-guided image manipulation.

    Source: §5 · p. 8

  • Results

    DiffusionCLIP manipulated trained and unseen domains, enabled multi-attribute edits and ImageNet images, and had preference rates above 50% over baselines in general and hard cases.

    Source: §1 · p. 2, §4.1 · p. 8, §4.2 · p. 8

  • Takeaways & Limitations

    The paper presents DiffusionCLIP as a step toward general text-driven manipulation of images beyond trained domains.

    Source: §4.2 · p. 8

    The authors acknowledge limitations and societal risks, but defer their details to supplementary sections.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recently, GAN inversion methods combined with Contrastive Language-Image Pretraining (CLIP) enables zero-shot image manipulation guided by text prompts. However, their applications to diverse real images are still difficult due to the limited GAN inversion capability. Specifically, these approaches often have difficulties in reconstructing images with novel poses, views, and highly variable contents compared to the training data, altering object identity, or producing unwanted image artifacts. To mitigate these problems and enable faithful manipulation of real images, we propose a novel method, dubbed DiffusionCLIP, that performs text-driven image manipulation using diffusion models. Based on full inversion capability and high-quality image generation power of recent diffusion models, our method performs zero-shot image manipulation successfully even between unseen domains and takes another step towards general application by manipulating images from a widely varying ImageNet dataset. Furthermore, we propose a novel noise combination method that allows straightforward multi-attribute manipulation. Extensive experiments and human evaluation confirmed robust and superior manipulation performance of our methods compared to the existing baselines. Code is available at https://github.com/gwang-kim/DiffusionCLIP.git.

1. Introduction

DiffusionCLIP uses CLIP-guided fine-tuning of a diffusion model’s reverse-process score function for robust text-driven image manipulation. It supports unseen-domain and multi-attribute edits and is reported to outperform state-of-the-art baselines in qualitative comparisons and human evaluation.

1. Introduction

  • CLIP-guided GAN inversion enables zero-shot text-guided manipulation [1–4,7,45,55,41,20,39], but limited inversion performance makes applications to diverse real images difficult.Encoder-based approaches can fail on novel poses, views, and details, changing input content or producing artificial architectures on high-variance images.Source: §1 · pp. 1–2
  • Diffusion models provide high-quality image synthesis, while DDIM accelerates sampling and enables nearly perfect inversion.The cited latest diffusion works are described as achieving higher-quality synthesis than VAEs, flows, autoregressive models, and GANs.Source: §1 · p. 2
  • DiffusionCLIP forward-diffuses an input into latent noise, then fine-tunes the reverse-process score function using CLIP loss to guide generated attributes from text prompts.With DDIM, retaining the forward score function for reverse diffusion enables nearly perfect inversion to the original image.Source: §1 · p. 2
  • DiffusionCLIP manipulates images across trained and unseen domains, translates between two unseen domains, generates from strokes, and changes multiple attributes in one sampling process.It also manipulates widely varying ImageNet images, a setting rarely explored with GAN inversion because of its poor reconstruction.Source: §1 · p. 2
  • Qualitative comparisons and human evaluation show robust, accurate manipulation that outperforms state-of-the-art baselines.The paper also proposes a systematic approach for finding sampling conditions that balance image quality and speed.Source: §1 · p. 2

2. Related Works

The related work describes diffusion models as forward-noising and learned reverse-generation processes, with deterministic DDIM sampling enabling image inversion. It also presents CLIP-based text guidance through global text-image matching and local directional alignment.

2.1. Diffusion Models

  • Diffusion models gradually add Gaussian noise in a forward Markov chain and learn a reverse process to generate data; deterministic DDIM sampling enables full latent inversion with fewer steps.Setting the sampling noise to zero makes DDIM deterministic.Source: §2.1 · pp. 2–3

2.2. CLIP Guidance for Image Manipulation

  • A pretrained CLIP text and image encoder supports text-driven manipulation, while global guidance minimizes cosine distance between a generated image and target text in CLIP space.The global loss is described as susceptible to low diversity and adversarial attacks.Source: §2.2 · p. 3
  • Local directional CLIP guidance aligns the source-to-generated image embedding direction with the reference-to-target text direction, addressing global loss’s low diversity and adversarial susceptibility.The passage says directional guidance is robust to mode collapse and adversarial attacks because perturbations differ across images.Source: §2.2 · p. 3

3. DiffusionCLIP

DiffusionCLIP inverts an input with a pretrained diffusion model and fine-tunes reverse diffusion under directional CLIP guidance. It also enables unseen-domain manipulation and attribute mixing by combining noises from fine-tuned models.

3. DiffusionCLIP

  • The method converts an input image to a diffusion latent, then fine-tunes the reverse process under directional CLIP guidance to generate a target-text-driven sample.The deterministic forward-reverse processes use DDIM; forward DDPM is also used for translation between unseen domains.Source: §3 · p. 3, Figure 2 · p. 3

3.1. DiffusionCLIP Fine-tuning

  • Fine-tuning the reverse diffusion model with directional CLIP and identity losses guides the target-text change while limiting unwanted changes to the object.Identity preservation needs vary by control: expression or hair-color edits call for pixel or face-identity preservation, while artwork or species changes favor larger shape and color changes.Source: §3.1 · pp. 3–4

3.2. Forward Diffusion and Generative Process

  • Deterministic forward and reverse processes support image manipulation, while stochastic DDPM sampling can help translation between unseen domains.A fast strategy stops forward diffusion at return step t0<T; for T=1000, t0=300–600 and (Sfor, Sgen)=(40, 6) are reported as suitable choices.Source: §3.2 · p. 4

3.3. Image Translation between Unseen Domains

  • DiffusionCLIP supports translation between two unseen domains and stroke-conditioned synthesis in an unseen domain by bridging through a pretrained domain.The source image is perturbed with forward DDPM until domain-related components blur while object identity or semantics remain; this is usually set to 500 steps.Source: §3.3 · pp. 4–5

3.4. Noise Combination

  • Combining noises from separately fine-tuned models enables multi-attribute manipulation in one sampling process and smooth interpolation between original and manipulated images.Weights control each attribute’s degree, avoiding new fine-tuning for combinations of target attributes.Source: §3.4 · p. 5

4. Experiments

DiffusionCLIP is evaluated for reconstruction and text-driven manipulation across faces, dogs, bedrooms, churches, general images, and unseen domains. The experiments report strong reconstruction and manipulation results, including user preference over baselines, broad-domain applications, and flexible control through noise combination.

4.1. Comparison and Evaluation

  • DiffusionCLIP achieved higher face-reconstruction quality than pSp, e4e, ReStyle, and HFGI across MAE, SSIM, and LPIPS.The comparison is reported in Table 1.Source: Table 1 · p. 5, §4.1 · p. 6
  • DiffusionCLIP results were preferred over baselines in both general and hard face-manipulation cases (> 50%), with ≈90% preference against StyleCLIP for out-of-domain manipulation.The study collected 6,000 votes from 50 participants; preference rates increased for DiffusionCLIP in hard cases.Source: Table 2 · p. 5, §4.1 · p. 8
  • In qualitative comparisons, GAN-inversion baselines distort novel poses and details or change church identity, while DiffusionCLIP reconstructs details and backgrounds and manipulates into unseen domains.StyleCLIP and TediGAN fail on the reported unseen-domain manipulation examples.Source: Figure 5 · p. 6, §4.1 · p. 7
  • DiffusionCLIP outperformed baselines on Directional CLIP similarity (Sdir), segmentation-consistency (SC), and face identity similarity (ID) for the evaluated face and church attributes.The results indicate attribute correspondence and preservation of identities without unintended changes.Source: Table 3 · p. 7, §4.1 · p. 8

4.2. More Manipulation Results on Other Datasets

  • On dog, bedroom, and ImageNet general images, DiffusionCLIP produced nearly flawless reconstructions and flexible high-resolution manipulations beyond trained-domain boundaries.The paper notes limited performance for GAN-based inversion and latent-space manipulation on diverse ImageNet images.Source: Figure 6 · p. 6, §4.2 · p. 8

4.3. Image Translation between Unseen Domains

  • DiffusionCLIP translated between unseen domains without collecting source- or target-domain images or using external models, including from rough strokes to unseen-domain images.Examples include portrait artworks and animations translated to Pixar, Gogh-style paintings, and Neanderthal men.Source: Figure 7 · p. 7, §4.3 · p. 8

4.4. Noise Combination

  • Noise combination enabled multi-attribute transfer in one sampling and controlled the degree of a single-attribute change through γ.The degree is controlled by mixing noise from the original and fine-tuned models; the reported approach avoids complex loss designs and extensive manual data collection.Source: Figure 8 · p. 7, Figure 9 · p. 7, §4.4 · p. 8

4.5. Dependency on Hyperparameters

  • Reconstruction depended on Sfor and Sgen, while larger t0 values were needed for severe shape changes than for skin-color changes.The selected settings were (Sfor, Sgen) = (40, 6) for training and (200, 40) for inference; t0 = 300 sufficed for skin-color changes, while Pixar-style shape changes required t0 = 500 or 700.Source: Figure 10 · p. 7, Figure 11 · p. 7, §4.5 · p. 8

5. Discussion and Conclusion

DiffusionCLIP uses pretrained diffusion models and CLIP loss for text-guided image manipulation, with fine-tuning supporting in-domain and out-of-domain manipulation. The paper also presents sampling-strategy applications and cautions that the method has limitations and societal risks.

5. Discussion and Conclusion

  • DiffusionCLIP combines pretrained diffusion models with CLIP loss for text-guided image manipulation, using fine-tuning and near-perfect inversion for in-domain and out-of-domain manipulation.The authors report excellent performance in both settings.Source: §5 · p. 8
  • The paper presents applications of fine-tuned models that combine various sampling strategies.Source: §5 · p. 8
  • The authors identify limitations and societal risks and advise users to apply DiffusionCLIP carefully for proper purposes.Further discussion is in Supplementary Sections G and H.Source: §5 · p. 8

Supplementary Material · A. Details on Related Works · A.1. DDPM, DDIM and ODE Approximation

This supplementary section outlines DDPM’s forward and reverse diffusion processes, then explains how DDIM enables deterministic sampling and relates to an ODE approximation.

A.1. DDPM, DDIM and ODE Approximation

  • DDPM [23] gradually adds noise to data through a variance-scheduled Markov chain, ending at a standard Gaussian latent.Its reverse process denoises that latent toward the data.Source: §A.1 · p. 12
  • The DDPM forward marginal has a closed-form Gaussian distribution conditioned on the original sample.Its mean is √ᾱ_t x_0 and its covariance is (1 − ᾱ_t)I.Source: §A.1 · p. 12
  • DDPM’s reverse transitions are Gaussian, and its noise-prediction network is trained with the improved objective [23].The objective minimizes squared error between injected noise and the network’s prediction.Source: §A.1 · p. 12
  • DDIM [52] uses an alternative non-Markovian forward process with the same marginals as DDPM and a corresponding reverse sampler.The reverse update uses a prediction of x_0 and a variance-controlled noise term.Source: §A.1 · p. 13
  • Setting DDIM’s reverse-noise variance to zero makes sampling deterministic, enabling consistent latent-to-data conversion and sampling with fewer steps.Changing the variance allows different reverse samplers.Source: §A.1 · p. 13
  • DDIM can be viewed as an Euler method for an ODE; in the small-step limit, reversing that ODE yields the forward DDIM formulation used in Section 3.2.The supplementary derivation states that the reversed ODE is equal to that forward formulation.Source: §A.1 · p. 13

A.2. Additional Related Works

DiffusionCLIP extends diffusion-based image editing to broad text-guided attributes and unseen-domain translation, while addressing the scope limits of prior diffusion methods and the training and inversion challenges of GAN-based approaches.

Diffusion-based image manipulation

  • Diffusion models offer strong image generation, broad mode coverage, and stable training [16] [23] [26] [51] [52] [53] [54].Source: §A.2 · p. 13
  • Prior diffusion editing studies [9] [33] addressed local edits or translation from unseen to trained domains, but their extension to broader manipulation was unclear.ILVR [9] conditions low-frequency reference-image components during sampling; SDEdit [33] noises user edits or strokes before reverse-SDE denoising.Source: §A.2 · p. 13
  • DiffusionCLIP supports infinitely many text-driven attributes and translates images from pretrained or unseen domains to another unseen domain.It can also generate images in an unseen domain from strokes.Source: §A.2 · pp. 13–14

GAN-based image manipulation

  • GAN inversion methods [3] [4] [7] [22] [45] [46] [55] [58] [61] [69] [70] can manipulate latent codes or fine-tune generators, and CLIP-based variants [20] [39] enable text-prompt editing without target-distribution training data.Source: §A.2 · p. 13
  • GAN inversion often fails on novel poses, views, details, or high-variance datasets, causing unintended changes; conditional GANs [8] [25] [38] [40] [59] [71] [72] require new data collection and training for new controls.Even encoder-based inversion methods [3] [45] [55] can fail to reconstruct such images.Source: §A.2 · p. 13

Diffusion-based image manipulation

  • Its near-perfect inversions enable zero-shot text-driven manipulation while preserving important details, even for high-variance images.It can translate between unseen domains or generate an unseen-domain image from strokes.Source: §A.2 · pp. 13–14

B. Details on Methods … D.1. Reconstruction

DiffusionCLIP fine-tunes diffusion models through inversion and CLIP/identity-guided generation, then reuses them for unseen-domain translation and flexible attribute mixing; the appendix describes the network and reconstruction evaluation setup.

B.1. DiffusionCLIP Fine-tuning

  • Fine-tuning first deterministically inverts real or pretrained-domain sampled images with DDIM, then updates a copied diffusion model through reverse DDIM using CLIP direction and identity losses toward target text.The update is repeated until convergence.Source: §B.1 · p. 14
  • GPU-efficient fine-tuning backpropagates at each timestep, using half the VRAM but taking twice the time of original fine-tuning.The original method backpropagates from the last step; Figure 13 contrasts the gradient flows.Source: §B.1 · pp. 14–15, Figure 13 · p. 14
  • At manipulation time, the input is inverted with the original pretrained model to t0, then reconstructed with the fine-tuned model using the same t0.This yields the manipulated image through forward and reverse DDIM processes.Source: §B.1 · p. 15

B.2. Image Translation between Unseen Domains

  • For unseen-domain translation, the source image is perturbed with stochastic DDPM, returned toward the pretrained domain through repeated reverse DDIM, then manipulated with the fine-tuned model.The first step repeats KDDPM times before the second-step manipulation.Source: §B.2 · p. 15

B.3. Noise Combination

  • Noise combination mixes single-attribute fine-tuned models in one sampling process, with time-dependent weights summing to 1 to control attribute changes without training a multi-attribute model.Mixing pretrained and fine-tuned noise also interpolates smoothly between the original and manipulated image.Source: §B.3 · p. 15

C. Details on Network

  • The 256×256 CelebA-HQ, LSUN-Bedroom, and LSUN-Church DDPMs use a shared U-Net with encoder, middle, and decoder Res blocks, attention, skip connections, and time embeddings.The architecture uses self-attention at 16×16 in the encoder and in the middle block.Source: C. Details on Network · pp. 15–16

D.1. Reconstruction

  • Reconstruction evaluation inverts 1,500 CelebA-HQ images and measures MAE, LPIPS, and SSIM against GAN inversion baselines, using (Sfor, Sgen) = (200, 40) for DiffusionCLIP.Baselines include optimization, pSp [45], e4e [55], ReStyle [3], and HFGI [58]; their results are taken from [58].Source: §D.1 · p. 16

D.2. Human Evaluation … D.6. Other GAN Baselines

The supplementary sections describe human and quantitative evaluation setups and compare DiffusionCLIP with diffusion- and GAN-based alternatives. The reported comparisons highlight failures on unseen domains or hand poses by alternatives, while DiffusionCLIP manipulates images without the reported unintended changes or artifacts.

D.2. Human Evaluation

  • The human study compared DiffusionCLIP, StyleCLIP-GD, and StyleGAN-NADA on ordinary and hard CelebA-HQ cases, ranking attribute fidelity and semantic preservation.Fifty people cast 6,000 votes; cases covered four in-domain and two out-of-domain attributes.Source: §D.2 · pp. 16–17

D.3. Quantitative Evaluation

  • Quantitative evaluation uses directional CLIP similarity (Sdir), segmentation-consistency (SC), and face identity similarity (ID) to assess attribute correspondence and identity or shape preservation.Sdir compares CLIP text and image embedding differences; SC is pixel accuracy against source segmentation, and ID uses face identity loss.Source: §D.3 · p. 17

D.4. Comparison of Church Image Manipulation

  • The supplementary church comparison evaluates 256 × 256 LSUN-Church manipulations using StyleCLIP latent optimization and StyleGAN-NADA.The passage points to Figure 27 but gives no outcome details.Source: §D.4 · p. 17

D.5. Diffusion-based Manipulations

  • Latent optimization and conditional sampling fail on unseen domains, whereas model fine-tuning shows superior manipulation performance.The alternatives are restricted by the pretrained model’s learned distribution.Source: Figure 17 · p. 17, §D.5 · p. 18
  • The diffusion comparison tests model fine-tuning against latent optimization and CLIP-guided conditional sampling; conditional sampling uses normal CLIP gradients on predicted x0 rather than noisy CLIP.The noisy CLIP model was unavailable, and the passage says training it would be too expensive.Source: §D.5 · pp. 17–18

D.6. Other GAN Baselines

  • DiffusionCLIP translates ImageNet images into target styles while preserving object identity and background details; VQGAN-CLIP misses target-style properties and makes severe unintended changes.The comparison uses 512 × 512 ImageNet images.Source: Figure 18 · p. 18, §D.6 · p. 18
  • StyleSpace and InterfaceGAN fail on images with hand poses, while DiffusionCLIP manipulates them without artifacts.The compared methods are non-text-driven GAN inversion approaches.Source: Figure 19 · p. 18, §D.6 · p. 18

E. Additional Results

Additional results extend DiffusionCLIP to 512 × 512 ImageNet manipulation and translation between unseen domains without curated source or target datasets. They also show that failures can occur when CLIP representations do not support the requested manipulation.

Manipulation of 512 × 512 images from ImageNet

  • DiffusionCLIP manipulates 512 × 512 ImageNet images, showing style transfer, tennis-ball semantic changes, and frog-image results.The method uses GPU-efficient fine-tuning with a diffusion model pretrained on 512 × 512 ImageNet [16], with (Sfor, Sgen) = (40, 12).Source: E. Additional Results · p. 18

Image translation between unseen domains

  • Animation images, portrait art, and strokes are translated into Pixar, Gogh paintings, and Neanderthal men without curated source or target datasets.These are additional image-translation results between unseen domains.Source: E. Additional Results · p. 18

Failure cases

  • DiffusionCLIP can fail when CLIP representations do not support the target, including face-to-object edits and controls that became famous recently.The paper attributes these failures to the dependency on CLIP encoder performance and representations not reflected in CLIP encoders.Source: E. Additional Results · p. 19, Figure 20 · p. 19

F. Hyperparameter and Ablation Study … F.6. Hyperparameters according to Target Text ytar

The ablations show how inversion settings, identity losses, fine-tuning epochs, precomputed images, and sampling affect reconstruction or manipulation, while target text guides hyperparameter choices.

F.2. Identity Loss

  • Identity loss suits controls that require preserving face identity, such as makeup; ℓ1 can further preserve backgrounds, while identity loss can hinder artistic transfer.The choice of identity losses depends on the target control.Source: §F.2 · p. 19

F.3. Dependency on Fine-tuning Epochs K

  • With Adam initialized at 4e-6 and the learning rate increased linearly by 1.2 per 50 iterations, larger K moves generated images closer to the target control.Figure 22 illustrates changes across fine-tuning epochs.Source: §F.3 · p. 19, Figure 22 · p. 19

F.4. Dependency on the Number of Precomputed Images N

  • With a learning rate of 4e-6 and 100 iterations, increasing precomputed-image count N from 5 to 10, 25, 50, or 100 increases manipulation; recycling latents can reduce fine-tuning time.Figure 23 illustrates the effect of N.Source: §F.4 · p. 19, Figure 23 · p. 19

F.1. Dependency on Sfor, Sgen and t0

  • Reconstruction quality improves as Sfor and Sgen increase, but Sfor < Sgen yields similar or worse quality and artifacts; increasing t0 also decreases quality.The evaluation uses MAE, LPIPS, and SSIM.Source: §F.1 · p. 19, Table 4 · p. 20

F.5. Stochastic Manipulation

  • Stochastic DDPM sampling rather than deterministic DDIM sampling can produce varied modifications useful for artistic transfer.Figure 24 shows results with random seeds.Source: Figure 24 · p. 19, §F.5 · p. 20

F.6. Hyperparameters according to Target Text ytar

  • For severe shape or color changes, use high t0 without identity losses; when object identity matters, use low t0 with identity losses.The actual hyperparameters for different controls are t0, λL1, and λID; the method has a similar number of hyperparameters to StyleCLIP [39] and StyleGAN-NADA [20].Source: §F.6 · p. 20

G. Running Time and Resources

DiffusionCLIP’s training and inference costs vary with procedure and sampling settings, while its fine-tuned model can be reused across images from the same pretrained domain. The section also reports higher runtime and GPU use at larger image sizes.

DiffusionCLIP fine-tuning

  • With Sfor=40 and Sgen=6, precomputing 50 latents takes about 82.2 seconds and 50 model updates take 41.3 seconds per epoch; GPU-efficient updating roughly doubles fine-tuning time.Latent precomputing requires about 6GB VRAM; original and GPU-efficient updating require 23GB and 12GB, respectively.Source: G. Running Time and Resources · p. 20

Manipulation of images from pretrained domain

  • For pretrained-domain manipulation, (Sfor, Sgen)=(40,6) takes 1.958 seconds total, while (200,40) takes 10.132 seconds; these applications require at least 6GB VRAM.The quick setting uses 1.644 seconds for inversion and 0.314 seconds for generation.Source: G. Running Time and Resources · p. 20

Image translation between unseen domains

  • Unseen-domain translation takes 2.272–5.098 seconds at (Sfor, Sgen)=(40,6) and 10.446–13.272 seconds at (200,40), including KDDPM forward-DDPM/reverse-DDIM steps.Because x′t can be sampled in closed form, the added process takes 0.314 seconds per Kforward step, with Kforward=1–10.Source: G. Running Time and Resources · pp. 20–21

Multi-attribute transfer

Trade-off between the inference time and preparation time

  • Fine-tuning takes 1–7 minutes, faster than StyleCLIP’s latent mapper (10–12 hours) and similar to StyleGAN-NADA (a few minutes); latent optimization methods [39] instead optimize per image without preparation.After setup, DiffusionCLIP can be applied to all images from the same pretrained domain.Source: G. Running Time and Resources · p. 21

Increasing image size

H. Societal Impacts

DiffusionCLIP makes high-quality image manipulation accessible through simple text prompts, but realistic results could be misused to confuse people. Its pretrained CLIP and diffusion models can also produce biased results.

H. Societal Impacts

  • Simple text prompts let people manipulate images at high quality without professional artistic skills, but realistic results could be used maliciously to confuse people.The authors advise using the method carefully and for proper purposes.Source: H. Societal Impacts · p. 21
  • CLIP may introduce bias [41], and diffusion models can generate biased results, including CelebA-HQ [27] models producing mostly attractive-looking faces aged 20–40 [18].The authors call for more research into generative models and representation learning that resolves bias issues.Source: H. Societal Impacts · p. 21

Related papers

Loading 2110.02711v6…