Diffusion Guided Domain Adaptation of Image Generators

1. Introduction

• The method shifts a pre-trained GAN toward a text-indicated domain without requiring training images from that target domain.It uses a pre-trained text-to-image diffusion model as guidance.Source: §1 · p. 1
• CLIP-guided adaptation can be limited by text-image latent-space misalignment and local minima, leading to artifacts and reduced diversity.The paper cites StyleGAN-NADA as an example of text-guided fine-tuning that uses CLIP.Source: §1 · p. 1
• The authors motivate diffusion guidance as potentially more informative for generators than guidance from a discriminative model such as CLIP.Source: §1 · p. 1
• Diffusion score distillation sampling (SDS) is introduced for style-based generator domain adaptation and is reported to outperform prior art.The diffusion model acts as a frozen critic that predicts image-space edits.Source: §1 · p. 1
• The paper proposes a diffusion directional regularizer and adapts reconstruction guidance to SDS, while using layer selection to address blurry outputs.Source: §1 · p. 2

2. Related Work

• Few-shot adaptation uses several hundred or fewer target images and may control latent-space statistics or sampling, add collapse regularizers, or use auxiliary tasks.Source: §2 · p. 2
• Text-guided zero-shot methods use CLIP to identify editable directions or fine-tune StyleGAN generators using text-image similarity objectives.StyleCLIP optimizes latent codes, while StyleGAN-NADA directly fine-tunes the generator.Source: §2 · p. 2
• Diffusion research includes applying CLIP objectives to diffusion generators and fine-tuning diffusion models or text embeddings on personalized images.Source: §2 · p. 2
• Latent diffusion models perform denoising in a compressed autoencoder latent space; StableDiffusion is identified as a popular example.Source: §2 · pp. 2–3
• This work extends SDS guidance from DreamFusion to StableDiffusion's latent space for StyleGAN2 domain adaptation and also tests EG3D generators.DreamFusion used SDS to guide NeRF training in image pixel space.Source: §2 · p. 3

3.1. Background

• Classifier-free guidance combines conditional and unconditional noise predictions, with guidance weight s controlling the strength of the condition.The conditional model is trained with randomly dropped conditions, avoiding a separate pretrained classifier.Source: §3.1 · p. 3

3.2. Model Structure and Diffusion Guidance Loss

• The generator encodes generated images into Stable Diffusion latents, adds sampled noise, and backpropagates the difference between predicted and true noise through the encoder to update generator parameters.The noisy latent is sampled at a timestep below the diffusion model’s full timestep range.Source: §3.2 · p. 3

3.3. Directional and Reconstruction Regularizer

• SDS improves image fidelity at the cost of diversity and can collapse the generator to one image pattern across input noises.The paper attributes collapse to conditional and unconditional predictions becoming equal, a behavior it calls differentiable image parameterization.Source: §3.3 · p. 3
• The diffusion directional regularizer constrains SDS updates to maintain the frozen generator’s optimization direction, helping prevent model collapse.It compares normalized current and frozen classifier-free guidance scores; experiments report that it can prevent collapse and it is compatible with other regularizers.Source: §3.3 · pp. 3–4
• Reconstruction regularization encourages the estimated clean latent to match the frozen generator’s latent, and the overall objective combines SDS with weighted regularization.The reconstruction loss is the squared difference between the estimated and target latents.Source: §3.3 · p. 4

3.4. Timestep Range and Layer Selection

• The sampled timestep range controls change scale, while selecting the most changed W+ style-code layers helps reduce blurry outputs.Larger timesteps emphasize high-level structure; smaller ones provide more local detail, and layer selection follows N optimization iterations.Source: §3.4 · p. 4

4.1. Results

• The method adapts generators to face, cat, dog, and other domains, with examples including photorealistic cars, werewolves, Joker, and 3D-rendered animals.Figure 3 uses target prompts and does not require a source prompt.Source: §4.1 · pp. 4–5, Figure 3 · p. 5

4.2.1 Long Text Prompts

• On long prompts, our method better matches the requested face layout, reflective eyes, smooth texture, and realistic 3D lighting than the baseline.The baseline’s face images have distorted shapes and an oil-painting-like style that conflicts with the smooth-texture prompt.Source: §4.2.1 · pp. 5–6, Figure 5 · p. 6
• For 3D-stylized cats, our results have more cinematic lighting, smoother textures, stronger 3D style, and higher-quality detail than the baseline.The baseline fails to follow multiple prompt constraints and produces less realistic shading and undesired textures.Source: §4.2.1 · p. 6, Figure 6 · p. 6

4.2.2 Quantitative Evaluation

• In animal experiments, four uncurated samples per animal type show higher visual quality and prompt fidelity for our method than the baseline.The quantitative evaluation adapts an AFHQ-Cat generator to animal types specified by prompts.Source: §4.2.2 · p. 6
• Animal experiments show competitive CLIP scores, better LPIPS scores, and significantly better FID than the baseline; long-prompt keyword matching also favors our method by a large margin.The long-prompt evaluation measures image-text matching for individual keywords from the prompt.Source: §4.2.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

4.3. Timestep Range and Layer Selection

• Larger TSDS ranges enable global structural changes, while smaller ranges favor local details; this lets users trade original-domain fidelity against target-domain authenticity.For cat-to-otter adaptation, larger TSDS values make the otter’s ears smaller and more realistic.Source: §4.3 · pp. 7–8
• Our Cat-to-8-animals samples are more realistic and natural than the baseline, with undistorted faces, cleaner backgrounds, diverse poses, and higher pose faithfulness.The figure contrasts uncurated samples from the method and baseline.Source: Figure 8 · p. 8
• Selecting fewer layers reduces blur and better preserves hair details, but requires more training iterations.The comparison shows configurations selecting 12, 6, or 3 layers per iteration.Source: Figure 9 · p. 8

4.4. Directional and Reconstruction Regularizer

• Directional regularization preserves facial details and diversity and helps prevent collapse, whereas the nonregularized model eventually ignores the input latent and degrades to one dark image.The comparison uses a prompt requesting a 3D-stylized anime face with circular reflective eyes.Source: §4.4 · pp. 8–9
• Reconstruction guidance preserves most hair details but disables requested color changes; directional guidance allows more color modification at the cost of fewer details.The reconstruction-guidance figure illustrates added colors such as blue eyes.Source: §4.4 · p. 9, Figure 11 · p. 9

4.5. Extension to 3D-Aware Generative Models

• On EG3D face and cat models, adaptation produces smoother mesh geometry and larger eye areas as requested by the text prompt.During optimization, only tri-plane generator Conv-layer weights are updated, and LPIPS is added to strengthen faithfulness to the training generator.Source: §4.5 · p. 9

4.6. Extension to DreamBooth Guidance

• The method is also applied with DreamBooth StableDiffusion checkpoints for “Wa-vy” and “Woolitize” styles.The demonstrated prompts are “wa-vy style painting close up face” and “woolitize close up face”.Source: §4.6 · p. 9

5. Conclusion

• StableDiffusion guidance with Score Distillation Sampling, directional regularization, and layer selection shifts generators toward prompt-indicated domains with improved quality over existing methods.The modification magnitude is controlled by the TSDS value.Source: §5 · p. 9
• The approach also extends to 3D-aware style-based generators and can use DreamBooth models as guidance.The conclusion points to face and cat EG3D examples and DreamBooth-guided results.Source: §5 · p. 9, Figure 12 · p. 9, Figure 13 · p. 9

A. Full-Text Prompts

• The prompt list includes face, cat, and dog styles, animal substitutions, vehicle transformations, portraits, and other text-guided targets.Examples include “Joker,” “Werewolf,” “TRON wheel,” and “Sketch of a car, pen and ink sketch.”Source: A. Full-Text Prompts · p. 12
• Prompt IDs are mapped to the main-text figures and tables, including shared prompts across figures and separate animal runs for Tables 1 and 2.For example, Figures 7 and 8 use prompt 7, while Tables 1 and 2 each use prompt 7 with one animal type per run.Source: A. Full-Text Prompts · p. 12

B. Implementation Details

• The method uses StyleGAN2 PyTorch checkpoints and StableDiffusion v1.4; training freezes the mapping MLP, ToRGB layers, and biases, updating only Conv-layer weights.Training uses Adam with default parameters, a 5 × 10^-4 learning rate, 2,000 steps, and batch size 1.Source: B. Implementation Details · p. 12
• Training takes about 20 minutes on an A100 GPU, with memory use of 14.7G at 1024 resolution and 12.5G at 512 resolution.The appendix also reports experiment times of 20 minutes to 2 hours on a V100, with 25G memory at batch size one.Source: B. Implementation Details · p. 12

• Animal FID was calculated using manually extracted AFHQ wild-subclass images, while the models saw no ground-truth images.Source: C. More Quantitative Evaluations · p. 12
• The method achieved significantly better FID scores than StyleGAN-NADA in the animal experiments.Source: C. More Quantitative Evaluations · p. 12, Table 4 · p. 13, Table 5 · p. 13
• Additional visual results cover FFHQ faces, cars, and landscapes.Source: C. More Quantitative Evaluations · p. 13

• Figure 15 compares Car-to-TRON wheel settings with Tmax = 600 in the second column and Tmax = 300 in the third.Source: Figure 15 · p. 13
• In most cases, Tmax = 300 has a better FID score across the three compared TSDS configurations.Source: D. TSDS Configurations · p. 13
• Higher Tmax allows more compositional modifications, whereas lower Tmax allows only local changes.Source: D. TSDS Configurations · p. 13

• Figures 16 and 17 show the proposed model’s cat-to-animal outputs, while Figure 18 shows StyleGAN-NADA results.Source: E. Large-Scale Image Galleries · p. 13, Figure 16 · p. 14, Figure 17 · p. 15
• Figure 18 shows StyleGAN-NADA outputs with distorted facial components, unrealistic texture, and lower diversity than the proposed model.Source: Figure 18 · p. 16
• Additional galleries show FFHQ face results and car-to-concept-car and landscape results, with StableDiffusion samples at left and Gfrozen samples marked in red.Source: Figure 19 · p. 17, Figure 20 · p. 18, Figure 21 · p. 19