StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators

Introduction

• The method shifts a pretrained generator toward an out-of-domain target using only a text prompt, without seeing any target-domain images.The paper illustrates this blind adaptation with three examples.Source: §1 · pp. 1–2, Figure 1 · p. 1
• A frozen source generator and a trainable generator use a novel loss to constrain changes to a textually described direction in CLIP embedding space.This approach addresses the challenge that naïvely maximizing a CLIP-based classification score can produce adversarial solutions.Source: §1 · p. 2
• For stability under more drastic domain changes, an adaptive approach uses CLIP to select and restrict training to the most relevant network layers at each iteration.Source: §1 · p. 2
• Applied to StyleGAN2, the method adapts artistic styles, cross-species identities, and shapes such as dogs to bears, enabling changes beyond editing and few-shot alternatives without training data.Source: Figure 1 · p. 1, §1 · p. 2
• The shifted generator retains the original latent space’s editing capabilities and can reuse off-the-shelf editing directions and models trained for the original domain.Source: §1 · p. 2

• Unlike prior CLIP-guided methods that optimize a latent code for a specific image, this approach uses a text prompt to guide training of the image generator itself.CLIP maps images and text into a joint multimodal embedding space and has guided image synthesis and manipulation.Source: §2 · p. 2
• Few-shot generation seeks diverse distributions from few images; from-scratch approaches often call several thousand images “few” and use augmentation [20] [50] [64] [65] or discriminator auxiliary tasks [29] [60].Few-shot methods divide into training a new generator and fine-tuning a pretrained one; some from-scratch comparisons involve tens of thousands [23] or millions [8] of images.Source: §2 · p. 2
• Few-shot transfer learning spans hundreds to five images and addresses mode collapse or overfitting using weight restrictions [31] [38] [41], channel statistics [34], latent steering [55], regularization [28] [51], or consistency [35].Source: §2 · p. 2
• Unlike prior limited-data adaptation, this method trains without training images and adaptively selects layers based on network state and target class, rather than using fixed hand-picked subsets of weights.Source: §2 · p. 2

3.1. StyleGAN

• StyleGAN maps Gaussian latent codes z into learned vectors w, which control synthesis features and enable fine-grained semantic editing, but edits typically remain within the training domain.Prior work demonstrated such control by traversing W or mixing codes across layers [3] [19] [37] [45].Source: §3.1 · p. 2

3.2. StyleCLIP

• Patashnik et al. [37] use CLIP text prompts for editing through latent optimization, trained encoders or mappers, or global directions of disentangled change in latent space.Their third approach identifies latent entries whose image-space changes align with the CLIP-space direction between source and target text descriptors.Source: §3.2 · pp. 2–3
• StyleCLIP-style latent editing remains constrained by the pretrained generator’s domain, enabling changes like hairstyle or expression but not photo-to-painting or dog-to-cat conversion.The passage also gives wolf-to-lion conversion as possible when the generator has seen both domains.Source: §3.2 · p. 3

4.1. CLIP-based guidance

• The global CLIP loss compares generated images with the target text but can encourage pixel-level adversarial perturbations and mode collapse, so it is unsuitable for generator training.The method nevertheless uses this loss for adaptive layer selection.Source: §4.1 · p. 3
• Directional guidance aligns the CLIP-space change from a frozen source generator to a trained generator with the text-defined source-to-target direction.The generators share a frozen mapping network to maintain latent-space alignment; this loss resists mode collapse and makes adversarial solutions harder to reach.Source: §4.1 · pp. 3–4, Figure 3 · p. 4

4.2. Layer-Freezing

• For texture shifts, training quickly converges, but extensive shape changes require longer training and can destabilize the network and degrade results.The method selects the k layers with the largest latent-code changes after optimizing sampled codes with a frozen generator and global CLIP loss, then trains only those layers; it also freezes the mapping network, affine transformations, and toRGB layers.Source: §4.2 · p. 4

4.3. Latent-Mapper ‘mining’

• When a shape shift leaves outputs spanning source, target, and intermediate forms, a StyleCLIP latent mapper can map codes into the target-like region.For example, dog-to-cat adaptation may produce cats, dogs, and intermediate images because both domains remain in the shifted generator’s domain.Source: §4.3 · pp. 4–5

5.1. Results

• Text prompts enabled adaptations across styles, textures, and substantial shape changes, including zero-shot domains without real data, typically within minutes.Dog-to-animal adaptations included upright ears uncommon among AFHQ-Dog [11] breeds.Source: §5.1 · p. 5, Figure 7 · p. 6

5.2. Latent space exploration

• Inverted real-image codes preserved identity in adapted generators, while existing latent paths and pretrained editing models supported edits across the translated domains.The experiments reused ReStyle [5] for inversion and StyleCLIP [37], StyleFlow [3], and InterFaceGAN [45] for expression, hairstyle, pose, and age edits.Source: §5.2 · pp. 5–6, Figure 8 · p. 7, Figure 9 · p. 7
• Pretrained encoders also enabled conditional synthesis with adapted generators from segmentation masks and sketches, without encoder retraining.The experiments paired adapted generators with the official pSp [40] FFHQ encoder.Source: §5.2 · p. 6, Figure 10 · p. 7

5.3. Comparison to other methods

• StyleCLIP’s [37] three text-guided editing approaches failed to shift images out of the pretrained generator’s domain, even for minor changes such as celebrity identities on dogs.The comparison used out-of-domain textual directions and the same latent code across approaches.Source: §5.3 · p. 7, Figure 11 · p. 7
• StyleGAN-NADA, trained zero-shot, outperformed most 5- and 10-shot methods in quality and improved diversity over methods trained with 100 images.With 5 images, MineGAN [55] memorized the training data.Source: §5.3 · pp. 7–8, Figure 12 · p. 8, Figure 13 · p. 8
• Using StyleGAN-NADA before few-shot training improved synthesis performance in many cases.The paper reports this as an additional comparison finding and refers to Appendix C for details.Source: §5.3 · p. 8

5.4. Ablation study

• The directional-loss model with adaptive layer-freezing achieved the best visual results; the global CLIP-loss approach consistently failed, and a latent mapper sometimes improved quality.The mapper was needed only for more extensive shape changes.Source: §5.4 · p. 8, Figure 14 · p. 8

6. Conclusions

• StyleGAN-NADA uses CLIP-guided training rather than latent-space exploration to make large style and shape changes beyond a generator’s original domain.Source: §6 · p. 8
• Training generators without data could enable user-directed image editing and paired cross-domain data synthesis for downstream tasks such as image-to-image translation.Source: §6 · p. 8
• The method is limited to concepts CLIP has observed, while natural-language ambiguity can make prompts such as “Raphael Painting” refer to distinct subjects or styles.Source: §6 · p. 8
• The method works particularly well for style and fine details but may struggle with large-scale attributes or geometry changes, and good translations often require a similar pretrained generator.Source: §6 · pp. 8–9
• The authors raise training a generator from scratch using only CLIP guidance as an open question, encouraged by progress in classifier inversion and generative art.Source: §6 · p. 9

A. Broader impact

• The project aims to support creative content creation and machine-learning work in data-limited areas, but the tool could also be used for nefarious purposes.Source: A. Broader impact · p. 13
• Because CLIP may propagate biases in its internet-trained data, “doctor” prompts produce mostly male faces and “nurse” prompts the opposite; Appendix C proposes mitigation using a small image set.Source: A. Broader impact · p. 13

B. CLIP-space analysis

• The analysis embeds AFHQ cat and dog images [11] in CLIP’s multimodal space, then projects them to 2D using PCA, alongside embedded texts and generated images.The generated images were synthesized after training with both the global loss and the directional loss; results are shown in Fig. 15.Source: B. CLIP-space analysis · p. 13
• Global CLIP loss optimizes toward a single target, and generated results visibly collapse to one region of the embedding space.The behavior is visualized in Fig. 15b.Source: B. CLIP-space analysis · p. 13
• Directional loss discourages embedding-space collapse and successfully maintains a higher degree of diversity among generated images.The behavior is visualized in Fig. 15c.Source: B. CLIP-space analysis · p. 13

C. Few-Shot CLIP-Guidance

• With ∼3 −5 target images, image-based CLIP directions replace text-prompt directions by encoding the semantic difference between source-generated and target-domain images.This approach better maintains latent structure and identity, trains faster, and requires neither aligned nor source-domain-preprocessed target images.Source: C. Few-Shot CLIP-Guidance · p. 14
• Few-shot images can resolve linguistic ambiguity, target hard-to-describe styles, and counter CLIP biases, but semantic embeddings do not guarantee exact style realization.The few-shot set itself may also introduce biases.Source: C. Few-Shot CLIP-Guidance · p. 14
• Zero-shot pre-training first narrows the source-to-target domain gap, improving few-shot adaptation whose performance typically depends on source-generator similarity to the target.The proposed sequence applies NADA to the source generator with source and target text prompts, then uses the resulting intermediate generator for few-shot adaptation.Source: C. Few-Shot CLIP-Guidance · p. 14
• A brief discriminator-only “catch-up” session yields the most consistent improvements among three pre-training alternatives for dog-to-cat adaptation.The authors hypothesize that initial discriminator synchronization focuses it on features distinguishing real cats from fake ones, rather than cats from dogs.Source: Table 1 · p. 15, C. Few-Shot CLIP-Guidance · p. 15
• More than 40% FID-score improvements occur in some cases; zero-shot pre-training lowers FID in almost all tested settings.The evaluation uses the “catch-up” method, which first adapts the discriminator using NADA-generated fake images and few-shot target images.Source: Table 2 · p. 15, C. Few-Shot CLIP-Guidance · p. 15

D. Beyond StyleGAN

• For OASIS segmentation-mask generation, masking CLIP inputs to a designated class and applying L2 and LPIPS losses outside it enables localized class changes.The results support applying the framework beyond StyleGAN, to generative models trained without data.Source: D. Beyond StyleGAN · p. 15

E. Identity Preservation

• StyleGAN-NADA preserves source identity across domains in synthetic and inverted real images, including appropriate accessories such as hats and eyeglasses.The real images were inverted using e4e [49].Source: E. Identity Preservation · p. 16

F. Cross-Model interpolation

• Interpolating model parameters between domains produces smooth transitions across domains and demonstrates coupling between the transformed models’ latent spaces.The transitions enable applications such as videos spanning multiple domains.Source: F. Cross-Model interpolation · p. 16

• Figures 20 and 21 show models converted from the face domain, while Figure 22 shows models converted from the church domain.Source: G. Additional samples · p. 16
• Figures 23 and 24 show additional results from the dog domain and other animal transformations, respectively.Source: G. Additional samples · p. 16

• The qualitative comparison transforms the official StyleGAN-ADA [20] AFHQ-Dog [11] model to cats using zero-shot adaptation and few-shot models trained on AFHQ-Cat [11] samples.Source: H. Qualitative few-shot comparisons · p. 16
• In the extreme low-data regime, competing methods either fail to generate meaningful images across the domain gap or memorize training data; only our method and Ojha et al.’s [35] 100-image variant preserve consistent source-target correspondence.Source: H. Qualitative few-shot comparisons · p. 16
• On FFHQ-to-sketch transfer, our zero-shot method produces higher-quality images and preserves more source detail, though matching the target sketch style with text is difficult.The prompts were “Photo” to “Black and white pencil sketch”.Source: H. Qualitative few-shot comparisons · p. 16
• Using three target images narrows but does not eliminate the style gap; competing methods show artifacts, mode collapse, or weak source correspondence even across close domains.Source: H. Qualitative few-shot comparisons · p. 16

Hyper parameter choices

• Texture-based changes typically take 300 iterations with batch size 2 (about 3 minutes on one NVIDIA V100), while “Photo” to “Sketch” can converge in 50 iterations in under a minute, two orders of magnitude faster than recent speed-focused adversarial methods.Animal changes typically take 2,000 iterations, followed by StyleCLIP mapper training, for a total of about 6 hours on one NVIDIA V100.Source: §I · p. 16
• All experiments use ADAM with learning rate 0.002.Source: §I · p. 16
• Adaptive layer freezing uses optimization batch size Nw = 8 and Ni = 1 iteration; texture edits train all layers, small shape edits train k = 2/3 of layers (12 for FFHQ), and animal edits train k = 3.Source: §I · p. 17
• The method uses ViT-32/B and ViT-B/16 CLIP models; either can yield different artistic styles for texture changes, while using both together gives the best shape-change results.The two encoders tend to focus on different levels of detail, helping shape changes capture global shape and local texture.Source: §I · p. 17
• StyleGAN2 style-mixing probability can affect generated artistic style, and animal-change experiments were conducted without mixing.Source: §I · p. 17

StyleCLIP mapper

• The StyleCLIP mapper can introduce artifacts such as open mouths and enlarged tongues, which correlate with increased CLIP-embedding norms; mapper training constrains those norms with an additional loss.The mapper-training weights are λL2 = 0.5 and λembedding−norm = 0.2.Source: §I · pp. 17–18

J. Licenses and data privacy

• Tables 4 and 5 list the models and datasets used, their sources, and their respective licenses.Source: J. Licenses and data privacy · p. 18, Table 4 · p. 18, Table 5 · p. 18
• FFHQ [22] contains biometric face images crawled from Flickr without contacting owners; its licenses allow non-commercial use, redistribution, and adaptation, and curators provide removal contact details.Source: J. Licenses and data privacy · p. 18