PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators

Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, Daniel Cohen-Or

arXiv:2108.00946v2cs.CVcs.CLcs.GRcs.LG

TL;DR

The paper asks whether an image generator can be adapted to a new domain without target images. It introduces a CLIP-guided training method that shifts a pretrained generator using text alone, and reports broad adaptations across styles and shapes while preserving latent-space editing properties.

  • Problem

    Pretrained generators are typically restricted to domains with large image collections, although some target domains may have insufficient or no data.

  • Method

    The method uses CLIP to define a textual cross-domain direction and trains a generator against a frozen source copy with adaptive layer selection.

  • Results

    The method adapts StyleGAN2 across artistic styles, cross-species identity transfers, and substantial shape changes without direct access to target training data, while retaining latent-space editing capabilities.

  • Takeaways & Limitations

    Data-free generator training enables out-of-domain image generation and can support editing and paired cross-domain data synthesis for downstream applications.

  • Takeaways & Limitations

    The method is limited to concepts observed by CLIP, and natural-language prompts can be ambiguous.

Abstract

from arXiv · show

Can a generative model be trained to produce images from a specific domain, guided by a text prompt only, without seeing any image? In other words: can an image generator be trained "blindly"? Leveraging the semantic power of large scale Contrastive-Language-Image-Pre-training (CLIP) models, we present a text-driven method that allows shifting a generative model to new domains, without having to collect even a single image. We show that through natural language prompts and a few minutes of training, our method can adapt a generator across a multitude of domains characterized by diverse styles and shapes. Notably, many of these modifications would be difficult or outright impossible to reach with existing methods. We conduct an extensive set of experiments and comparisons across a wide range of domains. These demonstrate the effectiveness of our approach and show that our shifted models maintain the latent-space properties that make generative models appealing for downstream tasks.

1. Introduction

StyleGAN-NADA adapts pretrained image generators to out-of-domain targets using only textual guidance, addressing the lack of sufficient target-domain images. Its training scheme uses CLIP-guided domain directions, two generators, and adaptive layer selection while preserving latent-space properties.

  • Large image datasets are often unavailable for domains such as specific artists, rare medical conditions, and imaginary scenes, limiting generator applicability.
  • StyleGAN-NADA shifts a pretrained generator toward a new domain using nothing more than a textual prompt.
  • The method encodes source-to-target domain differences as a direction in CLIP’s embedding space rather than directly maximizing a CLIP classification score.
  • A frozen source generator and an optimized generator constrain changes to the textually described cross-domain direction, while adaptive layer selection improves stability for drastic changes.
  • Experiments with StyleGAN2 cover artistic styles, cross-species identity transfer, and major shape changes without direct access to training data.
  • The shifted generator retains the original latent space’s editing capabilities and can reuse off-the-shelf editing directions and models trained for the original domain.

2. Related work

Prior work addresses text-guided synthesis and limited-data generator training, but StyleGAN-NADA adapts generators without direct access to training images and selects trainable layers adaptively.

  • CLIP maps text and images into a shared multimodal embedding space and has guided image synthesis and manipulation through textual descriptions.
  • Few-shot generative methods aim to reproduce diverse data distributions from limited images through training from scratch or fine-tuning pretrained generators.
  • In new-generator approaches, “few” often means several thousand images rather than tens of thousands or millions.
  • Transfer-learning approaches typically use several hundred to as few as five images, while addressing mode collapse, overfitting, and source-diversity transfer.
  • Unlike prior limited-data methods, StyleGAN-NADA works without training images and uses adaptive layer selection instead of fixed hand-picked trainable-weight subsets.

3. Preliminaries

StyleGAN combines a learned latent space with a synthesis network, while prior CLIP-based editing methods guide latent-code changes within a pretrained generator’s domain. StyleGAN-NADA instead uses CLIP to guide training of the generator itself.

  • StyleGAN and its variants are established unconditional image generators with semantically rich latent representations.
  • StyleGAN maps Gaussian latent codes z through a mapping network into latent space W, whose vectors control feature statistics in the synthesis network.
  • StyleGAN-NADA’s setup initializes frozen and trainable generators from a source-domain generator, shifting the trainable copy while maintaining a shared latent space.
  • Earlier StyleGAN-CLIP methods use textual descriptions to discover latent-space editing directions, including directions between source and target text embeddings.
  • Latent-space editing generally remains constrained by the pretrained generator’s domain, limiting transformations such as converting photos to paintings or dogs to cats.

4. Method

The method adapts a pre-trained generator to a text-described target domain using only CLIP supervision, while preserving source-target alignment and latent-space structure. It combines directional CLIP guidance with adaptive layer selection and latent mapping to support substantial domain shifts without target images.

  • The method shifts a pre-trained generator toward a text-described target domain using only a pre-trained CLIP model, without target images.
  • CLIP-based guidance: A global CLIP loss matches generated images to the target text, but can produce adversarial perturbations and mode collapse.The loss is therefore used for adaptive layer selection rather than directly training the generator.
  • CLIP-based guidance: The directional loss aligns the CLIP-space difference between frozen-source and trainable-generator images with the difference between source and target text embeddings.A frozen generator supplies source-domain images for each latent code, while the trainable copy is optimized; both share a frozen mapping network.
  • CLIP-based guidance: The directional loss reduces mode-collapse sensitivity and makes adversarial solutions harder by requiring consistent alignment across source instances.
  • Layer-Freezing: Adaptive layer freezing selects the k most relevant layers from W+ latent-code changes, then trains only those layers with directional CLIP loss.Selection uses global-loss optimization with a frozen generator, while the mapping network, affine transformations, and toRGB layers remain frozen.
  • Latent-Mapper ‘mining’: For larger shape changes, the method can produce incomplete transformations, so latent-mapper editing maps codes into the target-like region.The paper notes this behavior for dog-to-cat transformation and describes the mapper as an in-domain latent-editing step.

5. Experiments

Experiments show that StyleGAN-NADA adapts generators across diverse styles, species, and shapes while preserving latent-space compatibility for inversion, editing, and translation. Comparisons indicate advantages over latent-editing and few-shot alternatives, and ablations support directional loss with adaptive layer freezing.

  • 5.1. Results: The method adapts generators across artistic styles, cross-species identity transfers, and substantial shape changes, including dogs to bears.These adaptations require no direct access to training data.
  • 5.1. Results: Dog-based generators translate to many animal domains, including cases requiring significant shape modifications such as upright ears.The source domain is AFHQ-Dog, while target domains are specified by text.
  • 5.2. Latent space exploration: Adapted generators preserve latent-code identity and remain compatible with existing inversion and editing methods.The same latent codes and source-domain editing models can be reused in translated domains.
  • 5.2. Latent space exploration: Pre-trained encoders support conditional synthesis in multiple domains from segmentation masks and sketches without encoder retraining.The approach extends image-to-image translation beyond domains for which a StyleGAN generator can be trained.
  • 5.3. Comparison to other methods: StyleCLIP’s three editing approaches fail to perform out-of-domain manipulations, whereas the converted generator produces the corresponding novel-domain images.The comparison uses the same randomly sampled latent code across methods.
  • 5.3. Comparison to other methods: The method consistently outperforms most 5- and 10-shot methods in quality and improves diversity relative to methods trained with 100 images.With 5 images, MineGAN memorizes the training set; using StyleGAN-NADA before few-shot training can further improve synthesis performance.
  • 5.4. Ablation study: Directional loss with adaptive layer freezing achieves the best visual ablation results, while the global loss consistently fails across domains and modifications.Training a latent mapper can improve quality further in some cases.

6. Conclusions

StyleGAN-NADA uses CLIP to adapt generators across broad style and shape changes without target-domain data, while preserving latent-space capabilities. Its scope is limited by CLIP’s concepts, linguistic ambiguity, large-scale changes, and dependence on a similar starting generator.

  • StyleGAN-NADA is a CLIP-guided zero-shot method that changes generator style and shape beyond the original domain.
  • Training generators without data supports creative editing and synthesis of paired cross-domain data for downstream applications.
  • CLIP-based adaptation is limited to concepts CLIP has observed, and natural-language prompts can be ambiguous, as with “Raphael Painting.”
  • The method works particularly well for style and fine details but may struggle with large-scale attributes or geometry changes.
  • Good translations often require a pre-trained generator that is fairly similar to the target domain.
  • The paper focuses on transforming existing generators and leaves training one from scratch using only CLIP guidance as an open question.

Supplementary Materials StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators

The project targets creative content creation and machine learning settings with limited data, while acknowledging potential misuse and bias inherited from CLIP’s internet-scale training data.

  • The tool is intended to help artists create content and support machine learning in areas with limited data.
  • The authors acknowledge that the tool could also be used for nefarious purposes.
  • Because CLIP learned from large internet image-text collections, StyleGAN-NADA may propagate biases present in those data.
  • For face-generator conversion, “doctor” prompts produce mostly males while “nurse” prompts have the opposite effect; a small image set is proposed for mitigation.

B. CLIP-space analysis

The CLIP-space analysis contrasts global distance minimization with the proposed directional loss: global optimization collapses outputs, whereas directional optimization preserves greater diversity.

  • The analysis embeds real cat and dog images, their text labels, and generated images into CLIP space before projecting them with PCA.
  • Global CLIP loss optimizes toward one target and visibly collapses generated images into a single embedding-space region.
  • Directional loss discourages collapse and maintains a higher degree of diversity in the generated distribution.

C. Few-Shot CLIP-Guidance

The supplementary experiments extend CLIP guidance to few-shot adaptation, showing benefits for latent-space structure, identity preservation, training speed, and downstream FID. They also examine discriminator synchronization, image-based guidance, and broader generator applicability.

  • Few-Shot CLIP-Guidance: CLIP guidance can also support few-shot training, including image-based directions when approximately 3–5 target images are available.
  • Few-Shot CLIP-Guidance: The image-based approach better maintains latent-space structure, improves identity preservation, trains faster, and avoids target-image alignment or source-domain preprocessing.
  • Few-Shot CLIP-Guidance: Image examples reduce linguistic ambiguity and can target styles that are difficult to describe, but semantic embeddings do not guarantee the exact style realization.
  • Few-Shot CLIP-Guidance: Zero-shot pre-training improves few-shot adaptation by providing an intermediate generator before image-based fine-tuning.
  • Few-Shot CLIP-Guidance: A brief discriminator-only catch-up session produces the most consistent improvements among the evaluated alternatives.
  • Few-Shot CLIP-Guidance: In almost all tested cases, zero-shot pre-training lowers FID, with some improvements exceeding 40% and reducing the later method’s domain gap.
  • Few-Shot CLIP-Guidance: The framework transfers beyond StyleGAN, indicating a general approach for training generative models without data.
  • Few-Shot CLIP-Guidance: Adapted models preserve source identity across domains, including associated accessories such as hats and eyeglasses.

G. Additional samples

The additional samples demonstrate zero-shot and few-shot domain adaptations across faces, churches, dogs, animals, and challenging shape or style changes. Compared with few-shot methods, the method preserves image correspondence and detail but struggles to target an exact artistic style.

  • Additional synthesized results: Additional samples cover models adapted from face, church, and dog domains to textually prescribed target domains.Animal transformations are also shown using a StyleGAN-ADA AFHQ-Dog model and a StyleCLIP latent mapper.
  • Few-shot comparisons: The zero-shot dog-to-cat transformation preserves correspondence where competing low-data methods produce artifacts or memorize their training sets.The comparison begins from the official StyleGAN-ADA AFHQ-Dog model and uses few-shot AFHQ-Cat samples for the alternatives.
  • Few-shot comparisons: In the extreme low-data regime, competing methods either fail on the domain gap or memorize training images.Only the proposed method and Ojha et al.’s 100-image variant maintain consistent source-target correspondence.
  • Few-shot comparisons: The zero-shot photo-to-pencil-sketch transformation maintains higher quality and more source detail, but natural-language prompts struggle to match an exact artistic style.Using three target images allows adaptation of more features from the specific style, although a gap remains.

I. Training details

Training details vary with the type of domain change: texture edits converge quickly, while animal changes require longer training and may use adaptive layer freezing, CLIP-model combinations, and a StyleCLIP mapper.

  • Optimization: Texture-based changes typically require 300 iterations with batch size 2, taking roughly 3 minutes on one NVIDIA V100 GPU.Photo-to-sketch training can converge in 50 iterations and under one minute.
  • Optimization: Animal changes typically use 2000 iterations followed by StyleCLIP mapper training, taking roughly 6 hours on one NVIDIA V100 GPU.The optimizer is ADAM with learning rate 0.002.
  • Model behavior: Cross-domain identity transfer preserves identities and accessories for both synthesized and real inverted images, using textual or image targets.Cross-domain interpolation is performed by interpolating model weights rather than latent codes.
  • Layer selection: Adaptive layer freezing allows all layers for texture changes, 12 trainable FFHQ layers for small shape changes, and k = 3 for animal modifications.The optimization batch size is 8 and the number of optimization iterations is 1.
  • CLIP models: Using both ViT-32/B and ViT-B/16 CLIP models gives the best results for shape changes by combining global-shape and local-texture focus.For texture changes, the model choice mainly produces minor artistic-style variations.
  • StyleCLIP mapper: A StyleCLIP mapper can induce semantic artifacts such as opened mouths and enlarged tongues, so training adds a loss constraining CLIP embedding norms.The mapper is used to identify latent-space regions matching the target domain.

J. Licenses and data privacy

The supplementary materials document the models, datasets, and licenses used, while noting privacy considerations for the FFHQ biometric face dataset. The accompanying figures provide additional randomly sampled examples and comparisons.

  • Licenses and sources: Tables 4 and 5 list the models, datasets, sources, and licenses used in the work.The tables provide the stated provenance and licensing information for the experimental resources.
  • Data privacy: FFHQ contains biometric face images crawled from Flickr without contacting their owners.The images were uploaded under permissive licenses allowing free use, redistribution, and adaptation for non-commercial purposes.
  • Data privacy: FFHQ curators provide contact details for individuals who want their images removed from the dataset.
  • Additional examples: Additional figures show randomly sampled adaptations from face, church, and dog generators, plus animal transformations and few-shot comparisons.The displayed samples use truncation with ψ = 0.7 where specified.
Loading 2108.00946v2…