Source-linked AI summary
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, Dani Lischinski
TL;DR
StyleGAN manipulation commonly requires manual discovery of semantic directions or annotated data for each desired edit. StyleCLIP combines StyleGAN with CLIP to support text-driven optimization, local mapping, and global style-space directions. The methods demonstrate diverse and fine-grained manipulations across faces, animals, cars, and churches, while remaining limited by the pretrained models’ domains and by difficult drastic transformations.
Problem
Discovering semantically meaningful StyleGAN manipulations typically requires painstaking manual examination or annotated images for each desired manipulation.
Method
StyleCLIP combines StyleGAN and CLIP through latent optimization, a prompt-specific latent mapper, and input-agnostic directions in StyleGAN’s style space.
Results
The methods demonstrate a wide range of unique, fine-grained, and previously un demonstrated text-driven manipulations across human faces, animals, cars, and churches.
Takeaways & Limitations
Text-driven manipulation provides a flexible image-editing tool with controls over edit specificity, manipulation strength, and disentanglement.
Takeaways & Limitations
The method is constrained by the pretrained StyleGAN and CLIP domains, and drastic transformations in visually diverse datasets can be difficult.
Abstract
from arXiv · showhide
Inspired by the ability of StyleGAN to generate highly realistic images in a variety of domains, much recent work has focused on understanding how to use the latent spaces of StyleGAN to manipulate generated and real images. However, discovering semantically meaningful latent manipulations typically involves painstaking human examination of the many degrees of freedom, or an annotated collection of images for each desired manipulation. In this work, we explore leveraging the power of recently introduced Contrastive Language-Image Pre-training (CLIP) models in order to develop a text-based interface for StyleGAN image manipulation that does not require such manual effort. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable text-based manipulation. Finally, we present a method for mapping a text prompts to input-agnostic directions in StyleGAN's style space, enabling interactive text-driven image manipulation. Extensive results and comparisons demonstrate the effectiveness of our approaches.
1. Introduction
StyleCLIP addresses the difficulty of discovering new semantic StyleGAN manipulations by using CLIP to provide text-driven controls without manual discovery or manipulation-specific annotations. It introduces optimization, local mapping, and global style-space directions, demonstrating diverse edits across several image domains.
- Existing StyleGAN controls are restricted to preset semantic directions, so new directions require further manual effort or large annotated datasets.
- StyleCLIP combines CLIP’s text-image representations with StyleGAN to enable text-based semantic manipulation without additional manual control discovery.The approach leverages CLIP’s training on 400 million image-text pairs and the broader visual concepts expressible through natural language.
- Text-guided latent optimization uses CLIP as a loss network to manipulate an image, offering versatility but requiring several minutes per edit.
- A latent residual mapper predicts a local latent-space manipulation step for a specific text prompt and input image.
- A global method maps text prompts to input-agnostic directions in StyleGAN’s style space, controlling manipulation strength and disentanglement.
- The methods produce diverse, fine-grained, and previously un demonstrated manipulations of faces, animals, cars, and churches using pretrained StyleGAN and CLIP models.
2. Related Work
Related work spans multimodal representation learning, text-guided generation, and latent-space manipulation. StyleCLIP differs by combining StyleGAN’s image quality with CLIP’s broad semantics while deriving manipulations directly from text and pretrained supervision.
- Joint representations: CLIP learns a joint language-image embedding that can estimate semantic similarity between text and images and supports zero-shot image classification.
- Text-guided image generation and manipulation: Earlier text-guided generation methods condition GANs on text embeddings, while later approaches add multi-scale generation, attention, or additional supervision.
- Text-guided image generation and manipulation: Text-guided manipulation methods have used encoder-decoder architectures or text-image combination modules, whereas StyleCLIP combines StyleGAN with CLIP’s multi-domain semantics.
- Text-guided image generation and manipulation: Compared with DALL·E’s substantial memory requirement, StyleCLIP is deployable on a single commodity GPU.
- Text-guided image generation and manipulation: TediGAN maps text and images into StyleGAN’s latent space for style-mixed manipulation, while StyleCLIP reports better alignment with driving-text semantics.
- Latent space image manipulation: StyleGAN latent-space methods include end-to-end encoders and latent-path discovery based on annotations or other strategies.
- Latent space image manipulation: StyleCLIP’s optimizer and mapper operate in W+, while its input-agnostic directions operate in StyleSpace S and use only pretrained CLIP supervision.Because CLIP was trained on hundreds of millions of text-image pairs, the approach is described as generic across domains without domain-specific annotation.
3. StyleCLIP Text-Driven Manipulation
StyleCLIP presents three text-driven manipulation schemes that combine StyleGAN with CLIP: per-image optimization, a prompt-specific local mapper, and a global style-space direction. The methods trade versatility, stability, input dependence, and fine-grained disentanglement.
- All three StyleCLIP methods combine StyleGAN’s generative power with CLIP’s joint vision-language representation for text-driven image manipulation.
- Latent optimization: Latent optimization minimizes a CLIP-space loss for each image-prompt pair, but takes several minutes and can be difficult to control.
- Local mapper: A local mapping network predicts a manipulation step in one forward pass after training for a text prompt, with the step varying by starting point in W+.Training takes a few hours but is performed only once per text prompt.
- Global directions: StyleSpace S is used for global directions because it is better suited to fine-grained and disentangled manipulation than W+.
- Global directions: The global-direction method uses a one-time preprocessing stage to produce an input-agnostic direction applicable to different image-prompt pairs.
4. Latent Optimization
Latent optimization edits a StyleGAN image code by minimizing a CLIP-based objective for a user’s text prompt while regularizing similarity to the input. Gradient descent backpropagates through fixed StyleGAN and CLIP components.
- The optimizer starts from a source latent code in StyleGAN’s W+ space and minimizes a loss defined using the text prompt and CLIP embeddings.
- The CLIP term is the cosine distance between CLIP embeddings, while latent-space L2 and identity losses regulate similarity to the input image.
- Optimization uses gradient descent by backpropagating through the pretrained, fixed StyleGAN generator and CLIP image encoder.
- Edits shown after 200-300 iterations include explicit attributes and implicit identity-based changes, with identity-loss weighting adjusted for identity shifts.
5. Latent Mapper
The latent mapper infers text-specific manipulation steps for input images, offering faster editing than per-image latent optimization while preserving unrelated visual attributes. Experiments show identity-preserving hairstyle edits and multi-attribute control, although inferred directions remain similar across inputs.
- Latent Mapper: The mapper is trained for a specific text prompt to infer a manipulation step in W+ space for any input latent image embedding.It uses three fully connected networks corresponding to StyleGAN’s coarse, medium, and fine layer groups.
- Latent Mapper: Mapper variants can train only a subset of the three networks when particular attribute levels should remain fixed.For hairstyle edits, the color-related mapper is omitted so the image color scheme remains unchanged.
- Latent Mapper: The mapper combines CLIP guidance with L2 regularization and an optional identity loss to change prompted attributes while preserving other visual attributes.The losses are combined with weights, and identity loss is omitted when the edit is expected to change identity.
- Results: The mapper preserves identity and most unrelated visual attributes in hairstyle edits while adapting the resulting hair appearance to each individual.The examples use inverted real images and separate mappers for different hairstyle prompts.
- Results: A single mapper controls combined hair attributes, including straight or curly and short or long hairstyles, with the expected outcomes.The paper states that this combined control had not been demonstrated by prior methods known to the authors.
- Results: Despite input-adapted manipulation steps, cosine similarity is high across inputs for a given text prompt, indicating similar directions in practice.The similarity statistics are reported in Table 2.
6. Global Directions
The global-direction method maps text-derived CLIP directions into StyleGAN’s style space by measuring channel relevance. It supports interactive control of manipulation strength and disentanglement, trading stronger effects for more attribute coupling.
- Global Directions: The method seeks a style-space direction ∆s such that G(s + α∆s) introduces or amplifies a prompted attribute without significantly affecting others.The manipulation strength is controlled by α, and the target direction is obtained from CLIP text embeddings.
- Global Directions: Stable CLIP text directions are obtained with prompt engineering that averages embeddings from multiple semantically equivalent templates.The method uses the ImageNet prompt bank and contrasts a target attribute with a corresponding neutral class.
- Global Directions: Channel relevance is estimated by perturbing one style channel at a time and measuring the mean projection of its image-induced CLIP direction onto the target direction.The procedure generates paired images from positive and negative channel perturbations.
- Global Directions: The relevance estimate uses 100 image pairs, with each pair generated by perturbing one style channel by a standard-deviation-sized amount and α = 5.Only the selected channel is nonzero in the perturbation direction.
- Global Directions: Thresholding channel relevance controls disentanglement: higher β yields more disentangled but weaker manipulations, whereas lower β retains correlated attributes.For grey hair, lowering β can also affect wrinkles and face shape.
- Results: The method produces text-driven edits across faces, cars, and dogs, including complex attributes such as facial expressions and hairstyles.The reported face manipulations include 18 attributes using real images embedded in W+ space.
7. Comparisons and Evaluation
The evaluation compares StyleCLIP’s mapper and global-direction methods with prior approaches across facial attributes and other StyleGAN domains. The mapper handles complex, specific edits better, while global directions provide more disentangled results for simpler attributes.
- Evaluation setup: The methods demonstrate edits on real images and across portraits, cars, and dogs using pretrained StyleGAN models.
- Text-driven image manipulation methods: For complex and specific attributes, especially identity-related prompts, the latent mapper produces better manipulations than the global direction.For simpler or more common attributes, the global direction suffices and is more disentangled; TediGAN fails in all three shown manipulations.
- Text-driven image manipulation methods: The global direction removes wrinkles while mostly preserving other attributes, whereas the mapper fails because W+ is less disentangled.
- Other StyleGAN manipulation methods: StyleCLIP’s manipulation remains similar to StyleSpace, changing the target attribute while leaving other attributes unchanged.GANSpace entangles manipulation with skin color and lighting, while InterFaceGAN can significantly change identity during lipstick manipulation.
- Other StyleGAN manipulation methods: StyleCLIP produces results of similar quality to StyleFlow while requiring no extra supervision.StyleFlow uses several attribute classifiers and regressors, limiting it to a restricted set of attributes.
8. Conclusions
The paper combines StyleGAN’s generative power with CLIP’s visual concept encoding to enable varied, fine-grained text-driven image manipulations. It reports controls over manipulation strength and disentanglement, including edits difficult for annotated-data methods.
- The three methods combine StyleGAN’s generative power with CLIP’s visual concept encoding abilities.
- The techniques enable unique manipulations, including some that existing annotated-data methods cannot achieve.
- The approach supports fine-grained controls such as specifying a desired hairstyle, while controlling manipulation strength and disentanglement.
A.1. Architecture
The latent mapper uses separate networks for coarse, medium, and fine StyleGAN style entries, but selected edits can omit networks targeting unaffected attributes. Ablations show architecture choices affect edit quality and identity-related appearance.
- A three-network mapper better supports simultaneous changes at multiple image-detail levels than a single mapping network.The single-network design produces weak expression and hairstyle changes and unnecessary background-color changes.
- The mapper architecture has relatively few parameters and negligible effect on inference time.
- Removing the color-focused network slightly improves hairstyle edits that should not alter the image’s color scheme.The full three-network architecture remains the paper’s described default for simplicity and generalization.
A.2. Losses
The loss ablations show that CLIP supplies a distinctive identity-oriented edit, while identity loss is important for preserving the input person during manipulation.
- CLIP guides a unique celebrity edit that cannot be achieved by using facial-recognition identity loss alone.The comparison uses an image of Beyonce as the identity-loss target.
- Without identity loss, the mapper fails to preserve the input person’s identity and introduces large changes.Increasing λL2 to 1.6 still does not preserve the original identity well enough.
B. Additional Results
The supplementary results show StyleCLIP manipulating faces, expressions, hair, celebrities, and non-face images through latent mappers and global text-driven directions. Comparisons indicate competitive attribute control and visual quality, while drastic cross-domain transformations remain difficult for global directions.
- Latent mapper: The latent mapper produces hair-style, celebrity, and expression edits on inverted CelebA-HQ face images.The supplementary gallery includes hair-style manipulations, celebrity resemblance edits, and expression edits.
- Global directions: Global latent directions enable text-driven edits on non-face datasets, including cats and churches.The examples use StyleGAN2 variants trained on AFHQ cats and LSUN Church.
- Global directions: Increasing manipulation strength makes gender edits stronger, while lowering the disentanglement threshold introduces additional correlated attribute changes.Higher thresholds restrict changes more closely to facial features; lower thresholds allow more correlated attributes to change.
- Comparisons: At matched classifier-logit changes, StyleCLIP avoids some entanglement seen in GANSpace and identity changes seen in InterFaceGAN, resembling StyleSpace control.The comparison covers Gender, Grey hair, and Lipstick, which all evaluated methods can manipulate.
- Comparisons: StyleCLIP achieves visual quality comparable to StyleFlow without extra supervision and compares global directions with the latent mapper across three attribute types.StyleFlow uses multiple Microsoft face API classifiers and regressors, whereas StyleCLIP supports a broader set of demonstrated manipulations without that supervision.
- Limitations: Global directions transform tigers into lions more successfully than into wolves, showing that larger domain gaps and drastic shape changes remain challenging.The tiger-to-lion gap mainly involves color and texture, whereas tiger-to-wolf transformations also require more drastic shape deformations.