Source-linked AI summary
GAN-Control: Explicitly Controllable GANs
Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, Gerard Medioni
TL;DR
Existing GAN editing methods usually adjust attributes relatively, while explicit face-control methods depend on morphable 3D models. This paper uses contrastive learning to disentangle GAN latent subspaces and trains encoders from human-interpretable attributes to latent vectors, achieving explicit control across faces and other image domains. It reports state-of-the-art qualitative and quantitative performance and extends the framework to real-image projection.
Problem
Prior GAN editing methods provide relative rather than exact attribute control, while explicit face-control methods are constrained by morphable 3D face-model parameters.
Method
Contrastive learning disentangles attribute-specific GAN latent subspaces, after which encoders map human-interpretable attribute values to suitable latent vectors.
Results
The framework achieves state-of-the-art qualitative and quantitative performance while controlling attributes in human faces, painted portraits, and dog images.
Takeaways & Limitations
Explicitly controllable GAN generation can extend beyond human-face 3D-model parameters to additional domains and support real-image editing through disentangled projection.
Takeaways & Limitations
Attribute control depends on differentiable predictors whose embeddings place similar attribute values near each other and different values far apart.
Abstract
from arXiv · showhide
We present a framework for training GANs with explicit control over generated images. We are able to control the generated image by settings exact attributes such as age, pose, expression, etc. Most approaches for editing GAN-generated images achieve partial control by leveraging the latent space disentanglement properties, obtained implicitly after standard GAN training. Such methods are able to change the relative intensity of certain attributes, but not explicitly set their values. Recently proposed methods, designed for explicit control over human faces, harness morphable 3D face models to allow fine-grained control capabilities in GANs. Unlike these methods, our control is not constrained to morphable 3D face model parameters and is extendable beyond the domain of human faces. Using contrastive learning, we obtain GANs with an explicitly disentangled latent space. This disentanglement is utilized to train control-encoders mapping human-interpretable inputs to suitable latent vectors, thus allowing explicit control. In the domain of human faces we demonstrate control over identity, age, pose, expression, hair color and illumination. We also demonstrate control capabilities of our framework in the domains of painted portraits and dog image generation. We demonstrate that our approach achieves state-of-the-art performance both qualitatively and quantitatively.
1. Introduction
The paper targets controllable, photorealistic image generation without restricting control to 3D face-model parameters. It proposes disentangled GAN training, explicit attribute encoders, and real-image editing across multiple domains.
- 1. Introduction: The framework trains high-resolution GANs with explicit control over image attributes and supports real-image editing through disentangled projection.The approach is presented as a unified framework for controllable generation and editing.
- 1. Introduction: The method constructs latent sub-spaces for specific properties and enforces similar or different predicted attributes for matching or differing latent sub-vectors.This training design is used to obtain disentanglement between latent sub-spaces.
- 1. Introduction: The paper demonstrates explicitly controllable generation in facial photographs, painted portraits, and dog images.The framework is described as applicable across these three domains.
- 1. Introduction: The approach is presented as state-of-the-art for explicitly controllable, high-resolution GAN training.
- 1. Introduction: Unlike 3D-model-guided methods, the approach extends control beyond attributes supported by graphical modeling and rendering frameworks.This broader scope enables application to additional domains.
2. Related work
Prior work controls GAN outputs either relatively through latent-space manipulation or explicitly through conditioning and 3D face models. These approaches face limitations in continuous or multiple-attribute control, domain coverage, photorealism, variability, or dependence on handcrafted 3D assets.
- 2. Related work: Relative latent-space methods adjust attribute intensity but do not provide explicit settings for multiple continuous attributes.They identify semantic directions or boundaries using latent-space structure and classifiers.
- 2. Related work: Conditional GANs generally support a single discrete class variable and do not address controlling multiple attributes simultaneously.
- 2. Related work: The paper focuses on controllable de novo generation while also enabling real-image editing through projection into GAN latent space.
3. Proposed approach
The framework creates explicitly disentangled GAN latent sub-spaces using contrastive learning, then maps human-interpretable attribute values to those sub-spaces for controllable synthesis.
- 3.1. Disentanglement by contrastive learning: Contrastive learning divides the latent space into sub-spaces that encode separate image properties, producing disentanglement across attributes.Training pairs share one attribute sub-vector while differing in the others, and attribute-specific losses encourage corresponding image similarities and dissimilarities.
- 3.1. Disentanglement by contrastive learning: The StyleGAN2-based architecture assigns each controllable property its own latent sub-space and multilayer perceptron before combining the sub-vectors for generation.An additional sub-space encodes other image properties that are not controllable.
- 3.1. Disentanglement by contrastive learning: Differentiable attribute models define embedding distances between generated images, providing the supervision used to measure similarity for properties such as identity, pose, and age.The framework can use classification- or regression-trained models and distance metrics such as L1, L2, or cosine distance.
- 3.2. Interpretable explicit control: For explicit control, an encoder maps each human-interpretable attribute value, such as age or pose, to its corresponding intermediate latent sub-vector.Separate encoders are trained for the attributes, and their outputs can be combined with sampled sub-vectors at inference time.
- 3.2. Interpretable explicit control: The encoders are trained from generated images whose attribute values are predicted by the corresponding pretrained attribute models, forming one training dataset per attribute.The authors report that this simple encoding scheme does not compromise control accuracy compared with other methods.
4. Experiments
Experiments evaluate controllability, photorealism, control precision, identity preservation, and disentanglement across face and painting generation. The results show explicit attribute control, strong perceived disentanglement, and competitive or improved performance across several measures, with a limitation for expression control on paintings.
- Face generation: 47% of users selected the generated image as real when comparing it with FFHQ, while the method achieved the highest photorealism in synthetic-image comparisons.The study used triplets of synthetic images and pairs containing one synthetic and one FFHQ image.
- Face generation: Control precision was comparable to or better than competing methods when measured by distances between requested and generated attribute values.The evaluation generated 10K attribute-controlled images from feasible FFHQ attribute values and compared average control distances.
- Face generation: Users perceived the generated attributes as disentangled while judging other attributes to be preserved across 1,300 image pairs.Fifty users evaluated attribute presence or absence and preservation of non-target attributes on a five-level scale.
- Face generation: The model provides explicit visual control over age, pose, illumination, expression, and hair color through separate control encoders.Age changes also tend to produce glasses, formal clothing, graying hair, and wrinkles.
- Painting generation: In painted portraits, the method controls age, pose, and expression and can alter artistic style without changing other attributes, although expression control is limited by low expression diversity.The results also indicate that control networks can tolerate some domain gap between their training domain and the GAN’s domain.
- Ablation study: Identity preservation remains strong under changes to pose, illumination, expression, and even age, while ablations show disentanglement is important for preserving identity and control precision.The two-phased model outperformed end-to-end alternatives across the evaluated trade-offs, whereas the NoDis variant did not preserve identity.
5. Conclusions
The paper concludes that disentangled GAN training enables explicit control over generation attributes across multiple domains, while projection extends this control to real-image editing.
- Disentangled GAN training enables explicit control over generation attributes.
- The framework extends explicitly controllable GANs beyond human faces to additional domains.
- A real-image projection method maps images into a disentangled latent space while maintaining explicit control.