Source-linked AI summary
Towards Open-Set Identity Preserving Face Synthesis
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, Gang Hua
TL;DR
Identity-preserving face synthesis has been limited for identities absent from training data, while labeled face datasets may lack diverse poses and illuminations. The paper proposes a GAN framework that separates identity and attributes, recombines them from input images, and trains without attribute annotations. Experiments report realistic identity-preserving synthesis, broader face applications, and improved fidelity from unlabeled data for identities outside the labeled dataset.
Problem
Identity-preserving face synthesis remains limited for identities absent from training data, whose diverse poses and illuminations are also underrepresented in labeled datasets.
Method
A GAN framework disentangles identity and attributes, recombines vectors from input images, uses asymmetric losses, and extracts attributes without annotations.
Results
Experiments demonstrate realistic identity-preserving synthesis, applications to frontalization, attribute morphing, and adversarial-example detection, and improved fidelity from unlabeled images for identities absent from labeled data.
Takeaways & Limitations
The framework supports identity-preserving face synthesis for subjects outside the labeled training dataset and extends to several related face tasks.
Takeaways & Limitations
The authors leave application to other datasets, including birds, flowers, and rendered chairs, for future work.
Abstract
from arXiv · showhide
We propose a framework based on Generative Adversarial Networks to disentangle the identity and attributes of faces, such that we can conveniently recombine different identities and attributes for identity preserving face synthesis in open domains. Previous identity preserving face synthesis processes are largely confined to synthesizing faces with known identities that are already in the training dataset. To synthesize a face with identity outside the training dataset, our framework requires one input image of that subject to produce an identity vector, and any other input face image to extract an attribute vector capturing, e.g., pose, emotion, illumination, and even the background. We then recombine the identity vector and the attribute vector to synthesize a new face of the subject with the extracted attribute. Our proposed framework does not need to annotate the attributes of faces in any way. It is trained with an asymmetric loss function to better preserve the identity and stabilize the training process. It can also effectively leverage large amounts of unlabeled training face images to further improve the fidelity of the synthesized faces for subjects that are not presented in the labeled training face dataset. Our experiments demonstrate the efficacy of the proposed framework. We also present its usage in a much broader set of applications including face frontalization, face attribute morphing, and face adversarial example detection.
1. Introduction
Identity-preserving face synthesis remains difficult for identities absent from training data. The proposed GAN framework disentangles identity from attributes, recombines them without attribute annotations, and supports several face-synthesis applications.
- Identity-preserving synthesis is especially challenging when the target identity is absent from the training images.
- Previous methods manipulate limited attributes, require full attribute annotations, or cannot synthesize identities outside their training datasets.
- The framework extracts an identity vector from one subject image and an attribute vector from another, then recombines them to synthesize the subject with those attributes.
- Its five end-to-end components are identity and attribute encoders, a generator, a classification network, and a discriminator.
- Attribute extraction requires no attribute annotations and uses reconstruction and KL-divergence losses to enforce attribute information in the attribute representation.
- An asymmetric loss uses cross-entropy for discriminative networks and pairwise feature matching for the generator to better preserve identity and stabilize training.
- The framework can leverage unlabeled face images to improve synthesized-face fidelity for identities absent from labeled training data and supports frontalization, attribute morphing, and adversarial-example detection.
2. Related Work
Related work spans deep generative models, conditional synthesis, face-specific generation, and disentangled representation learning. The paper addresses prior limitations by avoiding attribute annotations and supporting identities outside the training datasets.
- Deep generative-modeling research includes deterministic models, GANs, variational auto-encoders, and autoregressive networks.
- Conditional image-synthesis methods condition generation on discrete labels, reference images, or text sentences.
- Face-generation research includes aging, attribute-driven identity-preserving synthesis, frontalization, and pose manipulation.
- Prior disentangled-representation methods require explicit attribute control or annotations, including fixed attributes in batches or pose labels.
- The proposed framework requires no attribute annotations, disentangles identity and attributes, and synthesizes faces with identities outside the training datasets.
3. Identity Preserving GANs
The framework synthesizes faces by separately encoding a subject's identity and another image's attributes, then recombining them through a GAN. It uses unsupervised attribute learning, asymmetric generator training, and unlabeled data to support identity preservation and broader variation.
- Framework: The network extracts an identity vector from subject image xs and an attribute vector from image xa, then generates a new face from their concatenation.The attribute vector can capture pose, emotion, illumination, and background.
- Framework: Five end-to-end components encode identity and attributes, synthesize images, preserve identity through classification, and distinguish real from generated faces.The components are identity encoder I, attribute encoder A, generator G, classifier C, and discriminator D.
- 3.1. Disentanglement of Identity and Attributes: The identity encoder is trained with identity classification, using same-subject features as an identity representation and the last pooling response as the identity vector.The classification loss is based on P(c|xs), the probability that xs has identity c.
- 3.1. Disentanglement of Identity and Attributes: Attribute extraction is trained without attribute annotations using reconstruction and KL-divergence losses.When xs and xa differ, a low-weight pixel reconstruction loss maintains attributes such as background, illumination, and pose; λ is set to 0.1.
- 3.2. Asymmetric Training for Networks G, C, and D: The asymmetric training method uses discriminator and classifier cross-entropy losses but updates the generator with pairwise feature matching to preserve identity and stabilize training.Directly maximizing the discriminator loss can cause gradient vanishing when real and generated image distributions are separable.
- 3.3. Unsupervised Training: Unlabeled face images increase intra-class and inter-class variation, improving synthesized-face diversity and producing larger pose and expression changes for unseen identities.The framework uses unlabeled images in training while omitting identity-dependent losses when labels are unavailable.
4. Experiments
Experiments evaluate the framework through ablations, identity-preservation tests, qualitative synthesis, frontalization, attribute morphing, and adversarial-example detection. Results support identity preservation for seen and zero-shot identities, while unlabeled data increases generated variation.
- Experimental design: The study combines ablations, quantitative identification tests, and qualitative evaluations across synthesis, frontalization, morphing, and adversarial-example detection.The ablations vary losses, transformation training, and unsupervised learning; identification is evaluated on MS-Celeb-1M and Multi-PIE.
- Quantitative evaluation: All components improve identity-preserving capability, with LGC contributing the most, and generated images achieving comparable top-1 accuracy to real query images.The comparison reports identification results for identities appearing and not appearing in the training datasets.
- Ablation study: The full model preserves attribute details and identity better than ablated variants, while removing transformation training loses emotion details and removing LGD produces blur.Removing LGC causes generated samples to lose identity information; unlabeled learning increases variation, including wider mouth opening.
- KL divergence analysis: KL-trained attribute vectors achieve lower validation top-1 accuracy, indicating they contain less identity information and supporting KL loss for identity removal.The experiment compares models trained with and without KL divergence on FaceScrub.
- Face attribute transformation: For seen identities, the method preserves both identity and attributes during face synthesis, while zero-shot experiments generate high-quality images retaining the given identity and attributes.The zero-shot identities do not appear in the training data.
- Applications: The framework produces identity-preserving frontal faces without pose annotations, retains lighting and skin color, and gradually changes pose, emotion, or lighting through latent-vector interpolation.Attribute morphing uses linear interpolation between two extracted attribute vectors.
- Adversarial-example detection: Using input and reconstructed-image LBP features with a linear SVM achieves 92.41% accuracy when the feature-distance threshold is below 0.4.This experiment applies the framework without modification for face adversarial-example detection.
5. Conclusion
The framework generates realistic, identity-preserving faces even for identities outside the training dataset and supports several related face-analysis tasks. Its applicability beyond faces remains future work.
- The framework generates realistic, identity-preserving face images, including for identities outside the training dataset.
- Experiments show applications in face image frontalization, face attribute morphing, and adversarial example detection in face verification systems.
- The authors identify applying the framework to birds, flowers, and rendered chairs as future work.