Source-linked AI summary
UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition
Jiankang Deng, Shiyang Cheng, Niannan Xue, Yuxiang Zhou, Stefanos Zafeiriou
TL;DR
Facial UV maps extracted from fitted 3D face models are incomplete because of self-occlusion, while pose variation remains a challenge for face recognition. UV-GAN completes these maps with local and global adversarial networks, then uses them to synthesize arbitrary poses for training and testing. The paper reports 94.05% CFP frontal-profile verification accuracy and introduces WildUV with complete maps from 1,892 identities.
Problem
Self-occlusion makes facial UV maps extracted from images incomplete, while limited pose coverage and strong frontal-profile degradation challenge face recognition.
Method
UV-GAN combines local and global adversarial networks for identity-preserving UV completion, then renders arbitrary-pose faces from completed UV maps attached to fitted meshes.
Results
94.05% CFP frontal-profile verification accuracy is reported using pose augmentation during training and pose discrepancy reduction during testing.
Takeaways & Limitations
Completed UV maps support pose variation enrichment for training and pose discrepancy reduction for testing, while WildUV supplies complete in-the-wild maps from 1,892 identities.
Abstract
from arXiv · showhide
Recently proposed robust 3D face alignment methods establish either dense or sparse correspondence between a 3D face model and a 2D facial image. The use of these methods presents new challenges as well as opportunities for facial texture analysis. In particular, by sampling the image using the fitted model, a facial UV can be created. Unfortunately, due to self-occlusion, such a UV map is always incomplete. In this paper, we propose a framework for training Deep Convolutional Neural Network (DCNN) to complete the facial UV map extracted from in-the-wild images. To this end, we first gather complete UV maps by fitting a 3D Morphable Model (3DMM) to various multiview image and video datasets, as well as leveraging on a new 3D dataset with over 3,000 identities. Second, we devise a meticulously designed architecture that combines local and global adversarial DCNNs to learn an identity-preserving facial UV completion model. We demonstrate that by attaching the completed UV to the fitted mesh and generating instances of arbitrary poses, we can increase pose variations for training deep face recognition/verification models, and minimise pose discrepancy during testing, which lead to better performance. Experiments on both controlled and in-the-wild UV datasets prove the effectiveness of our adversarial UV completion model. We achieve state-of-the-art verification accuracy, $94.05\%$, under the CFP frontal-profile protocol only by combining pose augmentation during training and pose discrepancy reduction during testing. We will release the first in-the-wild UV dataset (we refer as WildUV) that comprises of complete facial UV maps from 1,892 identities for research purposes.
1. Introduction
The paper addresses incomplete facial UV maps caused by self-occlusion and proposes UV-GAN to complete them for pose-invariant face recognition. Completed UV maps enable arbitrary-pose synthesis for training augmentation and testing-time pose discrepancy reduction.
- Motivation: Facial UV maps sampled from fitted 3D face models contain missing pixels because of self-occlusion.The paper frames single-image facial UV completion as an understudied problem.
- Motivation: Over 10% performance degradation separates frontal-profile from frontal-frontal verification for most DCNNs.Web images also have long-tailed pose distributions, making comprehensive pose-labeled training data impractical.
- Approach: UV-GAN combines local and global adversarial networks to learn identity-preserving completion of incomplete facial UV maps.The completed UV is attached to the fitted mesh before pose synthesis.
- Approach: Completed UV maps and corresponding 3D shapes synthesize 2D faces at arbitrary poses for training augmentation and testing-time pose discrepancy reduction.The method enlarges pose diversity without extensive manually labeled pose coverage.
- Results: 94.05% verification accuracy is reported under the CFP frontal-profile protocol.The reported result combines pose augmentation during training with pose discrepancy reduction during testing.
- Resources: WildUV provides complete facial UV maps from 1,892 identities for research purposes.The paper describes it as the first in-the-wild UV dataset of this kind.
2. Related Work
Prior image-completion research includes inpainting, texture synthesis, recurrent and convolutional pixel models, and face completion. The paper distinguishes its approach by targeting high-resolution facial UV maps.
- Image Completion: Image completion research spans inpainting and texture synthesis methods designed to restore missing content.Examples include context encoders and multi-scale neural patch synthesis.
- Image Completion: Pixel-RNN and Pixel CNN architectures are described as possible deep-neural-network approaches to image completion.The paper states these architectures are mainly suitable for low-resolution images.
- Image Completion: The closest cited face-completion method combines reconstruction, adversarial, and semantic-parsing losses for local-global consistency.The paper contrasts this prior method with its goal of handling high-resolution facial UV maps.
3. Proposed Approach
UV-GAN completes self-occluded facial UV maps using adversarial generation, global and local discrimination, and identity preservation after 3DMM fitting. The completed textures support realistic arbitrary-pose face synthesis while the fitting method remains accurate across pose groups.
- 3.1. 3D Morphable Model Fitting: 3DMM fitting recovers shape, texture, and camera parameters from 2D facial images to construct facial UV textures.Shape and texture use learned eigenbases and mean models, while camera estimation uses a perspective transformation and image sampling.
- 3.1. 3D Morphable Model Fitting: The fitting method produces clearer mean faces than [33] across 13 CASIA pose groups, indicating more accurate estimated poses.The comparison reports that the baseline becomes blurrier as yaw increases, whereas the proposed means remain clear.
- 3.2. UV Texture Completion: UV-GAN uses an auto-encoder generator to complete missing UV regions with pixel-wise l1 reconstruction while preserving image information through skip connections.Incomplete UV textures are filled with random noise and concatenated with their mirror image; symmetry loss is avoided because faces are not exactly symmetric.
- 3.2. UV Texture Completion: Global and local discriminators improve completion by enforcing realistic full-map context and sharper, more informative central-face texture.The local discriminator focuses on the less noisy and more identity-informative face centre, while the global discriminator evaluates the entire UV map.
- 3.2. UV Texture Completion: A fixed pre-trained ResNet-27 identity embedding and centre loss constrain generated textures to preserve facial identity.The embedding network was trained with softmax loss to classify 10k CASIA identities, and generated samples are encouraged to remain near their class feature centres.
- 3.2. UV Texture Completion: The complete UV-GAN objective combines reconstruction, global adversarial, local adversarial, and identity losses as a weighted sum.Only the generator is required during testing; the discriminators and fixed identity network support training.
4. Experiments
Experiments evaluate UV completion on controlled and in-the-wild datasets, then assess its effect on pose-invariant face recognition. The results show that adversarial completion and UV-based pose augmentation improve completion realism and frontal-profile verification.
- Settings and Datasets: UVDB combines 3dMD, Multi-PIE, and WildUV data, including 3,564 3dMD identities, 337 Multi-PIE identities, and 1,892 WildUV identities.WildUV contains 5,638 unique UV maps from videos with large pose variations; 3dMD and WildUV identities use 90% for training and the remainder for testing.
- UV Completion: PSNR and SSIM measure pixel-level and holistic similarity between predicted UV maps and ground-truth UV maps.Table 1 reports PSNR in dB and SSIM for profile UV completion under different model settings and against Context Encoder.
- UV Completion: Adversarial losses progressively improve profile UV completion, with UV-GAN producing the most realistic and plausible completed content.Reconstruction-only outputs are smooth and blurry; adding flipped faces and global and local adversarial losses improves visual quality, especially in occluded regions.
- UV Completion: UV-GAN produces visually realistic and consistent completions across controlled and in-the-wild faces from 0° to 90° pose changes.Completion quality deteriorates as yaw increases because self-occluded regions become larger, with slight blurriness remaining around self-occluded eyes in profile views.
5. Conclusions
The paper presents facial UV-map completion as an understudied problem and uses identity-preserving global and local adversarial networks to complete UV maps. Completed maps attached to fitted 3D meshes enable arbitrary-pose faces, supporting pose variation during training and reduced pose discrepancy during testing.
- The paper addresses completion of facial UV maps produced by fitting 3D face models to images.
- Global and local adversarial networks learn identity-preserved UV completion from a large-scale dataset of completed facial UV maps.
- Completed UV maps attached to fitted 3D meshes produce faces with arbitrary poses for face-recognition training and testing.
- Experiments on controlled and in-the-wild UV datasets confirm the effectiveness of the proposed UV completion method.
- The method obtains state-of-the-art verification accuracy under the CFP frontal-profile evaluation protocol.