PaperlayerView exact revision on arXiv

Source-linked AI summary

GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction

Baris Gecer, Stylianos Ploumpis, Irene Kotsia, Stefanos Zafeiriou

arXiv:1902.05978v2cs.CV

TL;DR

Existing single-image 3D face reconstruction methods do not model facial textures in high fidelity. GANFIT uses a GAN-generated UV texture prior with optimization-based 3DMM fitting, differentiable rendering, and deep identity supervision, achieving photorealistic, identity-preserving reconstructions with high-frequency details.

  • Problem

    Existing single-image 3D face reconstruction methods remain unable to model facial textures in high fidelity, limiting detailed texture reconstruction.

  • Method

    GANFIT combines a GAN-based UV facial-texture generator with optimization-based 3DMM fitting, differentiable rendering, and deep identity-feature supervision.

  • Results

    GANFIT achieves high-fidelity 3D face reconstructions that are photorealistic and identity preserving, including high-frequency facial texture details.

  • Takeaways & Limitations

    GANs can serve as a powerful statistical texture prior for high-quality, identity-preserving 3D face reconstruction.

  • Takeaways & Limitations

    The fitting formulation inherits a limitation of prior statistical fitting approaches: the conventional optimization baseline does not reconstruct facial texture.

Abstract

from arXiv · show

In the past few years, a lot of work has been done towards reconstructing the 3D facial structure from single images by capitalizing on the power of Deep Convolutional Neural Networks (DCNNs). In the most recent works, differentiable renderers were employed in order to learn the relationship between the facial identity features and the parameters of a 3D morphable model for shape and texture. The texture features either correspond to components of a linear texture space or are learned by auto-encoders directly from in-the-wild images. In all cases, the quality of the facial texture reconstruction of the state-of-the-art methods is still not capable of modeling textures in high fidelity. In this paper, we take a radically different approach and harness the power of Generative Adversarial Networks (GANs) and DCNNs in order to reconstruct the facial texture and shape from single images. That is, we utilize GANs to train a very powerful generator of facial texture in UV space. Then, we revisit the original 3D Morphable Models (3DMMs) fitting approaches making use of non-linear optimization to find the optimal latent parameters that best reconstruct the test image but under a new perspective. We optimize the parameters with the supervision of pretrained deep identity features through our end-to-end differentiable framework. We demonstrate excellent results in photorealistic and identity preserving 3D face reconstructions and achieve for the first time, to the best of our knowledge, facial texture reconstruction with high-frequency details.

1. Introduction

Single-image 3D face reconstruction remains limited by texture models that miss high-frequency detail. GANFIT addresses this by combining a GAN-based UV texture prior, differentiable rendering, and deep identity supervision for photorealistic, identity-preserving reconstructions.

  • 3D face reconstruction from single images estimates facial surface and intrinsic components such as albedo for applications including recognition, editing, and virtual reality.
  • Traditional 3DMM-based methods can estimate shape under unconstrained conditions but may reconstruct only shape rather than facial texture.
  • Recent corrective-based methods add detail but remain unable to capture high-frequency texture details and can bake outliers such as glasses into shape and texture.
  • GANFIT uses GANs as a statistical parametric texture representation and revisits optimization-based 3DMM fitting under deep identity-feature supervision.
  • The authors report high-resolution complete facial-surface reconstruction in unwrapped UV space and photorealistic, identity-preserving shape and texture results under arbitrary recording conditions.
  • The method combines a differentiable-renderer-based 3DMM fitting strategy with content losses defined on deep identity features from a face-recognition network.

2. History of 3DMM Fitting

The paper situates its approach within 3DMM fitting, whose linear texture models struggle with in-the-wild and high-frequency facial detail. It replaces conventional texture modeling and fitting components with GAN-based UV textures, identity-feature losses, and differentiable rendering.

  • 3DMM foundations: 3DMMs represent facial shape and texture statistically, traditionally using PCA-based linear models and UV maps for texture correspondence.UV coordinates provide per-pixel alignment across textures, while shape models separately capture identity and expression variations.
  • Recent fitting methods: Recent DCNN and differentiable-renderer methods regress or fit 3DMM parameters from images or identity features, extending reconstruction beyond classical optimization.Some approaches use synthesized images, image-to-image translation, or additional corrective networks.
  • Texture limitations: Recent statistical texture models remain limited to low-frequency components, preventing high-frequency facial details and many identity characteristics from being reconstructed.Corrective representations can also produce outliers by allowing reconstructions outside the 3DMM space.
  • Proposed novelties: The proposed strategy uses a GAN on high-resolution UV maps as the facial texture representation, enabling reconstruction of high-frequency details.This replaces the conventional 3DMM texture model with a learned generator.
  • Proposed novelties: The fitting objective uses feature losses from a face-recognition embedding network and optimizes latent parameters through a differentiable renderer with gradient descent.The authors contrast this with low-level RGB or edge losses and handcrafted features such as SIFT.

3. Approach

The approach reconstructs a face from one in-the-wild image by jointly fitting a 3DMM shape model and a high-fidelity texture generator. A differentiable renderer connects these components to image-based cost functions optimized over their latent parameters.

  • Approach: The reconstruction mesh is formed by a 3D morphable shape model, textured with a generator-produced UV map, and projected into the image plane by a differentiable renderer.The system treats the shape-model and texture-network latent parameters as optimization variables.
  • Approach: Gradient descent updates the 3DMM and texture-network latent parameters by minimizing distances between the rendered reconstruction and the input image.The paper formulates these distances using multiple cost functions, primarily based on deep identity features.
  • Approach: The method is organized around a progressive-growing GAN texture model, differentiable image formation, and fitting procedures for the shape and texture models.These components are introduced in the subsequent sections.

3.1. GAN Texture Model

The GAN texture model addresses PCA’s tendency to produce blurry, low-frequency textures by learning a high-resolution, per-pixel-aligned UV texture distribution. Its latent generator is then fit to target UV textures through optimization.

  • Motivation: Conventional PCA models often miss high-frequency details and produce blurry textures because of their Gaussian modeling assumptions.This limitation is especially important because texture contributes to identity preservation and photo-realism.
  • GAN texture representation: A GAN trained on per-pixel-aligned UV representations avoids the 3D incoherence associated with semi-aligned training images and generates realistic, coherent UVs from 99.9% of its latent space.The authors also report generalization to unseen data.
  • GAN texture representation: The authors train a progressive-growing GAN on 10,000 high-resolution texture UV maps and use its generator as the statistical texture model.The generator replaces the conventional 3DMM texture model.
  • Texture fitting: Fitting the generator minimizes per-pixel Manhattan distance between a target UV texture and the generator output with respect to the texture latent parameter.This turns texture fitting into nonlinear optimization over the generator’s latent space.

3.2. Differentiable Renderer

The differentiable renderer projects the reconstructed 3D mesh into a 2D image using camera and illumination parameters while preserving gradients to the latent variables. It supports joint optimization of geometry, texture, camera, and lighting.

  • Differentiable rendering: The renderer uses deferred shading and interpolates vertex colors and normals with barycentric coordinates, allowing gradients to backpropagate to latent parameters.This makes gradient-based fitting possible through the rendering operation.
  • Camera model: A pinhole camera projects the textured mesh into the 2D image plane using camera position, viewing direction, and focal length.These camera parameters are part of the rendered-image formation process.
  • Illumination model: Phong shading models illumination with a direct colored light source and ambient-light color parameters.The lighting parameters are optimized alongside geometry, texture, and camera parameters.
  • Rendered image: The rendered image combines 3DMM geometry, GAN-generated texture, camera parameters, and lighting parameters, with UV samples converted into color vectors.The renderer constructs the shape from the 3DMM and the texture from the GAN generator.
  • Variation rendering: A secondary rendering samples expression, camera, and illumination variations while preserving identity and texture parameters.Expression parameters are sampled from a normal distribution, while camera and illumination parameters follow distributions from the 300W-3D dataset.

3.3. Cost Functions

The approach jointly optimizes reconstruction parameters using pixel, identity, content, and alignment-related signals. Deep face-recognition features provide identity supervision, while landmark alignment and pixel differences support geometric and illumination fitting.

  • Optimization: The method optimizes reconstruction parameters by backpropagating weighted errors through an end-to-end differentiable framework.Parameters are updated with gradient descent from the current reconstruction cost terms.
  • Identity loss: A pretrained face-recognition network supplies identity features for optimizing latent reconstruction parameters.The network maps aligned face images to 512-dimensional embeddings, whose cosine distance contributes to identity supervision.
  • Identity loss: An additional identity loss renders the reconstruction under random pose, expression, and lighting to preserve identity across conditions.This loss replaces the standard rendered image with a randomly conditioned rendering.
  • Identity loss: Intermediate face-recognition representations complement final-layer identity features because they retain mid-level variations useful for 3D reconstruction.The paper specifically notes that final activations discard some age-dependent information.
  • Pixel loss: Pixel-level ℓ1 loss directly guides lighting optimization and also helps recover albedo and illumination.The input and rendered images are downscaled to 112 × 112 before identity and content losses.
  • Alignment: Landmark alignment uses Euclidean distances between detected input landmarks and projected reconstruction landmarks to guide camera and geometry alignment.The landmarks are detected by a deep face-alignment network, and their positions depend strongly on camera parameters.

3.4. Model Fitting

Model fitting first performs a landmark-based rough alignment, then jointly optimizes the reconstruction parameters with weighted losses and regularization. The framework can also exploit multiple video frames by averaging identity-related parameters during optimization.

  • Single-image fitting: Rough fitting first optimizes shape, expression, and camera parameters using landmark alignment alone.This initializes the reconstruction before the full joint optimization.
  • Single-image fitting: The subsequent stage jointly minimizes a weighted combination of loss terms with gradient descent and backpropagation.The loss weights determine the contribution of the individual fitting objectives.
  • Single-image fitting: Regularization constrains shape, expression, and lighting parameters to prevent exaggerated values from arbitrarily biasing the losses.The regularizer is applied to the shape, expression, and lighting parameter groups.
  • Multiple-image fitting: For video, the method averages identity reconstruction parameters after every iteration instead of averaging completed reconstructions or regressed features.Using more images can improve reconstruction quality under challenging conditions such as outdoor or low-resolution recordings.

4. Experiments

Experiments evaluate qualitative reconstruction, implementation choices, 3D shape accuracy, and loss-component contributions. Results show high-fidelity identity-preserving reconstructions, improved MICC shape error, and complementary effects from the proposed components.

  • Implementation Details: The implementation uses ArcFace identity features, a progressive-growing GAN trained on about 10,000 512 × 512 UV maps, and the Large Scale Face Model.Faces are aligned using 68 detected landmarks before reconstruction.
  • Qualitative Comparison: Qualitative MoFA-Test comparisons show high-fidelity photorealistic textures and identity characteristics in both texture and shape reconstructions.Reported identity characteristics include eyebrow geometry and wrinkles around the mouth and forehead.
  • 3D Shape Recovery: 36% lower absolute point-to-plane error is reported on the MICC dataset compared with two other state-of-the-art methods.The results remain consistent across cooperative, indoor, and outdoor settings with minimal standard deviation.
  • Ablation Study: The full model reconstructs input faces better than leave-one-out variants, indicating contributions from the included components.The ablation also reports disentanglement of albedo from illumination and accurate light-direction capture.
  • Ablation Study: Identity terms contribute to identity preservation, identity features are significant overall, pixel intensities improve albedo and illumination, and GAN textures outperform PCA-based textures.These findings come from separate leave-one-out ablations in Figure 5.

5. Conclusion

The paper reframes optimization-based 3D face reconstruction by using GANs as a statistical texture model and face-recognition networks as the energy function. It reports high-quality, identity-preserving reconstructions with high-fidelity texture.

  • Conclusion: The method combines GAN-based statistical texture modeling with face-recognition-network-based energy optimization.This revisits optimization-based 3D face reconstruction under a new perspective.
  • Conclusion: The authors report identity-preserving, high-fidelity 3D reconstructions in qualitative and quantitative experiments.They characterize GAN-based model fitting as producing high-quality texture reconstruction.

Appendix A. Experiments on LFW

The LFW experiments evaluate whether reconstructed renderings preserve facial identity using VGG-Face embeddings and cosine similarity. The method also shows robustness across challenging image conditions and qualitative comparisons.

  • Identity preservation: The LFW evaluation compares cosine similarity between real images and rendered reconstructions, plus same- versus different-identity rendering pairs, using VGG-Face embeddings.Camera and lighting parameters are fixed for all renderings.
  • Identity preservation: More than 0.5 average rendering-to-photo cosine similarity is achieved by the method, compared with 0.35 for [16] and 0.16 for [42].The distributions are computed from VGG-Face embedding-layer activations.
  • Identity preservation: Same- and different-identity cosine-similarity distributions are separable by thresholding, indicating identity preservation in the rendered reconstructions.Camera and lighting parameters remain fixed across renderings.
  • Robustness: On BAM, the method is reported as robust to many image deformations and capable of recovering identities from paintings.The comparison is made against [16].
  • Qualitative reconstruction: Qualitative comparisons report high-fidelity texture with accurate shape, camera, and illumination fitting.The comparison overlays reconstructions on input images.

Appendix B. More Qualitative Results

Additional qualitative results examine reconstruction under special-rendering comparisons and difficult imaging conditions. The figures cover lighting, self-occlusion, facial hair, and comparisons with other methods.

  • Qualitative results: Figures 8–11 present reconstructions under different settings and compare them with other state-of-the-art methods.The paper directs readers to the figure captions for detailed explanations.
  • Comparisons: Figure 10 compares texture maps and whole or partial face renderings with [34].The caption notes that [34] uses the commercial Arnold renderer, whereas the method does not require a particular renderer for special effects.
  • Challenging conditions: Figure 11 evaluates results under strong illumination, self-occlusion, and facial hair.The figure includes input images, fitting overlays with and without illumination estimation, pixel-wise intensity differences, and estimated shape meshes.
Loading 1902.05978v2…