Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks

1. Introduction

• The 3DMM texture lacks input-image detail, while high-fidelity UV-map generation methods require a large, laborious-to-capture database that is not publicly available.The paper identifies this data requirement as infeasible for ordinary users.Source: §1 · pp. 1–2
• The proposed method refines the globally reasonable initial 3DMM texture with image details by predicting detailed colors for mesh vertices using GCNs instead of reconstructing a UV map.The stated aim is high-fidelity texture reconstruction from a single image without capturing a large-scale face texture database.Source: §1 · p. 2
• The coarse-to-fine framework combines 3DMM coefficient regression with image-feature extraction, differentiable rendering for self-supervision, and GAN loss for texture refinement.The CNN regresses 3DMM and rendering parameters; a pretrained CNN extracts face features for the GCN.Source: §1 · p. 2
• The authors report that their results outperform state-of-the-art methods in both qualitative and quantitative comparisons.The paper presents this as a contribution of its evaluation.Source: §1 · p. 2

2.1. Morphable 3D Face Models

• The Basel Face Model registers a template mesh to scanned faces using Optimal Step Nonrigid ICP, then applies PCA to create a low-dimensional model; this paper uses it as its 3DMM.The broader 3DMM family represents facial identity, expression, and texture from multiple scans.Source: §2.1 · p. 2

2.2. Fitting a Morphable Face Model

• Classical template fitting is sensitive to input lighting, expression, and pose, while deep-learning regressors still face difficulties under complex in-the-wild conditions.Differentiable rendering compares a rendered face with the input image, enabling unsupervised or weakly supervised training without paired data; this work adopts tf-mesh-renderer [13].Source: §2.2 · pp. 2–3

2.3. Texture of Morphable Face Models

• Deng et al. [11] regress 3DMM shape and texture coefficients, but their generated texture remains limited by the 3DMM texture model; this method uses that scheme for a coarse texture and refines it with image details.The GCN refinement is described as the stage that adds the image details.Source: §2.3 · p. 3

2.4. Graph Convolutional Networks

• Graph-convolution methods provide operations on mesh data, and CoMA [27] motivates this method's use of a learned image-feature embedding decoded into detailed face-mesh vertex colors.ChebyNet [9] approximates spectral filters with truncated Chebyshev polynomials, avoiding Fourier-basis computation.Source: §2.4 · p. 3

3. Approach

• The feature-extraction module uses a Regressor for 3DMM coefficients, pose, and lighting, and FaceNet for image features used in detail refinement and identity preservation.The outputs support the subsequent texture-refinement module.Source: §3 · p. 4
• The texture-refinement module uses a GCN Decoder to generate detailed vertex colors from FaceNet features and a GCN Refiner to refine Regressor-generated vertex colors.A combiner merges the two color outputs into final vertex colors.Source: §3 · p. 4
• A Discriminator applies adversarial training to improve the texture-refinement module's output.Source: §3 · p. 4

3.1. 3DMM Coefficients Regression

• The CNN regresses a 257-dimensional vector containing identity, expression, and texture coefficients, face pose, and lighting; the coefficients yield face positions and albedo.The vector contains 80 identity, 64 expression, 80 texture, 6 pose, and 27 lighting values.Source: §3.1 · p. 4
• A pretrained FaceNet embedding serves both as input for the Decoder to generate detailed mesh-vertex albedo and as the feature representation for identity-preserving loss.Source: §3.1 · p. 4

3.2. Differentiable Rendering

• The mesh is rendered with predicted parameters, and losses are computed from differences between the rendered image and the input image.The renderer interpolates mesh colors and normals at pixels.Source: §3.2 · p. 4
• The regressed shape, texture, and pose produce a projected face albedo, which is illuminated with estimated lighting to form the final rendered image.That rendered image is compared with the input to compute the loss.Source: §3.2 · p. 4
• Figure 3 presents the input image, the render with albedo only, and the render with illuminated texture, in that order.Source: Figure 3 · p. 4

3.3. Texture Refinement Module

• The module directly refines mesh-vertex albedo RGB values instead of representing facial texture as a UV map.It comprises a Decoder, a Refiner, and a combine net.Source: §3.3 · p. 4
• Chebyshev-polynomial spectral convolutions provide mesh filtering using a kernel over the scaled Laplacian.The convolution layer maps input features to output features with trainable parameters.Source: §3.3 · pp. 4–5
• The Decoder maps FaceNet features to vertex albedo colors, while the Refiner updates vertex colors from the 3DMM model.The Decoder uses spectral residual blocks; the Refiner uses similar blocks with downsampling and upsampling layers.Source: §3.3 · p. 5
• The combine net concatenates Decoder and Refiner outputs, then applies a graph convolutional layer and tanh activation to produce final vertex colors.Source: §3.3 · p. 5
• A discriminator is trained on input 2D images and rendered reconstructed faces as part of the adversarial strategy.The paper notes that it does not use real 3D face data and follows Wasserstein GAN training with gradient penalty.Source: §3.3 · p. 5

3.4.1 Pixel-wise Loss

• Pixel-wise loss compares the input and illuminated rendering only over segmented face regions, excluding areas outside the projected mesh.The regions include the face, eyebrows, eyes, nose, and mouth; segmentation follows a pretrained network.Source: §3.4.1 · p. 5, Figure 4 · p. 5

3.4.2 Identity-Preserving Loss

• Identity-preserving loss uses cosine distance to keep FaceNet features of the input and rendered images close, addressing cases where pixel loss may not preserve likeness.The feature vectors are extracted from the input and rendered images by FaceNet.Source: §3.4.2 · p. 5

3.4.3 Vertex-wise Loss

• Vertex-wise loss uses regressor-predicted texture or image-projected vertex colors as assistance, gradually reducing its weight during refinement training.It compares those albedo values with refined or illuminated albedo values across vertices.Source: §3.4.3 · pp. 5–6

3.4.4 Adversarial Loss

• The adversarial loss follows Improved Wasserstein GAN, using input images, rendered refined-texture images, and interpolated samples for its gradient penalty.The interpolated samples lie along straight lines between samples from the input and rendered-image distributions.Source: §3.4.4 · p. 6

4. Implementation Details

• The implementation uses aligned CelebA images, the Basel Face Model, and a regressor that predicts 3DMM coefficients, pose, and lighting.Inputs are 224 × 224 pixels, and meshes have 35,709 vertices.Source: §4 · p. 6
• During refinement training, σ1 starts at 0 and σ4 at 1, then after one warm-up epoch σ1 rises to 1 while σ4 falls to 0.σ2 = 0.2 and σ3 = 0.001 remain fixed.Source: §4 · p. 6
• At inference, the network processes about 50 images per second in parallel on an NVIDIA 1080Ti GPU.Source: §4 · p. 6

5.1. Qualitative Comparison

• Compared with [7] [11] [12] [13], our results show more detailed information, while [7] emphasizes local shape detail and [12] can lack lighting and holistic-color diversity.The authors also describe [11] as producing better results under complex lighting than other methods.Source: §5.1 · p. 6

5.2. Quantitative Comparison

• Evaluation compares input images with projected face-region renderings because the authors lack 3D ground-truth data.Metrics include L1, PSNR, SSIM, face-recognition feature similarity, and a distribution comparison on MICC; the authors note that MICC mesh coverage differs from [13] and may lower their scores.Source: §5.2 · pp. 6–7
• Our reconstructions outperform prior methods across multiple evaluation metrics, with both face-recognition networks finding them more likely to depict the same person as the input.The comparison uses pixel-level and perceptual metrics; the supplied table caption identifies Table 1 as the quantitative comparison.Source: §5.2 · p. 7, Table 1 · p. 8

5.3. Ablation Study

• The ablation describes progressively more detailed, realistic predictions as the model moves beyond coarse 3DMM texture, adds adversarial training, and incorporates vertex-level refinement.Replacing GCNs with fully connected or UV-space convolutional layers does not improve performance; the authors attribute this to parameter count and underuse of 3D facial spatial information.Source: §5.3 · p. 8, Table 2 · p. 8
• Training and testing on higher-resolution CelebA-HQ images helps reduce checkerboard-like artifacts and produce better results.Figure 8 compares results from 224 × 224 and 512 × 512 inputs.Source: §5.3 · p. 8, Figure 8 · p. 8

• The method reconstructs face shapes with high-fidelity textures from single-view images.Source: §6 · p. 8
• The authors report using graph convolutional networks to generate high-fidelity face textures without capturing high-resolution face-texture datasets.Source: §6 · p. 8
• Future work will target checkerboard-like artifacts and more detailed face shapes and expressions.Source: §6 · p. 8