PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthesizing Normalized Faces from Facial Identity Features

Forrester Cole, David Belanger, Dilip Krishnan, Aaron Sarna, Inbar Mosseri, William T. Freeman

arXiv:1701.04851v4cs.CVstat.ML

TL;DR

The paper addresses how to reconstruct a normalized face from identity features despite the inversion being highly underconstrained. It predicts geometry and texture separately from invariant facial-recognition features, then combines them with differentiable warping. The resulting faces are robust to nuisance variation and support applications such as 3-D avatar creation and white balancing.

  • Problem

    Mapping facial identity features back to images is highly underconstrained because the output image has 150× more dimensions than the FaceNet feature vector.

  • Method

    The method predicts facial landmarks and texture maps from identity features, combines them with differentiable spline warping, and trains on normalized face images.

  • Results

    The decoder is robust to lighting, pose, expression, and occlusion variation, and can reconstruct plausible faces from monochrome photographs and paintings.

  • Takeaways & Limitations

    Normalized outputs support downstream applications including automatic white balancing and custom 3-D avatar creation.

  • Takeaways & Limitations

    Image quality is limited by background overfitting and pixel-level squared-error training, which can produce noise artifacts and blurriness.

Abstract

from arXiv · show

We present a method for synthesizing a frontal, neutral-expression image of a person's face given an input face photograph. This is achieved by learning to generate facial landmarks and textures from features extracted from a facial-recognition network. Unlike previous approaches, our encoding feature vector is largely invariant to lighting, pose, and facial expression. Exploiting this invariance, we train our decoder network using only frontal, neutral-expression photographs. Since these photographs are well aligned, we can decompose them into a sparse set of landmark points and aligned texture maps. The decoder then predicts landmarks and textures independently and combines them using a differentiable image warping operation. The resulting images can be used for a number of applications, such as analyzing facial attributes, exposure and white balance adjustment, or creating a 3-D avatar.

1. Introduction

The paper maps facial identity features to normalized, front-facing, neutral-expression faces by exploiting their invariance to pose, lighting, and expression. Its decoder separates geometry and texture, enabling robust reconstructions and applications including 3-D avatar fitting and color correction.

  • 1. Introduction: The method converts identity features into evenly lit, front-facing, neutral-expression images, reducing an underconstrained inversion problem to normalized face synthesis.The output has far more dimensions than the input feature vector, so normalization supplies a constrained target.
  • 1. Introduction: The decoder is trained on paired identity features and normalized faces, with FaceNet features providing the strongest reported results and VGG-Face producing similar results.The approach treats the identity-to-normalized-face mapping as nearly one-to-one.
  • 1. Introduction: The system remains robust to occlusion, lighting, and pose variation, and also operates on monochrome photographs and paintings.This robustness distinguishes it from direct frontalization methods that cannot compensate for occlusion or lighting variation.
  • 1. Introduction: The resulting normalized faces support 3-D morphable-model fitting, automatic color correction and white balancing, and visualization of facial-recognition features.Data augmentation enables high-quality training with only 1K unique input images.
  • 1. Introduction: The decoder separately predicts facial landmarks and texture, then uses differentiable warping to combine them into the final image.This geometry–texture decomposition is central to the model design.
  • 1. Introduction: Geometry–texture decomposition, data augmentation, and differentiable warping are presented as techniques applicable beyond face normalization.

2. Background and Related Work

Prior methods invert deep features through pixel optimization, feed-forward reconstruction, or direct face warping, while this work uses normalized face synthesis with explicit geometry–texture modeling. Its approach builds on landmark-based alignment and identity-preserving feature representations.

  • 2. Background and Related Work: Pixel-level inversion methods require strong regularization because image space is much larger than feature space, and their outputs are intriguing but not realistic.Examples include total variation and Gaussian blur regularizers.
  • 2. Background and Related Work: Feed-forward embedding inversion reconstructs likely images, whereas this method imposes the more restrictive target of a normalized face.
  • 2. Background and Related Work: Earlier FaceNet inversion methods require no training data, but the paper reports better fine-grained details from its trained approach.
  • 2. Background and Related Work: Active appearance and morphable-face models decouple texture and landmark geometry, align images by warping landmarks to mean locations, and reconstruct by reversing that process.Separate texture and geometry models can be fit reliably with substantially less data than models of raw images.
  • 2. Background and Related Work: FaceNet embeddings are trained to preserve identity while ignoring capture-specific variation such as lighting, expression, and pose, making them suitable inputs and training targets.The method also produces similar results from VGG-Face features.
  • 2. Background and Related Work: Non-parametric frontalization is largely parameter-free and needs no labeled training data, but does not normalize lighting, expression, or occlusion variation.

3. Autoencoder Model

The autoencoder maps wild face images to landmark and texture estimates using fixed FaceNet features, separate prediction branches, and differentiable warping. Training combines explicit geometry, texture, and perceptual identity losses.

  • 3. Autoencoder Model: The model assumes front-facing, neutral-expression training images decomposed into texture maps and landmarks, then predicts those quantities directly from wild input images at test time.
  • 3. Autoencoder Model: A pretrained, fixed FaceNet encoder supplies identity-robust features, with a learned fully connected layer reducing the 1024-D avgpool representation to the decoder feature dimension.VGG-Face uses its 4096-D fc7 layer instead.
  • 3. Autoencoder Model: The architecture separately generates landmarks with an MLP and registered textures with a deep CNN, then renders the image through differentiable warping.The separate branches avoid simultaneously modeling geometry and texture in a single direct image decoder.
  • 3. Autoencoder Model: The texture branch maps features through localized representations and transposed convolutions, while a fully connected alternative directly predicts 224 × 224 × 3 pixels.
  • 3.3. Training Loss: Training penalizes landmark mean squared error and texture mean absolute error separately, rather than relying only on final-image reconstruction error.Separate losses avoid charging correct color predictions for errors caused by incorrect landmark locations.
  • 3.3. Training Loss: A fixed FaceNet loss additionally penalizes negative cosine similarity between input and generated-image embeddings to preserve perceptually relevant identity details.This objective increases computational cost because it requires differentiable warping during training.

4. Differentiable Image Warping

The method warps a texture image according to sparse landmark displacements by interpolating them into a dense flow field and applying that field differentiably.

  • The warping pipeline converts sparse landmark displacements into a dense flow field and applies it to the texture image.Spline interpolation produces the dense field, while bilinear sampling applies it differentiably.
  • Boundary control points enforce zero displacement outside the face while landmark points define the geometric transformation.Horizontal and vertical displacement fields are interpolated independently.
  • 4.1. Differentiable Spline Interpolation: Polyharmonic interpolation fits control-point values while minimizing a curvature criterion, with parameters obtained through differentiable operations.The implementation uses a linear radial basis function because it is more robust to overshooting than thin-plate splines.
  • 4.1. Differentiable Spline Interpolation: Algorithm 1 evaluates the fitted interpolant at query points using control points, function values, a radial basis function, and query locations.

5. Data Augmentation using Random Morphs

The augmentation method creates plausible new faces by independently interpolating landmarks and textures of neighboring training faces, then composites the morphed face onto an original background.

  • 5.1. Producing random face morphs: The method augments limited training data by morphing each seed face with a randomly selected nearby face.The target is sampled from the 200 nearest neighbors, and landmark and texture interpolation weights are drawn uniformly from [0, 1].
  • 5.1. Producing random face morphs: Landmarks and textures are interpolated independently, preserving the paper’s geometry–texture decomposition during augmentation.
  • 5.1. Producing random face morphs: Figure 6 contrasts average individual faces with random morphs produced from other individuals in the training set.
  • 5.2. Gradient-domain Compositing: Gradient-domain compositing improves realism by combining the morphed face with an original background before final landmark warping.The output satisfies gradient and color constraints using a softened convex-hull blending mask and least-squares solution.

6. Training Data

The training set is built by filtering a large face database, aligning the remaining images, and averaging multiple photographs per identity to reduce lighting variation.

  • 6.1. Collecting photographs: After filtering the 2.6M-image source, approximately 12K aligned 224 × 224 images remain, representing fewer than 0.5% of the original set.Filtering removes monochrome, blurry, highly emotional, eyeglass-wearing, and excessively tilted or panned faces.
  • 6.2. Averaging to reduce lighting variation: Averaging images with varied lighting and color tends to produce evenly lit results that retain facial detail.
  • 6.2. Averaging to reduce lighting variation: Averaging at least three aligned photographs per identity yields approximately 1K unique identities with more consistent lighting.Each image is warped to the individual’s average landmarks before pixel averaging.
  • 6.2. Averaging to reduce lighting variation: Highly variable training backgrounds can produce noisy backgrounds in the synthesized results.The paper suggests manual background removal as a possible way to obtain cleaner results.

7. Experiments

Experiments show that the model normalizes faces robustly across nuisance factors and benefits from data augmentation, landmark–texture decoupling, and FaceNet-based training loss. The normalized outputs also support 3-D fitting and face-focused color correction.

  • 7.1. Model Robustness: FaceNet features produce especially stable normalized faces across pose and illumination, while severe occlusions and paintings do not significantly reduce output quality.VGG-Face features produce comparable results, but FaceNet outputs are especially stable.
  • 7.2. Impact of Design Decisions: Without data augmentation, outputs contain more artifacts, while a fully connected texture decoder produces generic images because it lacks multiscale coordination.The comparison includes CNN and fully connected decoders trained with and without augmentation.
  • 7.2. Impact of Design Decisions: The CNN with data augmentation and decoupled landmarks and textures reproduces finer details than a same-capacity regular CNN.Warping training images to mean landmarks aligns facial features such as eyes and lips, improving fidelity.
  • 7.2. Impact of Design Decisions: All synthesized images pass FaceNet’s 1.242 identity threshold with the FaceNet loss, whereas about 2% fail without it.The loss reduces embedding distances between LFW inputs and synthesized outputs.
  • 7.3. 3-D Model Fitting: Normalized-face landmarks and texture enable straightforward fitting of a 3-D morphable model and produce a well-aligned mesh usable as a virtual-reality avatar.The reconstructed shape remains limited by the morphable model’s range.
  • 7.4. Exposure and White Balance: Face-focused color correction produces more consistent results across photos of the same person than Barron’s general white-balancing method.The method balances the face regardless of effects on other image regions.

8. Conclusion and Future Work

The method maps unconstrained face photographs to frontal, neutral-expression images that preserve identity despite lighting, pose, and expression variation. The authors identify downstream uses and future improvements involving differentiable spline interpolation and better image quality.

  • Conclusion: The network produces frontal, neutral-expression images that capture identity while remaining robust to lighting, pose, and expression variation.This robustness addresses factors that cause problems for prior frontalization methods.
  • Conclusion: The normalized outputs support automatic white balancing and custom 3-D avatar creation.These are presented as downstream opportunities of the normalization method.
  • Future Work: Differentiable spline interpolation is presented as a network module that may be useful beyond this method.The authors note that they are unaware of prior work using interpolation this way inside a network.
  • Future Work: Image quality is limited by background overfitting and pixel-level squared error, motivating broader training data and adversarial loss.The authors hope these changes will reduce noise artifacts and blurriness.

A. Additional Results

Additional experiments extend face normalization beyond standard LFW photographs to degraded images and illustrations. The supplementary 3-D fitting procedure projects synthesized colors onto a fitted mesh, balancing accuracy and cleanliness.

  • Additional Results: Degraded photographs and illustrations outside the training domain still produce credible face-normalization results.These examples are presented as tests beyond the method’s training domain.
  • 3-D Model Fitting: The 3-D fitting procedure matches 65 predicted landmarks to mesh vertices, optimizes morphable-model shape parameters, and projects the synthesized image onto the aligned mesh.The landmarks provide 130 constraints for 199 morphable-model parameters.

B.1. Corresponding Landmarks and Vertices

The landmark-to-vertex correspondence is fixed across faces because mesh topology does not change. It is stabilized by voting across projections from randomly jittered camera matrices.

  • B.1. Corresponding Landmarks and Vertices: The preprocessing establishes a fixed correspondence between 65 facial landmarks and mesh vertices because the mesh topology remains unchanged.Corresponding vertices are selected by comparing projected vertices with detected landmarks.
  • B.1. Corresponding Landmarks and Vertices: Voting across multiple randomly jittered camera matrices reduces unstable landmark-to-vertex matches caused by grazing-angle projections.The final result is a set of 65 vertex indices.

B.2. Shape Fitting

The method fits a 3-D morphable face model to synthesized 2-D landmarks, then uses confidence-weighted colors and interpolation to construct the normalized face. It also supports color adjustment and produces 2-D and multi-angle 3-D reconstructions.

  • B.2. Shape Fitting: The fitted shape uses 65 landmark-associated mesh vertices and a 65×199 basis with mean vertex positions to define the 3-D geometry.The basis matrices and mean positions define object-space vertices before projection.
  • B.2. Shape Fitting: Confidence weights suppress projected colors outside the landmark hull and at grazing angles, addressing artifacts caused by noisy synthesis and inaccurate projection.The mask is one inside the landmarks’ convex hull and smoothly decays outside; the vertex-normal z component captures grazing-angle effects.
  • B.2. Shape Fitting: The method projects vertex colors onto a 199-dimensional color basis with regularized least squares, then interpolates model colors with projected colors.The interpolation combines the projected color vector with the reconstructed morphable-model color vector using the confidence weights.
  • B.2. Shape Fitting: A per-channel piecewise-linear shift in YCrCb adjusts the synthesized image using the input and normalized images’ mean face colors.The adjustment is applied over pixels of the input image representation.
  • B.2. Shape Fitting: Additional figures compare the method with Hassner et al. on LFW and show 2-D and multi-angle 3-D outputs for degraded photographs and illustrations.The figures identify the input, generated 2-D image, and two 3-D reconstruction views, but do not state a quantitative comparison.
Loading 1701.04851v4…