PaperlayerView exact revision on arXiv

Source-linked AI summary

Morphable Face Models - An Open Framework

Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Lüthi, Sandro Schönborn, Thomas Vetter

arXiv:1709.08398v2cs.CV

TL;DR

Face registration for morphable models must handle artifacts, missing data, expressions, symmetry, and variation at different detail levels while keeping domain assumptions manageable. The paper addresses this with Gaussian Process Morphable Models, an open registration and model-building pipeline, and single-image model adaptation. It reports accurate registration of closed and open mouths, competitive face recognition, and a BFM-2017 model that outperforms models built on BU3D-FE and the 2009 Basel model.

  • Problem

    Dense correspondence is crucial for morphable face models, but face scans contain artifacts, missing data, outliers, expressions, multi-scale variation, and symmetry, while prior assumptions are often entangled with registration algorithms.

  • Method

    The paper models face-specific deformation knowledge with Gaussian-process priors and releases a pipeline for registration, model building, and probabilistic single-image model adaptation.

  • Results

    The pipeline accurately registers closed and open mouths, achieves competitive face recognition, and produces BFM-2017 that outperforms BU3D-FE and the 2009 Basel model.

  • Takeaways & Limitations

    The open-source release enables reproducible evaluation of face registration, model building, and 3D/2D model fitting on public data.

Abstract

from arXiv · show

In this paper, we present a novel open-source pipeline for face registration based on Gaussian processes as well as an application to face image analysis. Non-rigid registration of faces is significant for many applications in computer vision, such as the construction of 3D Morphable face models (3DMMs). Gaussian Process Morphable Models (GPMMs) unify a variety of non-rigid deformation models with B-splines and PCA models as examples. GPMM separate problem specific requirements from the registration algorithm by incorporating domain-specific adaptions as a prior model. The novelties of this paper are the following: (i) We present a strategy and modeling technique for face registration that considers symmetry, multi-scale and spatially-varying details. The registration is applied to neutral faces and facial expressions. (ii) We release an open-source software framework for registration and model-building, demonstrated on the publicly available BU3D-FE database. The released pipeline also contains an implementation of an Analysis-by-Synthesis model adaption of 2D face images, tested on the Multi-PIE and LFW database. This enables the community to reproduce, evaluate and compare the individual steps of registration to model-building and 3D/2D model fitting. (iii) Along with the framework release, we publish a new version of the Basel Face Model (BFM-2017) with an improved age distribution and an additional facial expression model.

I. INTRODUCTION

The paper introduces Gaussian Process Morphable Models for face registration, separating face-specific priors from the registration algorithm. It combines multi-scale detail, spatial variation, symmetry, and facial-expression modeling in an open, reproducible pipeline.

  • Motivation: Face registration is needed to establish dense correspondences for morphable models, but scans contain artifacts, missing data, outliers, expressions, multi-scale variation, and symmetry.Existing face-specific methods address these issues, but mix prior assumptions with algorithmic components.
  • Approach: Gaussian Process Morphable Models separate domain-specific adaptations from the registration algorithm by encoding them as a Gaussian-process prior.This separation makes the approach conceptually simple and supports intuitive kernel-based modeling.
  • Face-specific prior: Multi-scale B-spline kernels and spatial damping model facial variability at multiple detail levels.Predefined regions can be damped differently across deformation scales.
  • Face-specific prior: Mirror-symmetric kernels model the near mirror symmetry of facial shapes.
  • Face-specific prior: Expression prototypes provide a statistical kernel for modeling mouth opening and closing, while the model-building method handles regions with missing data.
  • Reproducibility and resources: The released open-source pipeline supports registration, model building, and single-image model adaptation on publicly available data.The authors also release BFM-2017 with facial expressions, improved age distribution, and controlled-environment training data.

II. RELATED WORK

The related-work section situates the method among ICP/NICP and alternative deformation models, then outlines Gaussian-process registration as a probabilistic, parameterized optimization problem.

  • Existing registration methods: ICP and NICP are widely used for establishing correspondence between 3D face shapes, but adapting NICP to domains or datasets requires changing search heuristics and stiffness weights.These adaptations make NICP complicated to modify in practice.
  • Existing registration methods: Alternative approaches use B-splines, thin-plate splines, local statistical constraints, or isometric deformations to address detail, symmetry, or facial expressions.
  • Gaussian-process formulation: The proposed registration models target faces as deformations of a reference surface using a Gaussian-process prior over deformation fields.The prior’s mean and covariance specify expected deformation and allowed deformation characteristics.
  • Optimization formulation: Registration becomes a MAP problem with a likelihood based on distances from deformed reference points to the target surface.The likelihood uses a loss function and closest-point operator, with errors treated as independent across vertices.
  • Optimization formulation: A truncated Karhunen–Loève expansion converts the infinite-dimensional optimization into a parametric model governed by basis functions and weights.The target-shape probability is determined by the finite parameter vector α.
  • Optimization formulation: The final registration problem is solved as a parametric optimization using L-BFGS.

B. Combining Kernels

The framework constructs valid deformation kernels from simpler kernels using scaling, pointwise products, and factorized functions.

  • Kernel construction: Symmetric positive semi-definite kernels can be combined through scaling, pointwise multiplication, or products of functions evaluated at each input.These rules generate new positive semi-definite kernels from base kernels.

C. A Shape Prior tailored for Face Registration

The face-registration prior uses the Basel Face Model’s mean shape as reference and assumes zero mean deformation.

  • Reference and mean: The reference surface is the Basel Face Model mean shape, so the prior uses zero mean deformation because the reference is treated as an average face.

1) Multi-scale Deformations:

The registration model represents facial variability across multiple deformation scales, with spatially varying detail and mirror-symmetric deformation priors. Coarse levels capture global shape, while finer levels model localized facial details.

  • Multi-scale Deformations: Multi-scale B-spline kernels represent facial variability from coarse to fine deformation levels.Coarse levels permit larger deformations, while finer levels add localized detail.
  • Spatially Varying Scales: Spatially varying indicator functions activate deformation scales differently across the face.Eyes and mouth receive small-scale deformations, whereas cheeks remain restricted to smooth, large-scale changes.
  • Symmetry: Symmetric kernels model the near-mirror symmetry of facial shapes while allowing deformation correlations across the symmetry plane.The construction uses sign changes in the component correlations to produce mirror-symmetric deformation fields.
  • Multi-scale Deformations: Increasing scale levels progressively reveals coarse head structure, facial placement, folds, and detailed eye and nose shape.Level 1 captures the lowest scale, while Levels 3–4 expose finer deformation changes.
  • Spatially Varying Scales: Spatially varying kernels produce coarse deformations around the ears and progressively more detail toward the eyes than globally uniform kernels.The region map controls which scale levels are active in each facial area.
  • Symmetry: Symmetrized-kernel samples are contrasted with normal-kernel samples, with strongly asymmetric examples described as unrealistic.The comparison visualizes the effect of imposing symmetry in the deformation model.

4) Core Expression Model:

The core expression model addresses mouth opening and closing, which smooth deformation kernels cannot represent reliably. It combines expression prototypes with the existing prior so one model can register both closed- and open-mouth faces.

  • Core Expression Model: Smooth kernels struggle to model mouth opening because nearby upper- and lower-lip points correlate strongly.A separate open-mouth reference works for open expressions but performs poorly on neutral and some other faces.
  • Core Expression Model: The proposed expression kernel combines a statistical shape model built from facial expression prototypes with the existing deformation prior.The prototypes include anger, disgust, fear, happy, sad, and surprise expressions.
  • Core Expression Model: The expression kernel accurately registers all tested closed- and open-mouth samples.Figure 4 reports successful registrations for neutral and expression examples using the core model.

D. The registration algorithm

The registration algorithm incorporates landmarks into a Gaussian-process posterior, then iteratively optimizes the fit while reducing regularization and rejecting unreliable correspondences.

  • Registration Procedure: The procedure first computes a posterior Gaussian-process model from landmarks and initializes the deformation solution.Landmarks are treated as noisy observations, so incompatible deformations receive low posterior probability.
  • Registration Procedure: Regularization weights decrease from 1e-1 to 1e-5 across successive optimization steps.At each step, the current fit is used to identify outliers before re-optimizing.
  • Registration Procedure: Available line annotations are matched using the current fit and incorporated through an additional posterior model.This line-landmark step is conditional on annotations being available.
  • Registration Procedure: Outliers and points nearest target-surface boundaries are discarded during optimization.Boundary points indicate holes in the target surface and are excluded from the registration objective.
  • Registration Objective: The distance metric uses the Huber loss function.The Huber loss defines the robust distance term used in the registration objective.

E. Building the Morphable Model

The morphable-model color component transfers corresponding target colors onto registered meshes while handling incomplete scans and missing color observations. Its covariance model combines empirical information with a smoothness prior.

  • Color Model: Target colors are sampled at corresponding points on each registered mesh to construct the color model.A binary indicator marks whether each point has a reliable assigned color.
  • Color Model: The color mean is computed using only points with available reliable color values.This avoids treating missing target colors as valid observations.
  • Color Model: The color covariance uses an additional kernel to encode a smoothness prior over color variation.Depending on data availability, covariance terms use empirical covariance or the prior kernel.

2) Expression Model:

The paper extends the face model to represent facial expressions and releases BFM-2017 with improved demographic coverage, controlled data, and expression examples.

  • Expression Model: The expression model adds a separate deformation model to independent face-shape and face-color models, representing expressions as differences from neutral shape.
  • Expression Model: The registration dataset uses 100 individuals with neutral and six-expression scans at four strength levels, retaining raw scans at Level 4.
  • Basel Scans: The Basel scans provide controlled acquisition, more individuals than BU-3DFE, high-quality texture, known age, and six controlled expressions with contour lines.
  • Basel Face Model: BFM-2017 includes 100 male and 100 female shape examples, 100 male and 100 female color examples, and 160 expression examples equally distributed across expression types.
  • Basel Face Model: The improved age distribution is closer to the EU distribution and includes more people over 40, capturing attributes such as sagging and wrinkles.

A. Landmark Evaluation

The paper evaluates registration against BU-3DFE landmarks and applies the model to Multi-PIE and LFW images. Registration matches database annotations, while recognition results are competitive with state-of-the-art inverse-rendering methods.

  • Landmark Evaluation: BU-3DFE landmark evaluation finds the proposed registration comparable to annotated correspondences and on par with prior evaluation.
  • Landmark Evaluation: Expression-dependent eye and eyebrow changes produce high standard deviation when all expressions are evaluated together.
  • Inverse Rendering: The framework adapts a probabilistic 3D morphable model to single 2D images using Markov chain Monte Carlo sampling and expression proposals.
  • Inverse Rendering: Multi-PIE experiments use neutral and smiling images of 249 individuals across four poses under frontal illumination, with fitting and unconstrained recognition evaluations.
  • Inverse Rendering: Recognition results on Multi-PIE are competitive with state-of-the-art inverse-rendering techniques, and qualitative fitting results are also shown on LFW.
  • Conclusion: The complete pipeline and BFM-2017 are released to support reproducible registration, model-building, model-fitting, and comparison across datasets and models.
Loading 1709.08398v2…