PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Morphable Face Models -- Past, Present and Future

Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, Thomas Vetter

arXiv:1909.01815v2cs.CVcs.GRcs.LG

TL;DR

3D Morphable Face Models address how to represent and analyze the broad variation of human faces from limited observations. This paper surveys their capture, modeling, image formation, and image analysis methods over 20 years, then identifies unresolved challenges and future directions. It concludes that 3DMMs remain broadly useful, while current progress is constrained by data, modeling, rendering, and reconstruction limitations.

  • Problem

    Face reconstruction from a 2D image is ill-posed because shape, albedo, illumination, and camera parameters are ambiguous.

  • Method

    The paper provides a 20-year survey of 3DMM capture, modeling, rendering, analysis, deep learning, applications, challenges, and future directions.

  • Results

    3DMMs remain relevant as general face representations, while their state of the art is limited by gaps in datasets, rendering, and monocular reconstruction.

  • Takeaways & Limitations

    3DMM research is moving toward more general and realistic learned models, including neural rendering, 3D representation learning, and unsupervised shape-model learning.

  • Takeaways & Limitations

    High-quality public datasets remain scarce, and many models lack broad appearance coverage, limiting model building and realism.

Abstract

from arXiv · show

In this paper, we provide a detailed survey of 3D Morphable Face Models over the 20 years since they were first proposed. The challenges in building and applying these models, namely capture, modeling, image formation, and image analysis, are still active research topics, and we review the state-of-the-art in each of these areas. We also look ahead, identifying unsolved challenges, proposing directions for future research and highlighting the broad range of current and future applications.

1 INTRODUCTION

3D Morphable Face Models were introduced as a general, learned face representation for reconstructing and manipulating faces across varying images. Their enduring value lies in combining 3D structure with image-based methods, while fitting remains challenging and applications have broadened beyond faces.

  • Blanz and Vetter’s model used dense correspondence, separable shape and color, and statistical learning to generate morphologically realistic faces.
  • The original visual concept combined 3D reconstruction from 2D images with a parametric face space for controlled manipulation.
  • 3D Morphable Face Models learn prior knowledge from example faces to represent object-class variation and address otherwise ill-posed vision problems.
  • 3DMMs combined 3D models with 2D algorithms such as optical flow, multiresolution processing, and interpolation on parameterized face surfaces.
  • The framework has extended to bodies, body parts, animals, and cars, although this survey focuses on human faces.
  • Unlike task-specific methods, 3DMMs were designed as general face representations and remain distinctive for their breadth across tasks and applications.

2 FACE CAPTURE

Face capture for 3DMMs combines geometric, photometric, and hybrid methods to acquire shape and appearance. Although capture quality can be high, publicly available datasets often remain limited in quality, coverage, and appearance information.

  • 2 FACE CAPTURE: 3DMM construction begins with representative 3D shapes, usually coupled with corresponding appearance data, acquired under controlled conditions.
  • 2.1 Shape Acquisition: Geometric acquisition estimates 3D coordinates from multiple views or projected patterns using active or passive systems.
  • 2.1.1 Geometric methods.: Structured-light systems project coded patterns to recover depth, trading acquisition complexity against resolution and frame requirements.
  • 2.1.2 Photometric methods.: Photometric methods recover shape from surface orientation and provide fine detail without requiring matchable features, but modeling errors can cause low-frequency positional bias.
  • 2.1.3 Hybrid methods.: Hybrid methods merge geometric positional information with photometric surface normals to reduce low-frequency bias and increase high-frequency detail.
  • 2.6 Open challenges: The main 3DMM capture bottleneck is publicly available data: most datasets provide only moderate-quality shape and no appearance information, while high-quality systems are costly and specialized.
  • 2.6 Open challenges: Open acquisition questions concern dataset size, natural expressions, demographic sampling, method bias, and whether synthetic data should supplement limited capture.
  • 2.6 Open challenges: High-detail face datasets raise legal and ethical concerns because realistic synthesized images are difficult to control once data are publicly distributed.

3 MODELING

3DMM modeling represents variation in face shape, expression, and appearance using statistics over digitized faces. Dense correspondence and generative synthesis are central requirements, and the survey reviews global, local, and other model families.

  • 3 MODELING: 3DMM construction commonly models identity shape, expression geometry, and facial appearance as three principal variation types.
  • 3 MODELING: The original 1999 work already proposed models for all three variation types that remain commonly used.
  • 3 MODELING: Model statistics are computed over face databases, traditionally 3D scans and increasingly 2D images, requiring explicit or implicit correspondence.
  • 3 MODELING: 3DMMs are generative models whose key capability is synthesizing novel faces.

3.1 Shape models

3DMM shape models represent plausible human faces in a low-dimensional parameter space, commonly learned with PCA after establishing correspondence across registered scans. The survey describes global, local, Gaussian, and nonlinear approaches, while noting limitations in data availability and fine-scale detail.

  • Classical shape modeling: Template deformation establishes anatomical correspondences across scans before distances and statistical models can be computed.Raw scans may contain different numbers of unstructured vertices, making direct distance computation complex.
  • Shape representation: A 3DMM shape space represents plausible 3D human faces with low-dimensional parameter vectors rather than all 3n vertex coordinates.Each face is associated with a vector w ∈ R^d, where d ≪ n.
  • Global models: The seminal global model uses PCA to express each shape as the mean face plus a weighted combination of dominant covariance eigenvectors.The model assumes training faces can be linearly interpolated to generate new faces and uses a multivariate normal prior over parameters.
  • Global models: The original 3DMM was computed from 200 subjects, while a later model used 9,663 scans and found the original shape generator highly relevant.The model’s usefulness is attributed to its ability to generate plausible shapes and its simple underlying formulation.
  • Local models: Global models may miss fine-scale geometry, motivating localized models that improve reconstruction fidelity in regions such as the eyes and nose.Later work includes automatic segmentation, sparse PCA, dictionaries, and multiresolution wavelet-based representations.

3.2 Expression models

Expression models extend identity shape models by representing facial expression separately or jointly with identity. The survey covers additive, multiplicative, and nonlinear formulations, alongside the data-completeness constraints of multilinear models.

  • Model families: Expression models explicitly separate identity and expression coefficients and classify their combination as additive, multiplicative, or nonlinear.This differs from simple linear models learned over identities and expressions without explicit decoupling.
  • Additive models: Additive models transfer expression by adding expression offsets to a neutral identity shape.The offset is computed as the difference between an expressive and neutral shape of the same subject.
  • Additive models: In additive models, Es and Ee contain shape and expression basis vectors, while ws and we are their respective coefficients.Expression bases can be interpreted as data-driven blendshapes, but they are generally orthogonal and not semantically interpretable.
  • Additive models: Statistical expression models learned from expression data across subjects do not provide control over specific facial expressions.Earlier approaches learned separate PCA models for shape and expression, with expression data initially coming from one subject and later multiple subjects.
  • Multiplicative models: Multilinear models use higher-order tensor decomposition to model correlations between identity-driven and expression-driven shape changes.They are expressive and simple, and have also been applied to wavelet coefficients for localized variation.
  • Multiplicative models: HOSVD-based multilinear models require every identity to appear in every expression with semantic correspondence, severely restricting training data.Optimization, custom tensor decomposition, and autoencoder methods were proposed to address this limitation.
  • Nonlinear models: Nonlinear expression models include articulated head models, physical muscle simulation, and deep learning-based approaches.FLAME combines jaw articulation with linear expression blendshapes to provide nonlinear expression control.

3.3 Appearance models

Appearance models represent facial albedo and illumination using per-vertex values or texture-space representations, commonly with low-dimensional statistical models. The survey highlights trade-offs between representation formats and limits of Gaussian and illumination-separation assumptions.

  • Representation: Facial appearance modeling concerns albedo and illumination and distinguishes linear from nonlinear representations.Training appearance is usually stored as per-vertex values or textures in UV space.
  • Representation: Most 3DMMs do not fully separate illumination from albedo, so illumination is often baked into the modeled albedo.This motivates treating statistical capture of both factors as appearance modeling.
  • Linear models: Linear appearance models use a low-dimensional subspace to explain color variation, analogously to linear shape models.The representation uses a low-dimensional texture parameter vector.
  • Linear models: Back-projected image features can learn per-vertex appearance models with some invariance to illumination changes.Booth et al. use convex matrix factorization and features such as SIFT rather than raw color images, assuming known image geometry.
  • Texture-space models: Per-vertex models require compatible shape and appearance resolutions, whereas texture maps support lower-resolution geometry with higher-resolution 2D detail.Texture-space representations also permit image-processing operations on texture maps.
  • Statistical assumptions: Gaussian assumptions for appearance variation may be inaccurate and sub-optimal, motivating Copula Component Analysis and later joint models.Subsequent work jointly models facial shape, texture, and attributes, while deep learning approaches address appearance modeling within analysis-by-synthesis frameworks.

3.4 Joint shape and appearance models

3DMMs can model shape and appearance either separately or jointly. Joint models learn correlations between these factors as fitting constraints, while separate models retain greater flexibility.

  • Joint versus separate models: Separate shape and texture models are more flexible, whereas joint models can exploit shape–texture correlations during fitting with fewer parameters.The original 3DMM used independent models, while 2D Active Appearance Models used a combined representation.

3.5 Correspondence

3DMM construction depends on establishing dense point-to-point correspondence across face scans, typically by deforming a template after rough alignment. Methods also use sparse landmarks, temporal information, non-template mappings, or groupwise optimization to initialize or refine correspondence.

  • Template-based registration: Dense registration establishes point-to-point correspondence by deforming a template mesh to each scan, usually after rough alignment with sparse correspondences.The process leads from an initial alignment to dense correspondence.
  • Sparse correspondence: Sparse landmark prediction provides a common set of salient points that typically initializes subsequent dense correspondence methods.These methods focus on correspondence between 3D scans rather than image landmark localization.
  • Sparse correspondence: Existing landmark methods combine geometric descriptors such as shape index, spin images, curvature, SIFT, and volumetric features, sometimes modeling descriptor distributions.Other approaches additionally encode geometric or statistical relations between landmarks.
  • Registration variants: Static and dynamic registration methods differ in their scan targets, with motion-sequence methods exploiting optical flow or the registration of previous frames.Examples include AFM-, non-rigid-ICP-, and FLAME-regularized tracking.
  • Registration variants: Non-template and groupwise methods establish correspondence through conformal mappings, consistently resampled geodesic parts, compact multilinear models, or functional maps.Some frameworks also jointly update person-specific blendshape models and corrective deformations during tracking.

3.6 Synthesis of novel model instances

3DMMs synthesize realistic novel faces by manipulating coefficients in identity, expression, shape, or appearance spaces. Decoupled identity and expression coefficients also support dynamic sequences, including videos driven by semantic labels or audio.

  • Static synthesis: 3DMMs generate realistic faces unlike observed training examples by altering coefficients through interpolation or extrapolation in parameter spaces.Depending on the model, coefficient sampling can alter identity, expression, or appearance of a static 3D face.
  • Static synthesis: Entertainment uses heavily feature static-face synthesis and are discussed as a major application area.The passage specifically identifies synthesis works as heavily used for entertainment purposes.
  • Static synthesis: Static 3D face synthesis includes caricatures created by moving identity coefficients linearly away from the mean.This application is mainly used to study human face processing.
  • Dynamic synthesis: Decoupling identity and expression enables dynamic sequence synthesis by fixing identity coefficients while modifying expression coefficients.Examples synthesize coherent 4D videos from static meshes paired with semantic labels or audio information.

3.7 Publicly available models

The survey catalogs publicly available human-face shape and appearance models and visualizes geometry or appearance variations for selected models.

  • Public resources: Table 2 lists publicly available human-face shape and/or appearance models, while Figure 4 visualizes variations from selected models.The survey also provides a curated, updated list of 3DMM software and data.

3.8 Open challenges

Open challenges include incomplete coverage of facial details, limited interpretability, unresolved detail-versus-compactness trade-offs, application-specific model strengths, and alignment-dependent shape modeling. Figure 4 compares variation spaces across several existing models.

  • Open challenges: Most models cover the face but omit eyes, mouth interiors, or hair, although these details are crucial for many applications.Combining a 3DMM with specific models for omitted details is not straightforward.
  • Open challenges: PCA components often do not align with human facial attributes because PCA is an unsupervised representation method.The survey identifies improved interpretability as an open challenge.
  • Open challenges: Representations with multiple detail levels are less compact, and the parameter count needed for accurate geometry and appearance at varying detail remains unknown.The survey also notes that existing models have different advantages and drawbacks for specific applications.
  • Open challenges: Existing models differ in their application-specific advantages and drawbacks, leaving it unknown whether one integrated optimal model exists for all applications.Figure 4 visualizes shape, expression, appearance, and pose variations for CFHM, FaceWarehouse, BFM 2019, and FLAME.
  • Open challenges: Rigidly aligning faces before modeling residual nonlinear deformations makes the resulting model dependent on the selected alignment metric.Estimated skull position has been used for facial rigid alignment, while rigid-motion-invariant nonlinear components have also been proposed outside face modeling.

4 IMAGE FORMATION

This section explains how 3DMMs generate and analyze face images by combining parametric face models with geometric and photometric image-formation models. It surveys camera projections, reflectance and color transformations, available model resources, and remaining rendering challenges.

  • 4 IMAGE FORMATION: 3DMM synthesis samples face geometry and appearance parameters, then renders the resulting instance into a 2D image through simulated image formation.
  • 4 IMAGE FORMATION: Table 2 surveys publicly available human-face models that provide shape data, appearance data, or both.
  • 4 IMAGE FORMATION: The section covers geometric and photometric image formation, fitting renderers, and opportunities to apply state-of-the-art rendering techniques to 3DMM synthesis.
  • 4.1 Geometric image formation: Camera models specify how 3D points project into image-plane locations, with extrinsic parameters describing rotation and translation and intrinsic parameters describing camera properties.
  • 4.1 Geometric image formation: Scaled orthographic and affine projections retain linearity for estimation, while scaled orthographic avoids size, distance, and perspective ambiguities and affine permits non-uniform scaling and skew.
  • 4.1 Geometric image formation: Perspective projection models distance-dependent shape changes but introduces ambiguities among shape, scale, focal length, and camera distance when calibration is unavailable.
  • 4.2 Photometric image formation: Photometric formation models illumination, reflectance, and camera color processing, including BRDF-based reflection, Lambertian diffuse reflectance, and nonlinear color transformations.
  • 4.4 Open challenges: Complex physically valid BRDF and subsurface-scattering models remain difficult to integrate into differentiable 3DMM fitting pipelines, so most systems use Lambertian or moderately complex non-physical models.

5 ANALYSIS-BY-SYNTHESIS

3DMM analysis-by-synthesis reconstructs faces by estimating model coefficients whose rendered images explain observations, but monocular reconstruction remains ill-posed and difficult. The section surveys input modalities, energy functions, optimization strategies, and unresolved accuracy and ambiguity challenges.

  • Core formulation: Analysis-by-synthesis estimates 3DMM coefficients by minimizing differences between observed images and synthesized faces.The optimization may be ill-posed, with ambiguities and multiple minima.
  • Input modalities: Multi-view methods produce detailed, high-quality reconstructions but require expensive capture setups, motivating lower-cost monocular approaches.Single RGB images have supported increasing commercial applications for the mass market.
  • Input modalities: Single-image reconstruction is the most challenging scenario, with methods differing in initialization, image features, occlusion handling, and formulated energy functions.Approaches use photometric consistency, edges, SIFT, or jointly solved segmentation to improve robustness and fidelity.
  • Energy functions: Background variation can create shrinking solutions in pose-and-shape optimization, so fixed visibility, restrictive regularization, landmarks, or explicit background models are used.Even simple constant, Gaussian, or histogram-based background models can address the implicit-background problem.
  • Optimization: Gradient-based methods are computationally efficient but sensitive to initialization and local minima, whereas Bayesian methods approximate a posterior over multiple possible solutions.Bayesian approaches avoid gradient computation when updating estimates and are less susceptible to getting stuck in local minima.
  • Open challenges: Monocular shape-and-albedo recovery has unresolved perspective and albedo-illumination ambiguities, leaving a substantial quality gap versus multi-view and 3D data.Discriminative methods can sidestep explicit illumination and camera estimation by matching identity rather than reproducing the exact appearance.
  • Open challenges: Current methods remain inaccurate across pose and contours, while qualitative evaluation and limited ground-truth benchmarks make comparisons difficult.A benchmark with natural images and ground-truth shape was introduced to improve comparison of competing methods.
  • Open challenges: Future face analysis should extend beyond separate faces by modeling interactions and exploiting shared scene constraints such as illumination.Such constraints are proposed for addressing albedo-illumination and perspective face-shape ambiguities.

6 DEEP LEARNING

Deep learning extends 3DMM modeling and fitting with nonlinear representations, learned regressors, and self-supervised combinations with differentiable rendering. The section surveys these approaches alongside persistent challenges in training data, overfitting, comparison, and reconstruction quality.

  • Overview: Deep learning can replace or combine with classical inverse-graphics processes, offering nonlinear models and faster, more robust parameter estimation.The survey organizes this work into deep modeling, deep model fitting, and jointly learned models and fitting.
  • Deep modeling: Classical linear 3DMMs correspond to encoder-decoder structures, while a linear autoencoder can learn a latent space spanning the same subspace as PCA.The autoencoder does not guarantee orthogonality unless suitable loss functions are used.
  • Deep modeling: Deep 3DMM research includes encoder-decoder, variational autoencoder, GAN, hybrid, and per-vertex appearance models.These approaches target nonlinear geometry, multiscale detail, texture generation, disentanglement, and improved albedo generalization.
  • Deep model fitting: Deep reconstruction methods move optimization from test time into training, using regressors that predict 3DMM parameters from monocular images.Supervised approaches require paired images and ground-truth 3DMM parameters.
  • Training data: Accurate dense annotations of geometry, appearance, and illumination are almost intractable, making ground-truth acquisition a central supervised-learning challenge.Manual annotation is popular for sparse reconstruction, but dense supervision is substantially more demanding.
  • Open challenges: Current priorities include analyzing method limitations, performing comprehensive comparisons, and assessing overfitting, especially with predominantly synthetic training data.The survey identifies these as pressing unresolved challenges in deep 3D face analysis.
  • Open challenges: Deep methods may shift overfitting from individual samples to the training-set face distribution, despite batch-based updates that can alleviate local-minimum issues.The underlying energy functions remain essentially similar to those used by classical methods.
  • Open challenges: Learning-based methods are typically fast and initialization-robust but produce lower-quality reconstructions than analysis-by-synthesis methods.Self-supervised approaches retain shared challenges including perspective ambiguity and occlusion handling.

7 APPLICATIONS

3DMMs support applications spanning recognition, entertainment, medicine, and related domains by providing a parametric representation that separates facial identity from imaging factors. Their benefits are strongest in controlled settings, while in-the-wild recognition remains difficult because fitting shape and color from a 2D image is ill-posed.

  • Application scope: 3DMM applications span face recognition, entertainment, medical analysis and synthesis, forensics, and cognitive and psychological research.Publicly shared models and code have helped push these applications.
  • Face recognition: 3DMM recognition uses shape and color coefficients to separate identity from camera and illumination variation.This separation supports cosine-distance comparisons between coefficient estimates from image pairs.
  • Face recognition: 3DMM-based recognition has been demonstrated across large pose and illumination variations and with robustness to facial expressions.The cited work also applies 3DMM parameters to recognition from facial texture features and task-specific regression.
  • Face recognition: In-the-wild face recognition remains unconvinving because estimating shape and color from a 2D image is ill-posed and requires high precision.The survey contrasts this limitation with data-driven methods, while noting their dependence on large-scale training data and limited out-of-distribution generalization.
  • Entertainment: Entertainment applications use 3DMM priors to constrain reconstruction from limited sensors, including single color cameras.Applications include avatars, virtual mirrors, re-texturing, face replacement, and facial reenactment.
  • Medical applications: Clinical 3DMM applications analyze facial shape for disease recognition and synthesize missing face parts for personalized implant design.The survey also describes reconstruction from 2D images for screening and controlled stimuli for studying brain responses.

7.4 Forensics

In forensics, 3DMMs support identikit generation, virtual aging, skull-based face reconstruction, and detection of manipulated media. These applications use the model to control facial attributes or represent uncertainty, but important limitations remain in modeling texture and hair.

  • Forensics: Forensic applications include identikit pictures, virtual aging, dry-skull face reconstruction, and detection of manipulated videos.
  • Identikit generation: A 3DMM-based identikit tool uses correlations within faces to generate pictures from vague feature descriptions.
  • Virtual aging: 3DMMs reduce subjectivity in age progression by modeling age trajectories in shape, and sometimes texture.Most methods focus on children and neglect textural details or wrinkles.
  • Skull-based reconstruction: Dry-skull face reconstruction is ill-posed because one skull can correspond to multiple possible faces.3DMM-based approaches control facial attributes, estimate posterior solutions, or model soft-tissue thickness.
  • Manipulated media: 3DMMs can generate or manipulate facial media and can also help detect such manipulations with high accuracy.

8 PERSPECTIVE

The survey identifies unresolved challenges in 3DMM detail, comparability, ethics, scalability, and generality. It proposes learning-based, hierarchical, and interpretable models that combine diverse data while balancing realism, task coverage, privacy, and misuse risks.

  • Perspective: The survey’s perspective section examines unsolved challenges, scalability beyond faces, data and model sharing, potential misuse, and 10- to 20-year directions.
  • Global challenges: A central challenge is balancing low-dimensional parameterization with detailed modeling of eyes, teeth, hair, skin, soft tissue, and muscles.Automatic extraction and modeling of these details remain at an early stage, while added complexity makes analysis-by-synthesis harder.
  • Global challenges: 3DMM components remain difficult to compare because models and adaptation algorithms are evaluated on task-specific results without comprehensive benchmarks.Existing datasets address shape reconstruction but not a unified evaluation of reconstruction, illumination, and albedo.
  • Scalability: Scaling beyond faces requires new representations because clothing, full hair, and other object classes exhibit greater appearance, structure, and shape variation.The survey points to unsupervised or weakly supervised learning and implicit representations as emerging directions.
  • Ethics and applications: Future reconstruction methods may create highly believable avatars from casually captured imagery, while also extracting sensitive information about emotion, health, and physical condition.The survey calls for privacy-preserving strategies and consent procedures alongside reconstruction advances.
  • Outlook: The field currently diverges into task-specific models with different requirements for quality, realism, generalization, and performance, leaving a dramatic gap between single-face renderings and statistical generative models.
  • Outlook: Machine learning may produce more general and realistic models through neural rendering, 3D representation learning, unsupervised shape learning, and model learning.The survey notes that current learned models still inherit assumptions similar to traditional 3DMMs, while 2D GAN embeddings are difficult for humans to interpret.
  • Outlook: A proposed living 3DMM would learn from 3D and 2D data, represent multiple detail levels hierarchically, generalize across tasks, support real-time analysis and photorealistic rendering, and retain interpretability.The proposal aims to automate currently manual modeling work while avoiding common oversimplifications.
Loading 1909.01815v2…