PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards a complete 3D morphable model of the human head

Stylianos Ploumpis, Evangelos Ververas, Eimear O' Sullivan, Stylianos Moschoglou, Haoyang Wang, Nick Pears, William A. P. Smith, Baris Gecer, Stefanos Zafeiriou

arXiv:1911.08008v2cs.CV

TL;DR

Existing 3DMMs can differ in coverage and detail, and combining them is difficult when raw scans are unavailable. The paper fuses face, head, and ear models and incorporates eye and oral structures into a more complete head model. In reconstruction experiments, the fused UHM performs best in both reported settings, with its ear variation credited for the result.

  • Problem

    Combining existing 3DMMs with different coverage and detail is difficult when the raw scans needed to train or enrich them are unavailable.

    Source: §1 · p. 1

  • Method

    The paper combines LSFM face and LYHM head models using regression and Gaussian-process covariance fusion, then adds a detailed ear model and eye and oral structures.

    Source: §1 · p. 1, §3.1 · p. 4

  • Results

    In both reported reconstruction settings, UHM achieves the best reconstruction, which the authors attribute to its fused ear variation.

    Source: §8.2 · p. 12

  • Takeaways & Limitations

    The model provides a more complete statistical representation of head shape, including face, cranium, ears, and eyes, with basic oral structures.

    Source: §9 · p. 15

    The model does not statistically represent internal mouth-shape variation, and its shape modelling relies on GPA and PCA rather than potentially preferable nonlinear models.

    Source: §9 · p. 15

Abstract

from arXiv · show

Three-dimensional Morphable Models (3DMMs) are powerful statistical tools for representing the 3D shapes and textures of an object class. Here we present the most complete 3DMM of the human head to date that includes face, cranium, ears, eyes, teeth and tongue. To achieve this, we propose two methods for combining existing 3DMMs of different overlapping head parts: i. use a regressor to complete missing parts of one model using the other, ii. use the Gaussian Process framework to blend covariance matrices from multiple models. Thus we build a new combined face-and-head shape model that blends the variability and facial detail of an existing face model (the LSFM) with the full head modelling capability of an existing head model (the LYHM). Then we construct and fuse a highly-detailed ear model to extend the variation of the ear shape. Eye and eye region models are incorporated into the head model, along with basic models of the teeth, tongue and inner mouth cavity. The new model achieves state-of-the-art performance. We use our model to reconstruct full head representations from single, unconstrained images allowing us to parameterize craniofacial shape and texture, along with the ear shape, eye gaze and eye color.

1 INTRODUCTION

The paper addresses the difficulty of combining existing 3D morphable models when raw scans are unavailable and models differ in coverage and detail. It proposes fusing face, head, and ear models into a more complete human-head model, with an application to reconstruction from unconstrained single images.

1 INTRODUCTION

  • Raw scans needed to train 3DMMs are costly to collect and annotate, and may be unavailable, making it difficult to combine models with different coverage and detail.The authors identify data protection restrictions as one reason raw face and head scans may not be accessible.Source: §1 · p. 1
  • The authors present a large-scale head model covering face, cranium, ears, and eyes, with basic oral-cavity, tongue, and teeth models, and report an experiment reconstructing full heads from unconstrained single images.They describe the model as more complete in shape variation than previously published models.Source: §1 · p. 1
  • Unlike approaches that fit separate part models and linearly blend their outputs, the proposed framework fuses the models into one 3DMM to avoid possible discontinuities.The paper describes the separate-model approach as one used in prior work,.Source: §1 · p. 1
  • A complete head model could support clinical diagnosis and surgical planning, realistic avatar creation, ergonomic design, and reconstruction from 2D images.The paper names accurate craniofacial shape variation across the population as relevant to ergonomic design.Source: §1 · p. 1
  • The paper proposes regression and Gaussian-process covariance-combination methods to fuse shape-based 3DMMs into a single model.The human face, full head, and ear serve as exemplars for the fusion methodology.Source: §1 · p. 1

2 RELATED WORK

Related work spans face models, full-head models, and specialized eye and ear models. The paper builds on their strengths while addressing gaps in complete-head anatomy, facial detail, and ear variation.

2 RELATED WORK

  • The paper extends an earlier version with detailed ear and eye models, a full-head texture model, and further experimental evaluation.A recent survey identified more complete statistical modelling of the human head as an open challenge.Source: §2 · p. 2

2.1 Face models

  • Face-model work progressed from early 3DMMs and the Basel Face Model to LSFM, built from nearly 10,000 scans and supporting bespoke age, gender, and ethnicity models.The paper focuses on combining 3DMMs rather than developing alternatives to nonlinear face reconstruction methods,.Source: §2.1 · p. 2

2.2 Head models

  • Earlier full-head models have limitations in head resolution or statistical coverage, while LYHM models cranium-face correlation but has limited facial detail and ethnic diversity.The CEASAR-based model uses sparse topology, and a related nonlinear model still lacks full-cranium statistical variation.Source: §2.2 · pp. 2–3

2.3 Eye and ear models

  • Eye models include the eyeball and surrounding regions, while existing LSFM and LYHM templates have insufficient ear detail and variation; this work fuses a high-resolution ear model to enrich that variation.Prior ear models were built from small samples, and one was augmented with landmarked 2D ear images.Source: §2.3 · p. 3

3 FACE AND HEAD SHAPE COMBINATION

The section combines the LSFM face model with the LYHM full-head model through latent-parameter regression or Gaussian-process covariance blending. It then refines the resulting head model using registrations of MeIn3D scans.

3.1 Regression modelling

  • Regression maps LSFM face parameters to LYHM full-head parameters, allowing the cranium shape to be predicted for a given face.Random LYHM shapes are registered to the LSFM mean face to form paired face and head parameters for learning the mapping.Source: §3.1 · pp. 3–4
  • Nearly 10,000 MeIn3D faces are used to construct merged head shapes, replacing the less-detailed facial region with registered LSFM faces.Nonrigid registration deforms the outer facial mesh to match the cranium while keeping the inner mesh stiffer; a final registration addresses LYHM neck-circumference inconsistencies and uses 50 additional landmarks around the ear.Source: §3.1 · p. 4
  • PCA on the 10,000 processed meshes produces a generative full-head model with more facial detail and bespoke head shapes.For covariance blending, the face and head models must share the same scale space, unlike in the regression method; distance-based weights blend them smoothly across their boundary.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 Gaussian process modeling

  • Gaussian-process covariance blending offers a less complicated, more robust combination method that avoids irregular head shapes from poor regression values.The combined covariance is designed to retain LSFM facial detail and LYHM head-shape variation.Source: §3.2 · pp. 4–5

3.3 Model Refinement

  • The refinement procedure reduces the covariance model to fewer statistical components, then uses a new sample covariance from full-head reconstructions to refine scans and build the final PCA model.Scan correspondences on edges or beyond a distance threshold are excluded.Source: §3.3 · p. 6

4 EAR MODEL COMBINATION

The ear model addresses limited ear-shape variation in LSFM and LYHM by adding a high-resolution model built from 254 scans and fusing its variation into the combined head model.

4 EAR MODEL COMBINATION

  • The new ear model augments the combined face-and-head model to address the moderate ear variation in LSFM and LYHM.It was constructed from 254 distinct scans.Source: §4 · p. 6

4.1 High resolution ear model

  • The 254-ear dataset combines SYMARE ears, high-resolution adult scans, and children’s CT scans to broaden age and sex coverage.The adult scans came from 64 individuals, 32 male and 32 female, aged 20–70.Source: §4.1 · p. 6
  • After landmark annotation and non-rigid registration to a 2,800-vertex template, PCA of aligned meshes yields right- and mirrored left-ear models.All left ears were mirrored to match the right-ear orientation before registration and alignment.Source: §4.1 · pp. 6–7

4.2 Fusing the ear and the head model

  • The ear models’ covariance variation is fused into the head model using an unwrapped ear-space distance weighting that avoids discontinuities at the ear base.The flattened circular representation supports asymmetric ear variation without interfering with learned variation between ears.Source: §4.2 · p. 7

5 EYE MODEL COMBINATION

The model adds statistical representations of eyelid and eye-region shape, eye geometry and texture, and fits them to images to recover eye appearance and gaze. It also estimates eye color by classifying low-resolution eye images.

5.1 Eye models

  • The eye-region PCA model, built from 72 artist-sculpted head meshes, represents eyelid and peripheral shapes including round, almond, monolid, hooded, upturned, and downturned eyes.Its parameters describe variation in the eye-region shape subspace.Source: §5.1 · pp. 7–8
  • The eye model separates a static outer lens from an eyeball with iris and pupil, while a blendshape controls pupil size and a PCA texture model represents iris color.The texture model was built from 100 frontal iris images.Source: §5.1 · p. 8

5.2 Optimization-based eye model fitting

  • The fitting pipeline estimates eyelid shape from eyelid landmarks, gaze and pupil size from iris and pupil landmarks, and eye color through a texture loss with regularization.It first solves a whole-head perspective camera from 68 face landmarks, then optimizes the eye models with the camera fixed.Source: §5.2 · p. 8

5.3 Extending the traditional approach

  • Because multi-step Gauss–Newton fitting is relatively slow, a regression network estimates eyelid, pupil, and gaze parameters in one forward pass.The network is trained on images paired with parameters recovered by the 3DMM fitting pipeline.Source: §5.3 · pp. 8–9

5.4 Color estimation

  • The color estimator classifies cropped 64 × 64 eye images into seven manually labeled classes using an encoder trained on AgeDB.The training and test split uses 90% and 10% of AgeDB, respectively.Source: §5.4 · p. 9
  • Eye-color classification reaches about 92% accuracy on the AgeDB test set, with most errors between color groups that are also difficult for humans to distinguish.The seven classes are amber, blue, brown, gray, green, hazel, and dark brown.Source: §5.4 · p. 9

6 ORAL CAVITY AND TEETH

The model adds an inner-mouth representation for structures visible in raw images, but the tongue and teeth have no independently learned statistical variation.

6 ORAL CAVITY AND TEETH

  • The inner-mouth model includes cheek lining, the front two-thirds of the tongue, upper and lower gums, and the mouth floor and roof.These additions address oral structures that can be visible in raw images.Source: §6 · p. 9
  • Teeth are separate meshes fixed on top of the gums, while the tongue and teeth capture no independent statistical variation from training data.Source: §6 · p. 9
  • Overall scale is propagated smoothly from the outer lips into the inner cavity, decaying with distance.Source: §6 · p. 9

7 TEXTURE MODELING AND COMPLETION

The paper uses a GAN-based texture model to preserve high-frequency facial detail and completes missing head texture from facial texture using artist-created training pairs.

7 TEXTURE MODELING AND COMPLETION

  • A GAN architecture models UV-aligned textures to preserve high-frequency skin detail and avoid the blurriness of low-frequency PCA texture modeling.Source: §7 · p. 9
  • The source set contains approximately 10K facial textures and 1,200 full-head textures, but blue latex hair caps make the cranium textures unwanted.Source: §7 · p. 10
  • To address scarce full-head texture ground truth, an artist completed facial textures into face–head pairs used to train pix2pixHD for missing cranium texture completion.Figure 10 shows an input facial texture, the model’s completed texture, and an artist-produced ground truth completion.Source: §7 · p. 10, Figure 10 · p. 10

8 EXPERIMENTS

The experiments assess the intrinsic characteristics of CFHM and UHM and test full-head reconstruction from single images, including head shape, ears, and eyes.

8.2 Head reconstruction from single images

  • The single-image pipeline maps fitted face shape and ear landmarks to a full head, then completes texture and estimates eye parameters from a cropped eye image.Processing takes approximately 32 seconds on an NVIDIA RTX 2080 Ti and Intel Core i7 3.8 GHz system.Source: §8.2 · pp. 11–12

8.1 Intrinsic evaluation

  • All proposed models generalize better than LYHM; CFHM-ref tends to outperform demographic-specific models, while UHM shows marginally better, steadier error descent.The authors suggest smaller training cohorts may explain the bespoke models’ weaker generalization.Source: Figure 11 · p. 11, §8.1 · p. 11, Figure 12 · p. 12
  • CFHM-reg and CFHM-GP are more compact than LYHM, and UHM extends ear variation while describing the original ear variability with fewer components.CFHM-ref is less compact than CFHM-reg and CFHM-GP.Source: §8.1 · p. 10, Figure 11 · p. 11, Figure 12 · p. 12
  • CFHM-ref has the best specificity error; combined models maintain steady error measures of approximately 3.8 after 20 components.Bespoke models show specificity errors of 1 to 4 mm, while UHM has specificity measures identical to CFHM-ref.Source: Figure 11 · p. 11, §8.1 · p. 11

8.2.1 Special Cases

  • The pipeline produces plausible head shapes and textures from images with occlusions, paintings, and animated characters; when ears are not visible, it uses mean UHM ear landmarks.Examples include hair, hats, sunglasses, and hands covering parts of the face or head.Source: Figure 13 · p. 13, §8.2.1 · p. 14

8.2 Head reconstruction from single images

  • UHM achieves the best head reconstruction in both fitted-face and ground-truth-face settings, and significantly outperforms LYHM and CFHM-ref on ear estimation.The authors attribute its head-shape result to the increased ear-shape variation after fusion.Source: §8.2 · p. 12, Figure 14 · p. 14

8.3 Eye model evaluation

  • Eye-region shape, gaze direction, and pupil-size recovery achieve 94%, 98%, and 86% mean parameter accuracy, respectively.Qualitative reconstructions also simulate eyelid shape and gaze direction, with pupil size adapting reasonably where visible.Source: §8.3 · pp. 14–15
  • UHM outperforms eye3DMM by 0.59o in mean angular error on Eyediap, excluding extreme gaze directions for comparison.The UHM eye pipeline uses an end-to-end network to regress eye-model parameters.Source: §8.3 · p. 15

9 CONCLUSION

The paper combines face, head, ear, and eye models into a complete human-head 3DMM and demonstrates its use for single-image head reconstruction. The model still omits several aspects of head anatomy and retains a classical shape-modelling pipeline.

9 CONCLUSION

  • The combined model integrates LSFM face detail, LYHM cranial variation, a high-detail ear model, and eye-region deformations into a complete head representation.It also includes a basic oral cavity, tongue, and teeth model.Source: §9 · p. 15
  • The model is illustrated reconstructing full heads from a single image, including ears and eyelid shape.Source: §9 · p. 15
  • The model omits hair geometry, full internal-mouth shape variance, internal skull geometry, and volumetric skin, and retains classical GPA and PCA.The authors note that nonlinear models may be preferable.Source: §9 · p. 15
Loading 1911.08008v2…