PaperlayerView exact revision on arXiv

Source-linked AI summary

i3DMM: Deep Implicit 3D Morphable Model of Human Heads

Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, Christian Theobalt

arXiv:2011.14143v1cs.CVcs.GRcs.LG

TL;DR

Existing 3D morphable models largely focus on faces and rely on mesh templates requiring dense correspondence, whereas full-head modeling must also represent hair. i3DMM addresses this with an implicit full-head model trained from aligned scans, and demonstrates semantic control, dense correspondence, texture transfer, and reconstruction across identities and expressions.

  • Problem

    Existing 3D morphable models primarily model faces, while dense correspondence for full heads and hair is difficult to compute.

  • Method

    i3DMM uses implicit neural functions with separated reference shape, deformation, geometry, and color components to model full-head identities, expressions, hairstyles, and appearance.

  • Results

    i3DMM captures full-head geometry and appearance, establishes correspondences across subjects, expressions, and hairstyles, and supports texture transfer and semantic head applications.

  • Takeaways & Limitations

    The model provides semantic full-head control and correspondence-based applications while avoiding dense ground-truth correspondences during training.

  • Takeaways & Limitations

    Hair is only coarsely approximated physically, hairstyles covering the ear collapse the ear to a single point, and hair scans are noisy.

Abstract

from arXiv · show

We present the first deep implicit 3D morphable model (i3DMM) of full heads. Unlike earlier morphable face models it not only captures identity-specific geometry, texture, and expressions of the frontal face, but also models the entire head, including hair. We collect a new dataset consisting of 64 people with different expressions and hairstyles to train i3DMM. Our approach has the following favorable properties: (i) It is the first full head morphable model that includes hair. (ii) In contrast to mesh-based models it can be trained on merely rigidly aligned scans, without requiring difficult non-rigid registration. (iii) We design a novel architecture to decouple the shape model into an implicit reference shape and a deformation of this reference shape. With that, dense correspondences between shapes can be learned implicitly. (iv) This architecture allows us to semantically disentangle the geometry and color components, as color is learned in the reference space. Geometry is further disentangled as identity, expressions, and hairstyle, while color is disentangled as identity and hairstyle components. We show the merits of i3DMM using ablation studies, comparisons to state-of-the-art models, and applications such as semantic head editing and texture transfer. We will make our model publicly available.

1. Introduction

i3DMM is an implicit full-head morphable model that includes hair and learns semantic controls from rigidly aligned noisy scans without dense ground-truth correspondences. Its architecture separates geometry and color while supporting dense correspondences and applications including texture transfer and semantic editing.

  • Motivation: Existing 3DMM training commonly requires dense surface correspondence, which is already difficult for faces and close to impossible for the rest of the head and hair.This motivates avoiding dense correspondence computation during training.
  • Contributions: i3DMM models full-head geometry and appearance, including hair deformations, from a new dataset of photogrammetric scans of 64 people with varied expressions and hairstyles.The model extends beyond frontal-face identity, expression, and texture modeling to the entire head.
  • Contributions: The model disentangles geometry into identity, expression, and hairstyle components, while separating color into identity and hairstyle components.Geometry and color are learned through a design intended to provide semantic control over full heads.
  • Contributions: An implicit representation enables training from rigidly aligned noisy scans without computing dense ground-truth correspondence between training shapes.The method uses signed distance functions to represent surfaces and handle large deformations, especially in hair.
  • Applications: Its architecture learns dense correspondences between implicit 3D shapes with only sparse supervision and supports texture transfer, semantic head editing, segmentation, and landmark transfer.The introduction describes these as demonstrated applications of the learned model and correspondences.

2. Related Work

Related work covers mesh-based face and head morphable models, implicit shape and color representations, and implicit 3D hair modeling. Prior models use PCA, multilinear structure, articulated or parts-based templates, graph convolutions, optical flow, and volumetric implicit representations.

  • Face and Head Morphable Models: Classical face morphable models learn low-dimensional shape and texture spaces from registered examples, with later work improving registration, scans, and identity-dependent expression modeling.The cited approaches include PCA-based models and multilinear models.
  • Face and Head Morphable Models: Head models such as FLAME, autoencoder-based models, and LYHM incorporate articulated components, graph convolutions, hierarchical parts, or optical-flow texture refinement.LYHM is built using 1,200 identities, while FLAME combines linear shape with articulated and expression components.
  • Implicit Representation: Implicit-representation research models shape, temporally evolving geometry, and textured shape using signed distance fields, occupancy classification, and generative approaches.DeepSDF uses an autodecoder for signed distance fields, while other methods classify points as inside or outside shapes.
  • 3D Hair Modeling: A related hair-modeling approach learns a manifold of 3D hairstyles represented as implicit surfaces with a volumetric variational autoencoder.

3. Method

i3DMM learns full-head geometry and color from rigidly aligned scans using an implicit reference shape, deformations, and disentangled latent codes. The method combines multi-view scan preparation, signed-distance sampling, and networks for shape deformation, reference geometry, and color.

  • Data acquisition and preprocessing: The model is trained from 64 subjects scanned across facial expressions and hairstyles using a calibrated multi-view camera system.The scans are landmark-annotated, cropped, rigidly aligned, made watertight, and scaled to a common unit cube.
  • Implicit representation: The network maps a 3D query point and latent code to a signed distance and color, representing surfaces as the zero level-set of an implicit signed-distance function.The autodecoder jointly learns network weights and latent codes for the training heads.
  • Training data preparation: Training samples combine uniformly sampled surface points, extra samples in detailed facial regions, and perturbed interior and exterior points with signed-distance and color targets.Surface colors are obtained from the closest mesh point and its texture map.
  • Latent disentanglement: Separate geometry and color latent spaces disentangle geometry into identity, expression, and hairstyle, while color uses identity and hairstyle codes shared across subjects.At test time, the latent spaces can be controlled individually for semantically meaningful editing.
  • Network architecture: The architecture consists of Reference Shape, Shape Deformation, and Color networks, with color learned in the reference space.All three components process sinusoidal positional encodings and use fully connected layers with ReLU activations.
  • Shape representation: A reference shape and a learned deformation network represent each head by deforming query points toward the reference space, enabling dense correspondences without precomputed dense registration.The reference shape provides a shared coordinate space for the individual head geometries.

4. Experiments

Experiments evaluate reconstruction, correspondences, ablations, model comparisons, and applications of i3DMM on unseen scans. Results show generalization to novel identities and expressions, strong full-head reconstruction and color quality, dense correspondence transfer, and semantic editing and annotation applications.

  • 4.1. Reconstruction: i3DMM generalizes to unseen identities and expressions while preserving detailed facial regions and smoothing noisy hair areas.The test set contains six held-out identities with five novel expressions per identity.
  • 4.2. Correspondences: Dense correspondences across subjects, expressions, and hairstyles enable texture transfer between scans.The correspondences use extremely sparse landmark supervision and include long and short hair.
  • 4.3. Ablation Study: Landmark-based sampling, landmark supervision, and joint network training materially affect reconstruction quality in the ablation study.Without landmark-based sampling, the model emphasizes noisy hair and loses facial detail; without landmark supervision, it creates small artificial ear regions.
  • 4.4. Comparisons to Existing Models: i3DMM outperforms FLAME for full-head geometry and quantitatively outperforms BFM on color while achieving similar face-region geometric quality.The comparison uses symmetric Chamfer distance, F-score, and color metrics; Table 1 reports the quantitative evaluation.
  • 4.5. Application: Semantic Head Editing: Semantic latent disentanglement supports editing identity, hairstyles, caps, and mouth interiors while keeping other components fixed.Edits are selected by moving along principal components of variation in separate latent subspaces.
  • 4.6. Application: One-shot Annotation Transfer: Dense correspondences and low reconstruction errors enable one-shot transfer of annotations to model fits and real-world scans, including hair.Annotations are transferred from one manually annotated scan to the reference shape, then to reconstructions and scans via nearest neighbors.

5. Discussion and Future Work

i3DMM models varied hairstyles but only approximates hair physically and compresses ears hidden by hair. Noisy photometric reconstructions also limit hair quality and motivate finer hairstyle control.

  • Hair is represented as a coarse approximation of individual strands, limiting the physical fidelity of hairstyle geometry.
  • Hairstyles covering the ear are modeled by collapsing the ear to a single point rather than representing its layered geometry behind hair.
  • One-shot semantic-segmentation and landmark annotations can be transferred from one real-world head scan to others.
  • Photometric multi-view reconstruction produces noisy hair scans because hair interacts complexly with light, motivating finer-grained hairstyle control.

6. Conclusion

The paper presents a unified implicit morphable model for full-head geometry and appearance across identities, expressions, and hairstyles. Its reference-shape, deformation, and color decomposition supports semantic components and dense cross-scan correspondences.

  • i3DMM unifies geometry and appearance modeling for full heads containing different identities, expressions, and hairstyles.
  • Neural implicit functions are learned from 610 multi-view scans to represent head geometry and color.
  • Separating reference shape, shape deformation, and color components enables semantic disentanglement and dense correspondences across scans.

8.1. Model Visualization

i3DMM provides independent controls for identity, expression, and hairstyle in geometry, plus identity and hairstyle in head color. Identity spaces are visualized with PCA, while expressions follow learned semantic directions.

  • Geometry is independently controlled through identity, expression, and hairstyle components, while color uses identity and hairstyle components.
  • PCA is applied to identity geometry and color latent spaces to compute their principal components.
  • Expression variation is visualized by moving along directions corresponding to the training expressions.

8.2. Experiments

The supplementary experiments describe latent-space sampling and report a male-generation bias linked to the training-set imbalance. Random sampling uses PCA for several components and observed training values for hair.

  • 8.2.1 Sampling i3DMM: Randomly sampling shapes in a parametric space is an important 3D morphable-model capability for generating synthetic training data.
  • 8.2.1 Sampling i3DMM: i3DMM samples color-identity, geometry-identity, and expression latents with PCA, while sampling hair shape and color from their training values.Gaussian perturbations use N(0, 0.1) for color and geometry identity and N(0, 0.25) for expressions.
  • 8.2.1 Sampling i3DMM: The model is biased toward generating male heads because the training dataset contains 46 males and 18 females.The authors report no clear quality loss when fitting female test scans, but warn of possible bias when generated samples train another network.

8.3. Comparisons

The comparisons evaluate i3DMM fitting and correspondence quality against existing models, with carefully defined face and full-head regions. Additional analyses report correspondence errors and ablation results under consistent evaluation masks.

  • Model fitting: i3DMM is compared with BFM and FLAME using masked fitting regions, rigid landmark alignment, and optimization of model and global transformation parameters.BFM is evaluated for the face because it does not model the entire head; FLAME is also evaluated for full-head fitting after neck-region adjustments.
  • Model fitting: The fitting objective jointly uses geometric distances, color differences where available, and eight landmark constraints, with landmark weight w_l = 0.1.Color loss is enforced only for BFM, with channel-wise color scaling optimized through diagonal matrix K.
  • Evaluation protocol: Face-region evaluation uses manually annotated masks, while full-head metrics include the complete head region after removing incompatible neck regions.Hair is excluded from the quantitative ablation evaluation because its ground truth is noisy and small differences are not very indicative.
  • Ablation study: The ablation study compares uniform sampling, removal of landmark supervision, independent color and geometry training, the final i3DMM, and ground truth.The quantitative ablation metrics are computed on the face region using the manually annotated masks.
  • Correspondence evaluation: Correspondence errors average 5.08 mm for BFM-grounded evaluation and 2.36 mm for FLAME-grounded evaluation, with medians of 3.02 mm and 1.83 mm, respectively.These errors include inaccuracies from FLAME and BFM registrations, not only i3DMM correspondence prediction.

8.4. Applications

The applications demonstrate i3DMM-based head completion, semantic hairstyle control, annotation transfer, and rendering of implicit surfaces. These examples use the learned full-head prior for user-guided completion and downstream scan processing.

  • Full Head Completion: i3DMM completes face scans with short hair, long hair, or a cap by using its learned full-head prior.The hairstyle is controlled by regularizing geometry and color hairstyle codes toward hairstyle latent vectors learned during training.
  • Full Head Completion: The completion results generate plausible heads and allow user-guided control, including conversion of face-only 3DMMs into full-head 3DMMs.The paper also identifies monocular 3D reconstruction as a potential application of this prior.
  • Visualization Details: Visualization uses sphere tracing to render signed-distance-field surfaces, avoiding marching-cubes artifacts at the chosen reconstruction resolution.Geometry is shaded with Blinn–Phong reflection, while color renders use gamma correction with γ = 0.65.
  • Annotation Transfer: Annotation transfer applications include front-view and side-view segmentation transfer plus landmark transfer from manually annotated i3DMM reconstructions to head scans.The additional results are illustrated in Figure 13.
  • Additional results: Additional figures compare i3DMM with BFM and FLAME fits and show qualitative ablations against ground truth.The supplied captions identify these comparisons but do not report numerical outcomes.
Loading 2011.14143v1…