PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Formation of Physically-Based Face Attributes

Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, Hao Li

arXiv:2004.03458v2cs.CV

TL;DR

High-fidelity face modeling is costly and existing models often lack detailed anatomy, physically based materials, or geometry–albedo coupling. The paper introduces a nonlinear generative model trained from high-resolution scans, combining controllable identity and expression with coupled geometry and materials. It supports physically based rendering and applications including fitting, registration, and low-resolution enhancement, while retaining limitations in expression–identity correlation and usability.

  • Problem

    Existing face models can lack sub-millimeter geometry, anatomical coverage, physically based materials, and consistent geometry–texture correlation, despite costly high-fidelity data acquisition.

  • Method

    The paper builds a hybrid nonlinear 3D morphable model from high-resolution scans, using joint and separate discriminators to couple geometry with albedo while disentangling identity and expression.

  • Results

    The model generates novel subjects and expressions with physically based assets and performs well for mesh registration, fitting, and low-resolution data enhancement.

  • Takeaways & Limitations

    The released model provides higher-quality face assets for physically based rendering and analysis-by-synthesis research.

  • Takeaways & Limitations

    Expression and identity are modeled separately, so reconstructed expression offsets omit subject-specific middle-frequency expression geometry.

Abstract

from arXiv · show

Based on a combined data set of 4000 high resolution facial scans, we introduce a non-linear morphable face model, capable of producing multifarious face geometry of pore-level resolution, coupled with material attributes for use in physically-based rendering. We aim to maximize the variety of face identities, while increasing the robustness of correspondence between unique components, including middle-frequency geometry, albedo maps, specular intensity maps and high-frequency displacement details. Our deep learning based generative model learns to correlate albedo and geometry, which ensures the anatomical correctness of the generated assets. We demonstrate potential use of our generative model for novel identity generation, model fitting, interpolation, animation, high fidelity data visualization, and low-to-high resolution data domain transferring. We hope the release of this generative model will encourage further cooperation between all graphics, vision, and data focused professionals while demonstrating the cumulative value of every individual's complete biometric profile.

1. Introduction

The paper addresses the cost, limited fidelity, and missing material–geometry coupling of existing face models by introducing a high-resolution, physically based generative model. Its hybrid nonlinear design supports controllable identity and expression generation with anatomically consistent materials and geometry.

  • 1. Introduction: A unified model is intended to democratize high-fidelity asset generation, shorten production cycles, and improve quality and consistency.The motivation is the cost of scanning, specialized equipment, skilled labor, and restrictive studio data policies.
  • 1. Introduction: Existing face models lack sub-millimeter detail, comprehensive anatomy, or physically based material properties needed for modern VFX pipelines.The paper also identifies ambiguities in estimating material intrinsics from unconstrained imagery.
  • 1. Introduction: The proposed model combines high-resolution facial geometry, material capture, automatic registration, and compact nonlinear representation for controlled reconstruction.The authors use a hybrid approach: nonlinear modeling for face deformations and linear PCA models for head, eyes, and teeth.
  • 1. Introduction: A joint discriminator correlates geometry and albedo while separate discriminators preserve their individual quality, enabling recovered 4K skin-intrinsic maps.The model uses separate low-dimensional feature vectors to disentangle identity from expression.
  • 1. Introduction: The contributions include the first published 4K physically based face-asset upscaling and a cascading model controlling identity, expressions, and surface materials.The paper also presents a morphable model designed for full 3D real-time and offline rendering with more anatomical parts.

2. Related Work

Prior work established morphable face models, high-resolution capture, and image-based detail inference, but commonly separated geometry from texture and remained limited by linear smoothing and source resolution. This paper’s context motivates jointly modeling anatomically related geometry and materials.

  • 2. Related Work: Polarized spherical-gradient illumination remains a high-resolution capture approach because it supports rapid acquisition, material capture, and pore-level geometry.Photometric stereo separates specular and subsurface reflections to estimate diffuse albedo, specular intensity, and detailed geometry.
  • 2. Related Work: Early morphable models represented geometry and texture with separate PCA models, while later multilinear approaches jointly represented identity and expression variations.The cited models progressed from roughly 200 scans to larger linear models and tensor-based expression representations.
  • 2. Related Work: Existing morphable face models are limited by source resolution and linear smoothing, and typically generate texture and geometry separately.The paper notes that ignoring age-, gender-, and ethnicity-related correlations can produce mismatched albedo and geometry.
  • 2. Related Work: Image-based methods infer coarse through mesoscopic geometry and facial reflectance maps, complementing but not replacing high-fidelity captured assets.The surveyed approaches include texture synthesis, statistical detail models, GAN-based geometry inference, and single-image reflectance estimation.

3. Database

The database combines Light Stage capture, automated registration, and diverse subjects and expressions into a hybrid high-resolution face representation. Its generic model includes anatomical components and 4K material and displacement maps while using reduced-resolution base geometry for efficient learning.

  • 3. Database: The Light Stage system uses photometric stereo and polarization with 25 cameras to capture pore-level geometry and reflectance maps for database-scale acquisition.The camera array includes eight 12MP monochrome, eight 12MP color, and nine 4MP monochrome cameras.
  • 3. Database: The dataset contains 79 Light Stage subjects and 99 Triplegangers head scans, with the first group performing 26 expressions and the second 20.The 79 subjects include 34 female and 45 male participants aged 18–67.
  • 3. Database: Registration begins with an MVS-reconstructed neutral mesh, followed by PCA fitting, Laplacian deformation, manual corrections, and non-rigid ICP for expressions.The resulting neutral scans are added to the PCA basis as new scans are registered.
  • 3. Database: The generic model represents face, head, neck, teeth, gums, eyes, lacrimal fluid, occlusion, and eyelashes alongside 4K albedo, specularity, and displacement maps.These components support physically based rendering across multiple anatomical and material assets.
  • 3. Database: A 256 × 256 base-geometry representation recovers middle-frequency detail with a mean Hausdorff error of 0.068mm relative to 4K reconstruction.The lower-resolution representation reduces training-data load while retaining the targeted detail range.

4. Generative Model

The framework separates identity and expression generation while coupling geometry with albedo and inferring high-resolution material maps for physically based rendering.

  • Generative Pipeline: Latent identity and expression codes generate a neutral face and controllable expression offsets, respectively.The identity network samples Zid, while the expression network uses blendshape weights Zexp and adds offsets to neutral geometry.
  • Identity Network: The identity network uses separate albedo and geometry discriminators plus a joint discriminator to preserve individual quality and cross-modal consistency.The joint discriminator learns the correlated distribution of paired albedo and geometry, while the separate discriminators maintain map quality and sharpness.
  • Expression Network: The expression network represents expressions as identity-independent vector offset maps controlled by 25 orthogonal facial activation weights.A pretrained regressor predicts expression weights from generated outputs, encouraging each latent dimension to control its corresponding blendshape expression.
  • Expression Network: Expression training combines offset-map reconstruction and adversarial losses, with the regression-based loss backpropagated to enforce orthogonality.The cited passages identify L2 reconstruction and discriminator losses as components of expression training.
  • Material Generation: A transfer network infers 1K specular and displacement maps from upscaled 1K albedo and geometry, using geometry as an additional constraint.Displacement is separated into high- and low-frequency components before inference.

5. Implementation Details

The implementation follows StyleGAN-based training with expression-specific modifications for controllability and high-resolution offset learning.

  • Implementation Details: Expression training bypasses progressive growing and removes noise injection because high-resolution offsets and the input latent code provide distinguishability and control.The framework is implemented in PyTorch and trained on two NVIDIA Quadro GV100 GPUs.

6. Experiments And Evaluations

Experiments assess rendering quality, interpolation, novelty, distribution coverage, fitting, expression reconstruction, and low-quality scan enhancement. The model generates plausible coupled assets, fits unseen faces, and improves expression reconstruction over a linear baseline.

  • Results: The generative model directly outputs the assets needed for physically based rendering, including geometry, albedo, specular intensity, and displacement.Rendered examples demonstrate the generated model and the effect of adding each component.
  • Interpolation: Identity interpolation changes age and gender while jointly interpolating albedo and geometry toward subjects in a natural statistical distribution.The identity interpolation is nonlinear rather than simple linear blending.
  • Applications: The model supports generation of novel identities, nearest-neighbor analysis, scan fitting, and enhancement of low-resolution geometry and albedo.Additional experiments compare generated albedos with geometrically nearest training examples and transfer low-quality scans into the model domain.
  • Quantitative Evaluation: An average Hausdorff distance of 2.8mm when fitting 48 unseen faces indicates capacity beyond the training set.The evaluation fits faces from an external source dataset.
  • Quantitative Evaluation: 1.2mm MSE versus FaceWarehouse’s 2.4mm demonstrates lower expression-reconstruction error for the non-linear model.The comparison uses models with the same dimensionality.
  • Quantitative Evaluation: Generated identity samples align well with the training data’s gender and age distributions across 5000 random samples.The comparison is shown between the original training data and randomly generated samples.

7. Conclusion and Limitations

The model generates novel subjects and expressions controllably and performs well for mesh registration and low-resolution data enhancement, while several usability and modeling limitations remain.

  • Conclusion: The model generates novel subjects and expressions in a controllable manner and performs well for mesh registration and low-resolution data enhancement.
  • Limitations and Future work: Separating identity and expression without modeling their correlation omits individual-specific middle-frequency expression geometry from reconstructed offsets.
  • Limitations and Future work: Expression generation requires neural-network inference and 3D-geometry resampling, making it less user-friendly than blendshape modeling.
  • Limitations and Future work: Fitting the identity model to 2D imagery requires differentiable rendering, while jointly fitting lighting and shape becomes less tractable with complex materials.

Appendix

The supplied appendix passage is a bibliographic reference to a 2019 IEEE conference paper on face images with head-mounted displays.

  • Appendix: The passage identifies a paper concerning face images in the presence of head-mounted displays.
  • Appendix: The cited work appeared in the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces.
  • Appendix: The reference lists pages 267–276 and IEEE as the publisher.

Gender Control

Gender control averages identity latent codes classified as male or female, then blends those category means with a sampled identity code for controlled generation and interpolation.

  • Gender Control: The method samples 10k generated identity pairs, classifies them as male or female, and averages each category to obtain Zmale and Zfemale.
  • Gender Control: The gender activation function maps classifier outputs to binary values, with 1 denoting female and 0 denoting male.
  • Gender Control: Conditioned generation combines a randomly sampled identity latent code with the mean gender latent codes instead of using the sampled code directly.
  • Gender Control: Setting α = 0.5, β = 0.0 produces male generations, while α = 0.0, β = 0.5 produces female generations.
  • Gender Control: Gradually decreasing α while increasing β interpolates a generated identity from male to female.

Age Control

Age control replaces gender classification with age regression, averages latent codes for older and younger samples, and interpolates between them through coefficient changes.

  • Age Control: Age control uses a regressor predicting age in years and averages latent codes from samples older than 50 and younger than 30.
  • Age Control: The age-conditioned identity code combines a sampled identity latent with separate old and young mean vectors using α and β.
  • Age Control: The eye closeups retain the eyeball and eyelashes as default geometry while successively adding the model’s additional eye geometries.
  • Age Control: Figure 9 demonstrates aging interpolation by increasing α from 0.0 to 0.7 while decreasing β from 0.7 to 0.0.

3D Model Fitting

The fitting procedure maps a scanned face into the model’s latent space and iteratively optimizes its identity code, while low-resolution enhancement balances reconstruction with generative realism.

  • 3D Model Fitting: The model fits a scanned face by converting it into albedo and geometry maps, then optimizing a latent identity code against the input representation.The fitting objective minimizes mean squared error between generated and input identity representations.
  • 3D Model Fitting: Adam first updates the input latent code with learning rate 1.0, after which noise-injection variables refine fine details using learning rate 0.01.The fitting process repeats until convergence after about 4000 iterations.
  • 3D Model Fitting: For low-resolution data enhancement, the method adds discriminator loss to the MSE objective instead of enforcing true-to-ground-truth reconstruction.The discriminator loss receives an empirical weight of 0.001 relative to the MSE loss.
  • 3D Model Fitting: The model demonstrates real-time rendering of eye-region scan assets, including lacrimal fluid, blend mesh, eye occlusion, skin geometry, and materials in Unreal Engine 4.Materials and shaders are adopted from the Digital Human project.
Loading 2004.03458v2…