Learning Formation of Physically-Based Face Attributes

• High-fidelity face modeling requires costly specialized scanning and skilled teams, while restrictive studio data policies and the absence of a shared, sovereignty-respecting contribution platform leave public models less faithful than production assets.Source: §1 · p. 1
• Prior face methods [35] [43] [39] lack submillimeter detail, anatomical coverage, or VFX-ready materials; PCA and bilinear models are unsuitable for high-resolution data.High-resolution geometry, material capture, and automatic registration traditionally require extensive manual work, while PCA and bilinear models are limited by memory, computing power, and smoothing from linearity.Source: §1 · p. 2
• Light Stage scans provide diffuse albedo, specular intensity, and 4K displacement; automated registration aligns 26 expressions per subject, supporting a nonlinear face model with linear head, eyes, and teeth.Source: §1 · p. 2
• Identity and expression are fully disentangled into two low-dimensional feature vectors; a joint discriminator couples geometry and albedo while separate discriminators preserve each modality’s quality, and upscaling enables 4K texture maps.Source: §1 · p. 2
• The contributions include the first published upscaling of 4K physically based face assets and a cascading low-dimensional face model for identity, expression, and materials, supporting full-3D real-time and offline rendering with more anatomical parts.Source: §1 · p. 2

Facial Capture Systems

• Polarized spherical gradient illumination scanning remains state-of-the-art for high-resolution facial capture, using an MVS prior and photometric stereo to estimate pore-level geometry and material maps.Its reflectance model separates specular and subsurface skin reflections, supporting estimates of diffuse albedo and specular intensity.Source: §2 · p. 2

3D Morphable Face Models

• Blanz and Vetter modeled dense face geometry and texture as separate PCA models; the Basel Face Model and later models extended this approach to more diverse scans.The original models were learned from around 200 scans, while the later linear model used around 10,000 scans.Source: §2 · p. 2
• Vlasic et al. and Cao et al. used tensor-based expression models with discrete, corresponding poses; Li et al. introduced FLAME with pose-dependent corrective blendshapes.FLAME also includes global identity and expression blendshapes learned from many 4D scans.Source: §2 · pp. 2–3
• Locality-based models provide adaptive deformation modes, while Ekman’s FACS represents facial movements as action units associated with the full range of muscle motion.Neumann et al. extract sparse localized modes, and Brunton et al. [10] use localized multilinear wavelet modes.Source: §2 · p. 3
• Existing morphable face models are limited by source resolution and linearity, which smooth fine geometry, and generate texture and geometry separately, risking artifacts from mismatched attributes.The passage gives pairing an African-influenced albedo with Asian-influenced geometry as an example of such an artifact.Source: §2 · p. 3

Image-based Detail Inference

• Image-based methods infer facial detail through texture synthesis, statistical models, local geometry regression, neural feature correlation, and GAN-based synthesis.Yamaguchi et al. infer reflectance maps and displacement from a single image, while these methods lack explicit parametric control of generated results.Source: §2 · p. 3

3.1. Data Capturing and Processing

• Photometric stereo and polarization-based monochrome color reconstruction support pore-level geometry and reflectance capture with a 25-camera system.The setup includes eight 12MP monochrome cameras, eight 12MP color cameras, and nine 4MP monochrome cameras; additional cameras support stereo base-mesh reconstruction.Source: §3.1 · p. 3
• Across 178 subjects, 79 newly scanned participants performed 26 expressions from a condensed 40-action-unit FACS set [14], while 99 Triplegangers full-head scans each included 20 expressions.Source: §3.1 · p. 3
• Neutral meshes are reconstructed with MVS, fitted using a PCA basis and Laplacian deformation, then expressions are registered with blendshapes, non-rigid ICP, and optical-flow constraints.The PCA model combines and extrapolates Basel and Face Warehouse; optical flow from 13 virtual camera views helps preserve texture-space correspondence.Source: §3.1 · pp. 3–4

3.2. Training Data Preparation

• The generic model combines geometry with 4K albedo, specular-intensity, and displacement maps, while 3D vertex positions are rasterized into a 256 × 256 HDR bitmap for joint geometry–albedo learning.The learned face area contains 11,892 vertices.Source: §3.2 · p. 4, Figure 4 · p. 4
• To augment identity training from 178 individuals, the pipeline transfers skin tones between same-ethnicity and gender samples and adds small neutral-geometry offsets; expression training adds random expression offsets.Albedo augmentation also uses image enhancement to improve quality and remove artifacts; geometry offsets use FaceWarehouse expression components with random weights below ±0.5 standard deviations.Source: §3.2 · p. 4

4. Generative Model

• Identity and expression latent codes generate neutral albedo and geometry plus expression offsets; a transfer network infers 1K specular and displacement maps before maps are upscaled to 4K.Middle-frequency geometry remains at 256 × 256 because that resolution suffices for its details.Source: §4 · p. 4, Figure 6 · p. 5

4.1. Identity Network

• The Style-GAN-based identity network models geometry–albedo correlation using individual map discriminators and a joint discriminator to promote sharp, consistent identities.The joint discriminator learns the correlated distribution of paired albedo and geometry.Source: §4.1 · p. 5, Figure 7 · p. 5

4.2. Expression Network

• Expression offsets make expression learning independent of identity; a regressor trained on 25 blendshape activation units makes each latent dimension control its corresponding expression.The regression loss encourages orthogonality of blending units, alongside L2 offset-map reconstruction and adversarial losses.Source: §4.2 · p. 5, Figure 8 · p. 5

4.3. Inference and Super-resolution

• Albedo and geometry jointly constrain inference of 1K specular and displacement maps; cascading upsampling from 256 to 1K and then 4K speeds training and enables higher-resolution results.Displacement is separated into high- and low-frequency components; albedo is super-resolved while geometry is bilinearly interpolated to 1K.Source: §4.3 · p. 6

5. Implementation Details

• The framework is implemented in PyTorch and trained on two NVIDIA Quadro GV100s, following Style-GAN [24] with Expression-network modifications.The Expression network skips progressive training and noise injection because expression offsets are distinguishable only at relatively high resolutions and the latent code Zexp enables control of generated results.Source: §5 · p. 6
• The regression module matches the Expression discriminator’s structure except for its final-layer channel count, is pretrained on synthetic unit-expression data, and is finetuned on scanned expressions.During training, Rexp remains fixed without parameter updates.Source: §5 · p. 6
• The Expression network trains for 40 hours at batch size 128 on 256x256-pixel images, while the Identity network trains for 80 hours across 8x8-to-256x256 images.The Identity network progressively reduces batch size from 1024 to 128 as image size grows.Source: §5 · p. 6

6.2. Qualitative Evaluation

• Identity interpolation jointly reflects albedo and geometry and produces subjects from a natural statistical distribution, while expression interpolation smoothly varies combinations of blendshapes.Source: §6.2 · p. 6, Figure 9 · p. 6, Figure 10 · p. 6
• Generated models’ albedos differ from those of their geometrically nearest training-set neighbors, supporting the claim that generated models are not mere recreations of training examples.Source: §6.2 · p. 6, Figure 12 · p. 7

6.1. Results

• Rendered random samples provide assets for physically based rendering, and adding albedo, specular intensity, and displacement demonstrates each component’s effect.The samples are rendered under novel image-based HDRI lighting.Source: §6.1 · p. 6, Figure 11 · p. 7

6.3. Quantitative Evaluation

• The identity model produces more plausible rendered faces from jointly generated albedo and geometry than from random pairings, approaching ground-truth pairs in FID and IS evaluation.The evaluation compares randomly paired, model-generated paired, and ground-truth albedo and geometry; the supplied passages give no metric values.Source: Table 2 · p. 7, §6.3 · p. 7
• 2.8 mm average Hausdorff distance when fitting 48 faces from indicates the identity model generalizes to unseen faces.Source: §6.3 · p. 7
• 1.2 mm average MSE for expression fitting, versus 2.4 mm for FaceWarehouse, shows the non-linear model outperforms a linear model of the same dimensionality.The comparison uses expression recoveries for Light Stage scans, with FaceWarehouse’s 25 fitting weights fed to the expression network.Source: §6.3 · pp. 7–8
• The gender and age distributions of 5,000 generated samples align well with those of the training data.Source: §6.3 · p. 8, Figure 13 · p. 8

6.4. Applications

• The model reconstructs novel scanned faces through latent-code fitting after landmark alignment and UV parameterization, and enhances low-resolution geometry and albedo.The authors note that the fitting test does not establish registration of unconstrained data; enhancement uses both MSE and discriminator losses.Source: §6.4 · p. 8, Figure 15 · p. 8

Conclusion

• The model generates novel subjects and expressions controllably and performs well in mesh registration and low-resolution data enhancement.The authors aim for the database and model to support higher-quality face image rendering in analysis-by-synthesis research.Source: §7 · p. 8

Limitations and Future work

• Separately modeling expression and identity omits individual middle-frequency expression geometry; future work will model their correlation.Different subjects can have unique representations of the same action unit.Source: §7 · p. 8
• Expression generation requires neural-network inference and 3D geometry re-sampling, making it less user-friendly than blendshapes; animation retargeting remains untested conclusively.Source: §7 · p. 8
• Fitting the identity model to 2D imagery requires differentiable rendering, while jointly fitting lighting and shape may be difficult with complex materials.Image-based relighting with a neural network is proposed as a possible solution [41].Source: §7 · p. 8
• Planned extensions include variable eye textures, hair, and anatomical components such as skull, jaw, and neck joints, using collaborative data sources.The authors also plan to explore federated learning, homomorphic encryption, and zero-knowledge proofs to encourage democratization and wider use.Source: §7 · p. 8

Gender Control

• A classifier trained on ground-truth albedo and geometry maps labels 10k identity-network sample pairs as male or female.The samples are generated from randomly sampled identity latents.Source: Gender Control · p. 11
• The method computes separate mean latent vectors, Zmale and Zfemale, for the samples assigned to each gender category.It obtains the category means by averaging the corresponding identity-latent samples.Source: Gender Control · p. 11
• The gender activation function Ω converts classifier outputs into binary values, with 1 denoting female and 0 denoting male.The category means are computed separately using samples with activation values 1 and 0.Source: Gender Control · p. 11
• Conditioned generation combines a random identity latent Zid with the mean gender latent codes Zmale and Zfemale.This replaces using the randomly sampled identity latent alone as input.Source: Gender Control · p. 11
• Setting α = 0.5 and β = 0.0 produces all-male results, while α = 0.0 and β = 0.5 produces all-female results.These settings control the mixture of the gender latent codes.Source: Gender Control · p. 11
• Gradually decreasing α while increasing β interpolates generated results from male to female.The paper shows an example in Fig. 9.Source: Gender Control · p. 11

Age Control

• Age control replaces gender classification with a regressor that predicts age in years and averages latent vectors from predicted ages over 50 and under 30.These averages define Z_old and Z_young, respectively.Source: Age Control · p. 11
• The age latent code combines identity, old, and young vectors as Z_age = (1 − α − β) × Z_id + α × Z_old + β × Z_young.Source: Age Control · p. 11
• Aging interpolation gradually increases α from 0.0 to 0.7 while decreasing β from 0.7 to 0.0.Source: Age Control · p. 12

3D Model Fitting

• Input scans or models are converted to albedo and geometry maps using a linear face model, Laplacian warping, and attribute transfer.The fitting target is the input’s ground-truth latent code, Zid.Source: 3D Model Fitting · p. 12
• Fitting searches for a latent code that approximates the input code by minimizing mean squared error between their generated geometries.The optimized code is intended to retain the embodiment of the model.Source: 3D Model Fitting · p. 12
• The fitting procedure first updates the latent input with Adam at learning rate 1.0, then adjusts noise-injection-layer variables at learning rate 0.01 for details.Figure 10 shows geometry fitting results.Source: 3D Model Fitting · p. 12

Low-quality Data Enhancement.

• For low-resolution enhancement, the input is encoded with the fitting method, while the remaining high-fidelity assets are generated by the generative pipeline.Unlike true-to-ground-truth fitting, enhancement balances MSE with a discriminator loss weighted 0.001 to constrain realism and quality.Source: Low-quality Data Enhancement · p. 12
• The low-quality enhancement process iteratively feeds the updated latent code back as input until convergence, typically after about 4000 iterations.The supplied passage gives no further optimizer settings for this process.Source: Low-quality Data Enhancement · p. 12

Real Time Rendering Assets

• A real-time Unreal Engine 4 rendering demonstrates lacrimal fluid, blend-mesh, and eye-occlusion assets alongside scan-derived eye and surrounding skin geometry and material.The materials and shaders are adopted from the Digital Human project.Source: Real Time Rendering Assets · p. 12