Source-linked AI summary
Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video
Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong, Yudong Guo, Juyong Zhang
TL;DR
Personalized dynamic head reconstruction from monocular video remains challenging, especially for representing individualized geometry and non-face details. This paper addresses the problem with a semantic facial NeRF built from disentangled voxel-field bases and expression coefficients, producing personalized facial attributes and photo-realistic results with efficient construction and rendering.
Problem
Personalized head models must represent dynamic faces while preserving individualized geometry and texture that generalized or mesh-based models often miss, including hair and teeth.
Method
The method represents a personalized semantic head as multi-level voxel-field radiance bases combined with low-dimensional expression coefficients and interpreted by an MLP for volume rendering.
Results
The model reconstructs subject-specific 4D avatars with richer personalized facial attributes and photo-realistic results than mesh-based blendshape models, while supporting facial reenactment applications.
Takeaways & Limitations
The representation supports detailed facial motion and attributes such as hair, wearings, and muscle movements, with potential use in photo-realistic animation.
Takeaways & Limitations
Extrapolating expression coefficients far beyond the training distribution can produce local artifacts, and tracking errors may cause loss of details.
Abstract
from arXiv · showhide
We present a novel semantic model for human head defined with neural radiance field. The 3D-consistent head model consist of a set of disentangled and interpretable bases, and can be driven by low-dimensional expression coefficients. Thanks to the powerful representation ability of neural radiance field, the constructed model can represent complex facial attributes including hair, wearings, which can not be represented by traditional mesh blendshape. To construct the personalized semantic facial model, we propose to define the bases as several multi-level voxel fields. With a short monocular RGB video as input, our method can construct the subject's semantic facial NeRF model with only ten to twenty minutes, and can render a photo-realistic human head image in tens of miliseconds with a given expression coefficient and view direction. With this novel representation, we apply it to many tasks like facial retargeting and expression editing. Experimental results demonstrate its strong representation ability and training/inference speed. Demo videos and released code are provided in our project page: https://ustc3dv.github.io/NeRFBlendShape/
1 INTRODUCTION
The paper develops a personalized semantic facial NeRF to overcome mesh blendshape limitations in representing detailed, non-face attributes and to enable efficient dynamic-head modeling from monocular video.
- Motivation: Mesh-based parametric models provide intuitive semantic control but struggle with personalized non-face regions such as hair and teeth.They also require approximated differentiable rendering under RGB supervision, while deformation transfer has additional limitations.
- Motivation: Existing generative NeRF models couple expression, identity, and appearance, whereas generic semantic models lack personalized facial dynamics and details.The paper positions personalized, controllable representation as the missing combination.
- Method: The proposed model represents each semantic expression basis as a multi-level voxel-field radiance field and combines bases using expression coefficients.An MLP interprets the voxel field for volume rendering, while multi-resolution hash tables store features efficiently.
- Method: An expression-aware density-grid update strategy accelerates ray marching and focuses optimization on regions likely occupied by the head.This addresses the difficulty of using static density information for dynamic expressions.
- Applications: The representation supports applications including facial reenactment and can model personalized attributes such as muscle actions and detailed texture.The authors associate these capabilities with photorealistic digital avatars and potential photo-realistic animation use.
2 RELATED WORK
Related work spans parametric head models, human portrait synthesis, NeRF-based representations, and voxel acceleration, exposing trade-offs among controllability, personalization, rendering quality, and efficiency.
- Parametric Head Model: Parametric head models use low-dimensional spaces and blendshape-like bases to represent identity and expression, but explicit meshes have limited capacity for detailed personalized attributes.The related work traces extensions from PCA-based 3DMMs to multilinear, nonlinear, and user-specific blendshape models.
- Parametric Head Model: Implicit representations improve the capacity and visual quality of digital human models, yet generic models can lack personalized facial details and personalized NeRFs may require long training or inference.The paper contrasts its emphasis on photorealistic rendering and efficient training/inference with prior implicit blendshape approaches.
- Human Portrait Synthesis: Image-based portrait methods can produce artifacts under large pose or expression changes because they represent 3D deformation in 2D space.Morphable-model approaches instead use parametric 3D representations for digital portrait synthesis.
- Neural Radiance Field: NeRF represents scenes with an MLP and differentiable volume rendering, supporting 3D reconstruction, 4D synthesis, and digital human modeling.Generative head models provide pose control but do not support expression editing under their generative-adversarial training strategy.
- Voxel Representation for NeRF Acceleration: Voxel fields reduce NeRF training burden through local features and can accelerate inference by storing spatial information such as density distributions.Multi-level hash tables further compress voxel features while augmenting shallow MLPs.
3 METHOD
The method represents a personalized semantic head as expression-driven NeRF bases stored in multi-level voxel fields, then renders images through hash encoding, a lightweight MLP, and volume rendering. It reconstructs the model from tracked monocular video using staged losses and training strategies designed for detailed, efficient dynamic-head modeling.
- 3.1 NeRF based Linear Blending Representation: Each semantic basis is a radiance field for a specific expression, represented by a multi-level voxel field and combined using low-dimensional expression coefficients.The representation preserves blendshape-like semantic control while modeling personalized facial details, including non-face regions.
- 3.1 NeRF based Linear Blending Representation: The model combines expression bases in latent voxel space, using multi-resolution hash tables to encode features at multiple scales.Unlike mesh blendshape, basis blending occurs in multi-level voxel space before the implicit function processes the features.
- 3.2 Rendering: For each sampled 3D point, hash interpolation produces features that a four-layer, 64-neuron MLP maps to density and color for volume rendering.The encoding uses the sampled point and combined hash table, while view direction is represented with spherical-harmonics positional encoding.
- 3.3 Construction: The construction pipeline tracks expressions, poses, and camera parameters from monocular RGB video, freezes tracked expression coefficients, and optimizes the hash tables and MLP.Head pose is used as the corresponding frame’s extrinsic camera parameter, and selected frames receive generated head masks.
- 3.3.2 Training: Training combines photometric, mask, and perceptual losses, with staged weighting and patch sampling to learn geometry, colors, and personalized details.Mask supervision is emphasized initially, photometric loss is used for fine geometry and color, and perceptual supervision improves robustness to misalignment and shading variation.
- 3.3.2 Training: A shared density grid is needed because the density field of one expression may not cover heads under other expressions.The implementation computes density information across bases rather than using a single expression’s density field to guide ray marching.
4 EXPERIMENTS
Experiments evaluate reconstruction quality, facial reenactment, semantic correspondence, training efficiency, geometry, and novel-view consistency. Across these settings, the model preserves personalized details while producing high-fidelity dynamic heads efficiently.
- 4.2 Comparison: The model reconstructs high-fidelity expressions and facial details, with fine-level head details represented and rendered consistently with the ground truth.
- 4.2 Comparison: It outperforms existing methods on every reported reconstruction-error metric, including MSE, L1, PSNR, SSIM, and LPIPS.
- 4.2 Comparison: Cross-identity reenactment transfers source expression coefficients to target bases, producing more personalized details and more reasonable heads than competing methods.
- 4.3 Comparison with “Concatenate” Operation: The model learns a dynamic head scene in less than 20 minutes, while the concatenate baseline and NerFACE obtain no plausible result within that time.
- 4.4 Geometry Visualization: Marching-cubes geometry extraction produces reasonable eyes, noses, and detailed hair without direct normal or depth supervision.
- 4.5 Semantic Meaning of Bases: Its bases retain mesh-blendshape semantic correspondence while adding personalized attributes such as hair, moles, wrinkles, and folds.
- 4.6 Novel View Synthesis: Disentangled camera parameters enable novel-view synthesis with remarkable multi-view consistency from expression-conditioned radiance fields.
5 ABLATION STUDY
The ablation study examines perceptual loss, density-grid updating, and the model’s geometry and basis behavior. These components improve visual detail, dynamic-expression handling, and rendering quality, although extracted surfaces remain noisy.
- 5.1 Discussion on Perceptual Loss: Perceptual loss improves rendering quality and personalized facial attributes by preserving feature similarity between predictions and ground-truth images.
- 5.2 Density Grid: A dynamic density-grid update strategy handles expressions such as mouth opening better than updating the grid from only a static neutral head.
- 5.4 Semantic Bases: The model’s bases preserve the semantic expression correspondence of mesh blendshapes while rendering more personalized facial details.
- 5.3 Geometry: The extracted density-field surface is geometrically reasonable and captures eyes, noses, and detailed hair, but noise remains on the surface.
- 5.2 Density Grid: Using a density grid yields better PSNR than omitting it during training.
6 LIMITATIONS
The model has practical limitations tied to expression extrapolation, input quality, and challenging non-rigid hair motion. Its ablations also examine view consistency, perceptual loss, density-grid updates, and density-grid training.
- Limitations: Artifacts may occur in local regions when expression coefficients are extrapolated far beyond the training distribution.The authors leave explicit modeling of underlying geometry for future work as a possible way to address this issue.
- Limitations: Large tracking errors may cause the reconstructed model to lose facial details.Camera parameters and input conditions are important for NeRF-based reconstruction.
- Ablation and analysis: The model is evaluated as a 3D-consistent representation that allows freely adjusting the view direction.This figure addresses view control rather than a failure case.
- Ablation and analysis: Within the same training time, rendering quality is better with a density grid than without one.The ablation compares training with and without the density grid.
- Limitations: Fast and heavy non-rigid hair deformation can produce artifacts in the hair region.The stated boundary is the lack of a non-rigid deformation condition.
7 CONCLUSION
The paper presents a personalized semantic facial NeRF model that reconstructs subject-specific 4D avatars from minutes of monocular RGB video. Compared with mesh-based blendshape and other NeRF-based parametric head models, it reports photorealistic rendering, richer personalized details, shorter construction time, and improved rendering quality.
- Conclusion: The proposed model reconstructs subject-specific 4D avatars from only minutes of monocular RGB video.It represents personalized attributes such as hair, wearings, and muscle movements.
- Conclusion: Compared with mesh-based blendshape, the model generates photorealistic results and represents more personalized facial attributes.The cited examples include hair, wearings, and muscle movements.
- Conclusion: Compared with other NeRF-based parametric head models, the model requires less construction time, provides better rendering quality, and contains richer facial details.
- Conclusion: The implicit linear blending architecture may extend to problems where linear combination relationships are important.The paper links this potential to the architecture's representation ability and training efficiency.