Source-linked AI summary
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
Guy Gafni, Justus Thies, Michael Zollhöfer, Matthias Nießner
TL;DR
Faithful 4D facial avatars require realistic appearance together with controllable viewpoint, pose, and expression changes, yet explicit geometry is difficult and image-based methods are limited. The paper combines an implicit neural scene representation with a low-dimensional morphable model and volumetric rendering, learning from monocular input alone. The resulting avatars support photo-realistic novel poses and expressions and surpass state-of-the-art facial reenactment quality.
Problem
Faithful 4D facial reconstruction must represent complex appearance while allowing novel viewpoints, head poses, and expressions for applications such as AR/VR telepresence and video editing.
Method
The method combines a dynamic neural radiance field with a low-dimensional morphable model and volumetric rendering, using pose and expression conditioning from monocular portrait video.
Results
The learned volumetric representation generates photo-realistic images under novel poses and expressions and beats state-of-the-art facial reenactment methods in quantitative and qualitative quality.
Takeaways & Limitations
A single fixed-camera view can support controllable 4D facial-avatar reconstruction without an expensive calibrated multi-view rig.
Takeaways & Limitations
The morphable model does not explicitly model eye blinks, eye movements, or upper-body dynamics.
Abstract
from arXiv · showhide
We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face. Digitally modeling and reconstructing a talking human is a key building-block for a variety of applications. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoints or head-poses is required. In contrast to state-of-the-art approaches that model the geometry and material properties explicitly, or are purely image-based, we introduce an implicit representation of the head based on scene representation networks. To handle the dynamics of the face, we combine our scene representation network with a low-dimensional morphable model which provides explicit control over pose and expressions. We use volumetric rendering to generate images from this hybrid representation and demonstrate that such a dynamic neural scene representation can be learned from monocular input data only, without the need of a specialized capture setup. In our experiments, we show that this learned volumetric representation allows for photo-realistic image generation that surpasses the quality of state-of-the-art video-based reenactment methods.
1. Introduction
The paper introduces dynamic neural radiance fields as an implicit, volumetric representation for controllable 4D facial avatars learned from monocular video. A low-dimensional morphable model supplies pose and expression control, enabling novel head-pose and expression synthesis.
- Contribution: Dynamic neural radiance fields represent 4D facial avatars while handling complex geometry and material properties through neural rendering.The approach combines classical volume rendering with a neural scene representation network.
- Contribution: The method reconstructs a dynamically changing human face from monocular camera recordings rather than specialized multi-view input.The authors describe this as an end-to-end approach using a single camera.
- Contribution: The implicit volumetric representation captures difficult structures such as hair and mouth interiors that classical methods struggle to model with explicit geometry.Its continuous spatial function avoids discretization artifacts associated with voxel grids.
- Contribution: A low-dimensional morphable model provides explicit control over facial pose and expressions for the dynamic representation.The model supplies expression and pose parameters used to condition the scene representation.
- Contribution: The learned scene representation stores the dynamically changing surface of a human head in a controllable manner.The representation supports synthesis under changed head pose and facial expression.
2. Related Work
The related work spans morphable-model reconstruction, human avatar and portrait-video synthesis, and neural scene representation networks. This paper differs by implicitly representing face geometry and appearance while combining dynamic facial conditioning with volumetric rendering from monocular input.
- Face Reconstruction based on a Morphable Model: Morphable-model methods commonly reconstruct and animate faces, but often rely on coarse surface representations or explicit template meshes.Prior work uses corrective shapes, adaptive blendshape bases, or non-rigid mesh deformation to compensate for coarse geometry.
- Human Avatar Reconstruction: Human avatar reconstruction methods target photo-realistic reproduction from monocular input, including approaches for static heads, single-image geometry, and combined face-hair modeling.The paper positions its implicit formulation as representing the face and hair in a single formulation.
- Human Portrait Video Synthesis: Human portrait video synthesis includes classical morphable-model rendering, image warping, and hybrids combining rendered representations with learned image synthesis.Examples include correspondence maps, neural feature descriptors, and rendered facial features.
- Pipeline Context: The pipeline combines facial-expression tracking, volumetric rendering, and a dynamic radiance field conditioned on a learnable per-frame latent code.A background image supplies the color of the final sample point along each viewing ray.
- Neural Scene Representation Networks: Neural scene representation networks encode geometry and appearance as neural functions sampled at spatial points and rendered by ray marching.Related extensions address positional encoding, sinusoidal activations, sparse voxel fields, and in-the-wild training data.
3. Method
The method represents a facial avatar with a dynamic neural radiance field conditioned on tracked expressions, pose, and per-frame latent codes. It renders the implicit representation volumetrically from monocular portrait video while separating the static background.
- Dynamic Neural Radiance Fields: A neural scene representation network implicitly stores the head’s dynamic geometry and appearance for 4D facial-avatar reconstruction.The approach uses a dynamic neural radiance field during volumetric rendering.
- Dynamic Neural Radiance Fields: The MLP takes canonical position, viewing direction, facial expression, and a learnable latent code, then predicts color and density for volumetric rendering.The latent code compensates for errors in expression and pose estimation and improves image sharpness.
- Dynamics Conditioning: A low-dimensional morphable model supplies facial-expression coefficients and rigid pose transforms, enabling expression and head-pose control.The expression vector has 76 dimensions, while pose transforms camera-space points into canonical head space.
- Volumetric Rendering of Portrait Videos: Rendering casts rays through image pixels, transforms samples into canonical space, and integrates predicted densities and RGB values along each ray.The method uses coarse and fine radiance-field networks, with coarse densities guiding importance sampling for the fine network.
- Volumetric Rendering of Portrait Videos: A single background image separates the static background from the dynamic facial foreground during differentiable volumetric rendering.The final ray sample uses the corresponding background-image color, encouraging low foreground density for background pixels.
- Network Architecture and Training: Training optimizes coarse and fine networks with photometric reconstruction losses using sampled rays focused primarily within the morphable-model head bounding box.Each iteration samples 2048 rays, with 95% inside the head bounding box, and uses 64 coarse plus 64 fine samples per ray.
4. Results
The experiments evaluate monocularly reconstructed facial avatars for controllable pose and expression synthesis, facial reenactment, and quantitative comparison with existing methods. Results also examine the effects of background input, dynamic conditioning, and training-corpus size.
- Monocular Training Data: The evaluation uses short monocular RGB videos captured at 1920 × 1080 resolution and 50 frames per second, with the final 20 seconds held out for testing.
- Controllable Synthesis: The avatars synthesize 3D-consistent novel head poses and expression changes by manually editing pose and the ‘open mouth’ blendshape coefficient.
- Comparison to the State of the Art: The method produces photo-realistic, 3D-consistent reenactment results, including view-dependent effects such as reflections on glasses, while baselines struggle especially with new head rotations.
- Facial Reenactment: The method transfers facial expressions from a source person to a target actor while requiring model training only for the target actor.
- Ablation Studies: All metrics significantly benefit from a larger training corpus because the dynamic radiance field must generalize across expression vectors.
- Ablation Studies: All components improve result quality, including conditioning on facial dynamics and using a background image to disentangle the foreground avatar from the background.
5. Limitations
The approach has limitations in both facial control and modeled body extent: the morphable model omits explicit eye-motion controls, and the current system models only the human head.
- The volumetric representation reconstructs more effectively than state-of-the-art methods quantitatively and qualitatively, despite these limitations.
- The morphable model does not explicitly control eye blinks or eye movements, although eye blinks are implicitly correlated with other expression parameters.
- The current method reconstructs the human head but does not model upper-body dynamics; extending it requires consistent torso tracking.
6. Conclusion
The paper presents controllable 4D facial avatars learned from a single fixed-camera view and rendered under novel poses and expressions. The avatars support complex appearance and photo-realistic reenactment without a specialized multi-view capture rig.
- The method learns and renders controllable 4D facial avatars using dynamic neural radiance fields.
- Volumetric rendering captures arbitrary geometry and topology, including hair, eyewear, and hats, which morphable-model methods typically do not support.
- A single fixed camera and about 2 minutes of capture make the method suitable for capturing end-user avatars at home.
- The reconstructed avatars render photo-realistically under novel poses and expressions and outperform state-of-the-art facial reenactment methods quantitatively and qualitatively.
A. Network Architecture
The dynamic neural radiance field is an MLP conditioned on facial expression coefficients and learned latent codes. It maps spatial samples and viewing directions to density and color for volumetric rendering.
- The dynamic neural radiance field is represented as a multi-layer perceptron whose outputs support volumetric rendering.
- Expression coefficients from the face tracker's blendshape basis and learned latent codes condition the MLP to compensate for missing information.
- For each sample location and viewing direction, the MLP predicts density and color used in volumetric rendering.
- The network uses eight fully connected layers with 256 neurons each, followed by separate density and color prediction branches.