Source-linked AI summary
RigNeRF: Fully Controllable Neural 3D Portraits
ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, Zhixin Shu
TL;DR
Standard NeRFs do not support editing objects within a scene, motivating controllable portrait rendering. RigNeRF uses a 3DMM-guided deformation field to control head pose and expressions from a short portrait video, while preserving scene detail and deformation fidelity. Its scope is limited by subject-specific training, required motion variation, camera-registration quality, and possible misuse.
Problem
Standard NeRFs support photo-realistic novel views but do not provide control over objects such as human heads within a scene.
Method
RigNeRF combines a neural radiance field with a 3DMM-guided deformation field that models rigid head-pose and non-rigid expression changes.
Results
RigNeRF provides explicit control of camera viewpoint, head pose, and facial expression while maintaining high fidelity to the driving model and captured scene.
Takeaways & Limitations
A short portrait video can support controllable 3D portrait reanimation, novel-view synthesis, and 3DMM-based portrait control.
Takeaways & Limitations
RigNeRF is subject-specific, requires 40–70 seconds of training footage with sufficient variation, depends on camera registration, and may be misused.
Abstract
from arXiv · showhide
Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls. The project page can be found here: http://shahrukhathar.github.io/2022/06/06/RigNeRF.html
1. Introduction
RigNeRF addresses the difficulty of producing photo-realistic portraits with simultaneous control over viewpoint, head pose, and facial expression. It combines neural rendering with 3DMM-guided deformation to enable controllable portrait synthesis from short video.
- Photo-realistic portrait editing seeks control over 3D viewpoint, lighting, head pose, and facial expression, but comprehensive control in natural scenes remains difficult.
- Existing neural rendering methods produce high-quality novel views but cannot simultaneously control objects and facial attributes within natural scenes.
- Latent-space deformation methods support dynamic portraits but do not provide explicit control, while naive 3DMM-conditioned deformation can cause rigidity and expression artefacts.
- Trained on a short mobile-device video, RigNeRF renders controlled camera views, head poses, and expressions while preserving detailed scene and head appearance.
2. Related works
Related work spans neural rendering, dynamic scene representations, controllable face generation, and hybrid classical-neural representations. RigNeRF combines these directions to provide explicit 3D portrait control while modeling the full scene.
- Neural Scene Representations and Novel View Synthesis: NeRF represents scenes volumetrically and renders novel views, but standard NeRFs are designed for static scenes without object-level control.
- Dynamic Neural Scene Representations: Dynamic NeRF extensions use deformation fields and canonical spaces, but latent deformation codes lack intuitive physical or semantic control.
- Dynamic Neural Scene Representations: RigNeRF extends canonical-space dynamic modeling with parameterized control of head pose and facial expression for full 3D portrait scenes.
- Controllable Face Generation: Image-based face manipulation can generate high-quality edits but generally lacks explicit 3D representation, limiting view synthesis and large pose or expression changes.
- Hybrid Representations: Hybrid representations combine neural volumetric methods with classical representations for reconstruction or deformation, while some related methods do not model the full 3D scene.
3. RigNeRF
RigNeRF uses a 3DMM-guided deformation field to control head pose and facial expressions while rendering a full portrait scene. It combines model-based deformations with learned residuals to preserve rigidity and represent details such as hair and glasses.
- A 3DMM-guided deformation field: RigNeRF defines its deformation as the sum of a 3DMM deformation field and a residual predicted by deformation MLP D.The residual captures deformations that head-pose and expression changes alone cannot explain.
- Deformable Neural Radiance Fields: Dynamic NeRFs use deformation fields to map scene points into canonical space, but learned deformation codes provide arbitrary rather than explicit controls.The deformation is represented by an MLP and conditioned on per-frame embeddings in the dynamic-scene formulation.
- RigNeRF: RigNeRF maps each frame’s rays into a canonical space defined by zero head pose and neutral facial expression.Head-pose and expression parameters are extracted per frame, and colors are sampled after deformation into this canonical space.
- A 3DMM-guided deformation field: The learned residual is necessary because 3DMMs are inaccurate for some head points and do not model hair, glasses, or the rest of 3D space.Directly predicting canonical-space deformation can also create artefacts by failing to preserve head rigidity and facial expressions.
- A 3DMM-guided deformation field: The 3DMM field transfers mesh-point deformation to arbitrary space points using the nearest mesh point and distance-based weighting.For mesh points, deformation is the difference between canonical and currently articulated positions.
- A 3DMM-guided deformation field: Figure 4 shows residual deformation around glasses, while the combined deformation remains concentrated on the head and depth stays consistent across head poses.The color MLP receives deformed points together with pose, expression, viewing direction, and appearance information before volume rendering.
4. Results
RigNeRF is evaluated on smartphone-captured portrait videos for held-out reconstruction, controllable reanimation, novel-view synthesis, and 3DMM-based portrait control. Across these tests, it preserves portrait detail while reproducing specified head poses and facial expressions more faithfully than the compared methods.
- Evaluation setup: The model is trained and evaluated on portrait videos captured with consumer smartphones, using held-out images and driving sequences.The capture protocol varies facial expressions, speech, camera viewpoints, and head rotations.
- Evaluation on Test Data: RigNeRF outperforms HyperNeRF, NerFACE, and FOMM on held-out test images across most metrics and subjects.The evaluation uses camera-view, pose, and expression parameters from held-out images; deformation codes are optimized for RigNeRF and HyperNeRF.
- Evaluation on Test Data: RigNeRF models head pose, facial expressions, and full-scene appearance with high fidelity, whereas HyperNeRF lacks explicit controls, NerFACE produces face artefacts, and FOMM cannot model novel views.The 3DMM-guided deformation module supports sharp reconstructions of the portrait scene, including head details.
- Reanimation with pose and expression control: During image-driven reanimation, RigNeRF reproduces driving head poses and expressions while preserving the reconstructed 3D scene.HyperNeRF cannot change pose or expression, while NerFACE generates significant face artefacts, especially under head-pose changes.
- Reanimation with pose and expression control: RigNeRF supports novel-view synthesis with changing pose and expression, including dramatic view changes, without compromising the background scene.The results also show user-specified 3DMM parameters being faithfully rendered as portrait appearance, pose, and expression.
- Comparison with HyperNeRF+E/P: Compared with HyperNeRF+E/P, RigNeRF produces more faithful reanimations because the baseline generates artefacts when modelling pose and expression deformations.Quantitative comparison reports better face reconstruction and lower perceptual distance to ground truth for RigNeRF.
5. Limitations and Conclusion
RigNeRF provides controllable human-portrait rendering through 3DMM-guided deformation, while requiring subject-specific training and a 40–70-second sequence with varied expressions and head poses.
- Limitations: RigNeRF trains an individual model for each scene and currently requires 40–70 seconds of training footage with sufficient expression and head-pose variation.Camera-registration quality also affects the quality of the results.
- Limitations: The authors identify potential negative societal impact if photorealistic facial reanimation is misused.
- Conclusion: RigNeRF controls head pose, facial expression, and viewing direction using a 3DMM-guided deformation field.The field models rigid head-pose changes and non-rigid facial-expression changes from a short portrait video.
- Conclusion: The method supports image-based face reanimation, portrait novel-view synthesis, and 3DMM-based control of 3D portraits.
6. Ablations
The ablation compares three conditioning designs and finds that deformation-based input plus 3DMM-conditioned appearance improves generalization and perceptual metrics over direct parameter conditioning.
- Ablation configurations: Config A directly conditions the deformation network on 3DMM expression and pose parameters without 3DMM-conditioned appearance.
- Ablation configurations: Config B replaces direct expression and pose inputs with 3DMM deformation, while still omitting 3DMM-conditioned appearance.
- Ablation configurations: Config C uses 3DMM deformation as input and 3DMM-conditioned appearance, forming the paper’s final model.
- Ablation results: Removing direct 3DMM parameters from Config A to Config B improves results, especially for LPIPS and FaceMSE.
- Ablation results: Conditioning appearance on 3DMM parameters from Config B to Config C further improves LPIPS and FaceMSE and generalizes better to a novel head pose.Config A shows poor generalization in both its final render and depth, whereas Config C generalizes well.
7. FOMM Qualitative Results
The qualitative comparison reports that RigNeRF preserves high-fidelity reanimation under large head-pose and facial-expression deformations, unlike FOMM.
- Qualitative comparison: RigNeRF models large head-pose and facial-expression deformations with high fidelity during reanimation.
- Qualitative comparison: FOMM is unable to model the large deformations caused by head pose and facial expressions in the comparison.
8. Generalization to Novel pose and expression
The evaluation examines extrapolation beyond the training data for head pose, jaw pose, and facial expressions using qualitative results and perceptual or L2 distance measures.
- Pose generalization: RigNeRF is evaluated on pitch extrapolation outside the training data and on head-pose yaw and pitch extrapolation.Yaw and pitch are measured by perceptual distance to the closest training frame.
- Expression generalization: The evaluation also measures jaw-pose and expression extrapolation.Expression generalization is assessed alongside qualitative extrapolation results.
9. Experimental Details
The experimental details identify RigNeRF configurations and provide training-setup information for the experiments.
- Training details include positional-encoding frequencies, coarse-to-fine regularization, hardware, sampling, and batch size.The models were trained on 4 V100 GPUs using 128 samples per ray and batches of 1550 rays.
- The experiments compare RigNeRF-Config A and RigNeRF-Config C.
- The qualitative comparison between Config A and Config C includes source images and depth images.
- Additional training information includes training epochs and the dimensionality of appearance and deformation codes.
10. Societal Impact
The societal-impact materials note that RigNeRF can be misused for deepfakes, while the supplied visual materials document comparisons and extrapolation experiments.
- Societal impact: RigNeRF’s face-reanimation capability creates a risk of misuse by bad actors to generate deepfakes.
- Societal impact: The authors identify discriminative classifiers and watermarking training images as possible ways to improve detection of synthetic media.
- Visual comparisons: The supplied figures also identify alternative RigNeRF configurations for qualitative comparison.
- Visual comparisons: Figure 10 compares RigNeRF and FOMM when transferring facial expression and head pose from source images.
- Visual comparisons: Figure 11 examines extrapolation by comparing sampled pitch values with the closest training pitches.