Source-linked AI summary
HeadNeRF: A Real-time NeRF-based Parametric Head Model
Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, Juyong Zhang
TL;DR
HeadNeRF addresses the challenge of representing human heads without explicitly modeling 3D geometry by introducing a NeRF-based parametric head model. It combines neural radiance fields with efficient 2D neural rendering and achieves high-fidelity real-time rendering with direct control over pose and semantic attributes.
Problem
Existing approaches face challenges in explicitly encoding or modeling 3D geometry for parametric human-head representation.
Method
HeadNeRF applies NeRF to human-head representation, predicts high-dimensional features for sampled 3D points, and uses 1x1-convolution-based 2D neural rendering with novel controls for pose, identity, expression, and appearance.
Results
HeadNeRF renders high-fidelity head images in real time and directly controls rendering pose, identity, expression, and appearance, outperforming related state-of-the-art models.
Takeaways & Limitations
HeadNeRF supports high-fidelity controllable head-image generation and enables applications based on direct semantic editing of rendered heads.
Takeaways & Limitations
The training data does not cover enough cases, making fitting difficult for images unlike the training distribution, including images with headgear.
Abstract
from arXiv · showhide
In this paper, we propose HeadNeRF, a novel NeRF-based parametric head model that integrates the neural radiance field to the parametric representation of the human head. It can render high fidelity head images in real-time on modern GPUs, and supports directly controlling the generated images' rendering pose and various semantic attributes. Different from existing related parametric models, we use the neural radiance fields as a novel 3D proxy instead of the traditional 3D textured mesh, which makes that HeadNeRF is able to generate high fidelity images. However, the computationally expensive rendering process of the original NeRF hinders the construction of the parametric NeRF model. To address this issue, we adopt the strategy of integrating 2D neural rendering to the rendering process of NeRF and design novel loss terms. As a result, the rendering speed of HeadNeRF can be significantly accelerated, and the rendering time of one frame is reduced from 5s to 25ms. The well designed loss terms also improve the rendering accuracy, and the fine-level details of the human head, such as the gaps between teeth, wrinkles, and beards, can be represented and synthesized by HeadNeRF. Extensive experimental results and several applications demonstrate its effectiveness. The trained parametric model is available at https://github.com/CrisHY1995/headnerf.
1. Introduction
HeadNeRF applies neural radiance fields to parametric human-head representation, combining high-fidelity, multi-view-consistent rendering with controllable pose and semantic attributes. A 2D neural-rendering strategy and specialized losses enable real-time inference and training from 2D images.
- Motivation and approach: 2D generative methods can produce photorealistic faces and semantic edits, but their outputs often lack cross-view consistency because they do not explicitly model 3D geometry.These methods directly render images without explicit 3D geometry.
- Motivation and approach: HeadNeRF applies a NeRF structure to human-head representation, using its implicit geometry for high-fidelity, multi-view-consistent images and pose editing.Unlike 2D generative methods, NeRF supports changing the rendering camera perspective and differentiable rendering.
- Efficiency and capabilities: HeadNeRF directly controls rendering pose, identity, expression, and appearance through a disentangled parametric representation.The model is designed for efficient control of multiple semantic attributes.
- Training and applications: A training strategy based on general 2D image datasets produces high-fidelity rendered images without requiring 3D scan data.The authors collect and process three large-scale human-head image datasets and design novel loss terms.
- Training and applications: Applications include novel-view synthesis from a single face image, semantic face-attribute editing, and facial reenactment.The authors report that these applications verify HeadNeRF's effectiveness.
2. Related Work
Prior work spans mesh-based parametric face models, 2D generative models, and NeRF-based systems. HeadNeRF combines a NeRF 3D proxy with 2D neural rendering to target controllable, detailed, and efficient head-image synthesis.
- Parametric face/head models: 3D morphable models encode identity, expression, and reflectance as low-dimensional vectors for reconstructing textured face meshes.Scene illumination is often modeled simultaneously with spherical harmonics.
- Generative models: 2D generative models can produce photorealistic face images but generally lack semantic, interpretable control and consistent geometry across views.Disentangled losses and embedded parametric face models are used to alleviate these limitations.
- NeRF-based models: NeRF-based methods improve 3D consistency through implicit scene geometry, but expensive radiance-field rendering makes high-resolution, high-fidelity synthesis difficult.GRAF and related methods introduce NeRF into generative models, while GIRAFFE combines NeRF with a 2D CNN renderer for greater speed.
3. Method
HeadNeRF represents human heads with an accelerated neural radiance field controlled by camera parameters and disentangled semantic codes. Its architecture combines volume rendering with 2D neural rendering, while training uses 3DMM initialization, geometric alignment, and multiple losses.
- Model representation: HeadNeRF uses an accelerated neural radiance field as a unified 3D proxy for controlling pose, identity, expression, and appearance.The model replaces the traditional textured mesh representation with neural radiance fields and supports real-time rendering on modern GPUs.
- Model representation: The renderer maps identity, expression, albedo, illumination, and camera parameters to an output image I through the parametric function R.P is the camera parameter; the four latent codes represent independent identity, expression, albedo, and illumination factors.
- Network architecture: The implicit function predicts density from 3D position, identity, and expression codes, while albedo and illumination influence the predicted feature vector.The model samples 3D points along rays and predicts a 256-dimensional feature vector rather than RGB directly.
- Network architecture: Volume-rendered features are converted into 256×256 images by a 2D neural module using 1×1 convolutions, leaky ReLU, and progressive upsampling.The 1×1 convolutions are used to alleviate possible multi-view inconsistent artifacts, and feature tensors are mapped to RGB before summation.
- Latent codes and canonical coordinate: Training initializes image-specific latent codes through 3DMM inverse rendering and aligns images’ underlying geometry to a similar center.The alignment uses a global rigid transformation from the 3DMM canonical coordinate system to camera coordinates.
- Loss function: The training objective combines photometric, perceptual, and disentangled losses to match head regions, improve details, and separate semantic controls.Disentanglement shares identity codes across images of a subject and expression codes across matching expressions, while regularizing learnable codes toward 3DMM initialization.
4. Experiments
Experiments evaluate HeadNeRF’s controllable rendering, disentangled attributes, training strategy, generalization, and comparisons with related methods. Results show efficient training and rendering alongside high-fidelity, multi-view-consistent outputs and applications including expression transfer.
- Implementation Details: More than 40fps rendering is achieved with 64 ray samples and without additional acceleration or optimization.The implementation removes hierarchical volume sampling to further speed up inference.
- Disentangled Control: HeadNeRF directly adjusts camera parameters, including rendering view, camera position, and FoV, while maintaining multi-view consistency.The experiments use fixed latent-code combinations to change the rendered view and camera settings.
- Disentangled Control: Linear interpolation of specified latent codes independently edits facial attributes while preserving the remaining semantic attributes.The procedure selects latent codes describing the same attribute, interpolates between them, and re-renders the result.
- Ablation Studies: HeadNeRF trains in about 3 days on one NVIDIA 3090 GPU and renders in real time, whereas HeadNeRF-vanilla trains about 7 days on three GPUs and takes about 5s per frame.The vanilla baseline produces blurrier results, while HeadNeRF achieves better rendered results with the 2D neural rendering module.
- Ablation Studies: The perceptual loss enhances fine-level details such as wrinkles and beards in generated results.The ablation compares HeadNeRF with a model trained without the perceptual loss.
- Ablation Studies: Adding FFHQ significantly improves generalization, enabling plausible single-image optimization followed by pose, identity, expression, and appearance editing.The resulting edits maintain excellent multi-view consistency, and the datasets are described as complementary for representation ability.
5. Limitation and Future Work
HeadNeRF remains limited by incomplete coverage of training data, affecting fitting for unusual head content and producing discontinuous illumination edits. The authors propose broader in-the-wild data and light-stage captures as future remedies.
- Data coverage: HeadNeRF struggles to fit images that differ substantially from its training data, such as images containing headgear.The authors attribute this to insufficient coverage and report that the model may return similar results instead.
- Data coverage: The training data contains only four illumination types, limiting coverage of illumination conditions.
- Observed failures: Figure 13 presents failure results for fitting and re-illumination.
- Future work: Illumination editing can produce shading changes unlike continuous movement of a light source.The authors suggest adding light-stage data to alleviate this problem.
6. Conclusion
HeadNeRF integrates neural radiance fields into a parametric human-head representation. It renders high-fidelity images in real time, enables direct pose and attribute editing, and is reported to outperform related state-of-the-art models.
- HeadNeRF integrates a neural radiance field into a parametric human-head representation.
- The model renders high-fidelity head images in real time on modern GPUs.
- HeadNeRF directly controls rendered-image pose and independently edits identity, expression, and appearance.
- Extensive experiments report that HeadNeRF outperforms state-of-the-art related models.