Source-linked AI summary
Instant Volumetric Head Avatars
Wojciech Zielonka, Timo Bolkart, Justus Thies
TL;DR
INSTA addresses the need for rapidly reconstructed, animatable digital avatars that reflect a subject’s current appearance from monocular RGB video. It uses a geometry-guided dynamic neural radiance field embedded around a 3DMM, producing avatars in under 10 minutes with interactive rendering and improved pose extrapolation. The method’s quality remains constrained by tracking accuracy, hair and teeth modeling, and rendering speed for high-quality AR/VR video conferencing.
Problem
Existing avatar-generation methods can require hours or days, while telepresence needs current-looking, metrically scaled avatars that can be driven interactively.
Method
INSTA optimizes a dynamic neural radiance field using neural graphics primitives embedded around a 3DMM, with expression and view conditioning.
Results
INSTA creates animatable avatars from monocular video in less than 10 minutes and demonstrates improved pose extrapolation through 3DMM-driven geometry regularization.
Takeaways & Limitations
INSTA supports adaptable online avatars that reflect a subject’s current appearance rather than a prerecorded appearance.
Takeaways & Limitations
Quality is limited by face-tracking misalignment, missing dynamically changing hair and teeth geometry, and rendering speed for high-quality AR/VR video conferencing.
Abstract
from arXiv · showhide
We present Instant Volumetric Head Avatars (INSTA), a novel approach for reconstructing photo-realistic digital avatars instantaneously. INSTA models a dynamic neural radiance field based on neural graphics primitives embedded around a parametric face model. Our pipeline is trained on a single monocular RGB portrait video that observes the subject under different expressions and views. While state-of-the-art methods take up to several days to train an avatar, our method can reconstruct a digital avatar in less than 10 minutes on modern GPU hardware, which is orders of magnitude faster than previous solutions. In addition, it allows for the interactive rendering of novel poses and expressions. By leveraging the geometry prior of the underlying parametric face model, we demonstrate that INSTA extrapolates to unseen poses. In quantitative and qualitative studies on various subjects, INSTA outperforms state-of-the-art methods regarding rendering quality and training time.
1. Introduction
INSTA targets quickly reconstructed, metrically scaled avatars for immersive telepresence, using a dynamic neural radiance field guided by a parametric face model. Its contributions emphasize minutes-scale reconstruction and improved pose extrapolation.
- Motivation: INSTA reconstructs a subject’s avatar in about 10 minutes and supports interactive driving for AR/VR telepresence.The pipeline uses commodity hardware for training and capture.
- Method: The method uses a canonical space to construct the dynamic neural radiance field and maps deformed points into it using a FLAME-based deformation field.A bounding volume hierarchy establishes the deformation field around the tracked face surface.
- Contributions: A surface-embedded dynamic neural radiance field based on neural graphics primitives reconstructs metrical avatars in minutes rather than hours or days.
- Contributions: 3DMM-driven geometry regularization of the dynamic density field improves pose extrapolation for AR/VR applications.
2. Related Work
INSTA builds on neural rendering and deformable radiance-field research while addressing long training times and limited view extrapolation. It combines multi-resolution hashing with a 3DMM geometry prior for RGB-only avatar reconstruction.
- Static Neural Radiance Fields: Existing static neural radiance-field methods commonly require 1–5 days of training, motivating accelerated representations.
- Static Neural Radiance Fields: Multi-resolution hashing replaces conventional coordinate-based processing with tiny MLPs and multilevel grid features for faster neural graphics computation.
- Geometry Priors: Unlike methods using extra depth maps, INSTA uses the 3DMM geometry prior to guide depth estimation from RGB images and improve view extrapolation.
- Deformable Neural Radiance Fields: Deformable neural radiance fields separate observations into deformed and canonical spaces and learn mappings between them.
- INSTA: INSTA applies differentiable volumetric optimization by canonicalizing sampled points before multi-resolution hashing and neural-network evaluation.
- Deformable Neural Radiance Fields: Prior facial approaches condition radiance-field networks on expression codes, whereas INSTA uses dynamic radiance fields to improve rendering in the mouth region absent from the face-model motion prior.
3. Instant Deformable Neural Radiance Field
INSTA represents an animatable avatar with a geometry-guided deformable neural radiance field. It canonicalizes ray samples through mesh-based deformation, conditions predictions on expressions and view direction, and optimizes color and geometry objectives.
- Overview: The method embeds a deformable neural radiance field in a multi-resolution hashing grid for instant learning and interactive rendering.
- Overview: Given monocular video, camera parameters, tracked FLAME meshes, expressions, and poses, INSTA constructs a controllable head avatar in canonical space.
- Canonicalization: Ray samples are canonicalized from deformed space before querying the neural radiance field, enabling rendering under specific facial expressions.
- Radiance Field: Expression codes condition samples in the dynamically changing mouth region, while spherical-harmonic view encoding conditions the avatar globally without per-frame learnable codes.
- Canonicalization: The deformation gradient maps each sampled point between corresponding deformed and canonical mesh triangles, using their Frenet frames.
- Canonicalization: An isotropic scaling factor accounts for triangle-area changes, while exponentially weighted averaging of adjacent-face transformations avoids discontinuities.
- Canonicalization: A GPU bounding volume hierarchy accelerates nearest-triangle searches for sampled ray points and establishes the mesh-based mapping function.
- Training Objectives: Training combines a color reproduction objective with a face-restricted 3DMM depth prior, using higher color-loss weight for the mouth region.The total loss weights geometry with λgeom = 1.25 and uses λcolor = 40 for the mouth versus 1 elsewhere.
4. Dataset
The evaluation uses monocular RGB videos from twelve actors, combining in-house recordings with YouTube sequences. The data are cropped, temporally subsampled, resized, and foreground-segmented before training.
- Dataset: The dataset contains twelve actors recorded in-house or sourced from YouTube for monocular-video avatar generation.
- Dataset: In-house captures provide approximately 2–3 minutes of monocular RGB Full HD video per actor.
- Preprocessing: The recordings are cropped, subsampled to 25 fps, resized to 5122 resolution, and processed with background–foreground segmentation.
- Tracking: Face tracking uses MICA’s analysis-by-synthesis tracker, extended with eyelid and iris blendshapes and optimized FLAME shape parameters.
5. Results
INSTA produces high-quality facial avatars and robust novel-view synthesis while training substantially faster than state-of-the-art methods. Ablations attribute improvements to local expression conditioning, mesh-based deformation, and the 3DMM geometric prior.
- Image Quality Evaluation: INSTA is on par with NeRFace on pixel-wise errors while achieving low perceptual error and significantly faster training and evaluation.Table 1 reports averages over 19 videos and denotes average single-frame rendering time in the rightmost column.
- Image Quality Evaluation: INSTA achieves the best image quality while being significantly faster to train than the compared methods.
- Image Quality Evaluation: INSTA robustly generates photo-realistic images under novel poses, with especially high visual quality in the skin and mouth regions.NeRFace produces blurry eyes and teeth, IMAvatar shows silhouette artifacts at grazing angles, and NHA has degenerated ear geometry.
- Ablation Studies: Local expression conditioning and the mesh-based deformation field help avoid overfitting to short training sequences.The ablation compares the full model with global conditioning and with a NeRFace-like variant lacking the deformation field.
- Ablation Studies: The 3DMM geometric prior regularizes depth estimation to support extrapolation to an unseen extreme profile view.During training, estimated ray termination is compared with depth from rendered per-frame 3DMM reconstructions.
- Ablation Studies: Good geometry and appearance are obtained after 30 seconds, while approximately 10 minutes improves high-frequency details such as teeth.
- Ablation Studies: Conditioning the mouth region on FLAME expression coefficients and increasing its color-term weight improves mouth-interior synthesis and visual quality.The approach compensates for the absence of explicit teeth geometry in publicly available 3DMMs.
6. Discussion
INSTA improves quality and speed over state-of-the-art RGB-video avatar methods, but dynamically changing hair, teeth geometry, and high-resolution rendering remain challenges.
- INSTA does not capture dynamically changing hair, whose detail remains below the quality of the face interior.
- Because the used 3DMM does not model teeth geometry, better mouth-region approximation could improve teeth quality during viewpoint extrapolation.
- Although rendering reaches real-time frame rates at 512^2 resolution, higher-resolution AR/VR video conferencing still requires faster rendering.
7. Limitations
INSTA’s quality is constrained by face-tracking accuracy and limited mouth geometry. Misalignment and expression extrapolation can produce visible artifacts.
- Face-tracking misalignment propagates into the final avatar, causing outline artifacts around the chin and hair.The failure cases attribute these artifacts to geometry misalignment of the tracker.
- Extreme expressions can cause artifacts in the mouth region, where the method lacks detailed geometry.
- Expression extrapolation can lead to artifacts beyond the observed expression range.
8. Conclusion
INSTA reconstructs geometry-guided digital avatars from monocular video in under 10 minutes. The authors position adaptable online avatars as a step toward immersive telepresence.
- INSTA optimizes a subject’s dynamic neural radiance field in less than 10 minutes from monocular RGB video.The representation uses neural graphics primitives embedded around a 3DMM.
- INSTA creates avatars intended to reflect a person’s current appearance rather than a prerecorded appearance.
- The authors describe adaptable online avatars as a stepping stone toward immersive telepresence applications.
– Supplemental Document –
The supplemental document presents additional avatar results and discusses applications beyond volumetric video conferencing. It also documents acceleration structures, expression transfer, and further qualitative and quantitative evaluations.
- Twelve in-house and YouTube sequences produce volumetric avatars that INSTA optimizes in less than 10 minutes.The figure reports photorealistic videos for both recording sources, including in-the-wild examples.
- The supplemental document covers acceleration structures for ray sampling with respect to neural graphics primitives.
- Additional sections demonstrate facial expression transfer and report predicted-normal and error-map results.
A. Implementation Details
The implementation uses occupancy-grid acceleration, simplified FLAME-based conditioning regions, and expression transfer through relative 3DMM expression deltas. It also demonstrates expression transfer across subjects.
- Accelerated Ray Marching: A separate 128^3 occupancy grid stores occupancy bits to skip empty-space samples during ray marching.The acceleration structure is constructed in deformed space, while neural graphics primitives are learned in canonical space.
- Accelerated Ray Marching: The acceleration structure converges to a Boolean union across all expressions in the training dataset.Voxels near the canonical-mesh isosurface can optionally be fixed as occupied during updates.
- FLAME masks: Expression conditioning uses the mouth region, while simplified eyeballs reduce unnecessary nearest-neighbor computation.
- Expression transfer: Relative source expressions are transferred to a target by computing deltas from manually selected neutral faces and adding them to the target neutral expression.The procedure uses ∆i = Si − Sneutral and Ti = Tneutral + ∆i.
- Expression transfer: INSTA supports expression transfer from a source subject to a different person’s facial avatar.The application is illustrated in Figure 12.
C. Additional Results
INSTA achieves competitive avatar quality while substantially reducing training time, though its geometric advantage is strongest where the face-model prior applies. The method also raises broader-impact concerns because accessible avatar synthesis can be misused.
- Geometry and rendering quality: INSTA matches IMAvatar's geometry quality in the face region while requiring only a fraction of its optimization time.In hair regions without a geometric prior, INSTA's geometry is similar to NeRFace.
- Evaluation: INSTA's photometric evaluation uses RGB-based ℓ1 error for a single frame and perceptual error over entire test sequences.These evaluations are presented in Figures 13 and 14.
- Training time evaluation: The training-time table measures avatar optimization only, excluding dataset generation, and uses 512^2 resolution except for IMAvatar at 256^2.This resolution difference is part of the table's stated evaluation setup.
- Training time evaluation: 10 min on a local RTX 3090 reconstructs a volumetric avatar with high-frequency details, making INSTA orders of magnitude faster than baselines.Baseline timings use original configurations on a compute cluster, including three A100 GPUs for NHA.
- Broader impact: Because INSTA needs no sophisticated capture setup, it can use webcam, smartphone or downloaded videos, but the same accessibility creates DeepFake misuse risks.The authors mention misinformation, cyber mobbing and identity theft, and propose open research for forgery-detection datasets.
- Geometry and rendering quality: The geometry comparison shows noisy NeRF normals, ear deformations and missing details in NHA, while INSTA reduces noise using an additional geometric prior.Hair remains problematic for INSTA because that region lacks the same geometric prior.