Source-linked AI summary
PointAvatar: Deformable Point-based Head Avatars from Videos
Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, Otmar Hilliges
TL;DR
Existing avatar methods struggle to combine realistic, animatable, relightable reconstruction with flexible topology and efficient deformation and rendering. PointAvatar learns a deformable point-based avatar from monocular video, disentangling albedo from normal-dependent shading. It achieves high-quality results across varied video sources and challenging structures while training more efficiently than competing implicit methods.
Problem
Existing mesh and neural implicit avatar methods have limitations in topology, deformation, rendering efficiency, and lighting disentanglement for casual-video reconstruction.
Method
PointAvatar jointly learns a canonical point cloud, continuous deformation driven by FLAME parameters, and a shading network that combines albedo with normal-dependent shading.
Results
PointAvatar achieves detailed geometry, state-of-the-art photo-realism, and superior results across commodity-camera and internet videos, including challenging hair and extreme poses.
Takeaways & Limitations
The representation combines flexible topology, efficient rendering, straightforward deformation, and lighting disentanglement for animatable head avatars.
Takeaways & Limitations
Its shading model does not further disentangle environment maps from surface reflectance, limiting relighting capability.
Abstract
from arXiv · showhide
The ability to create realistic, animatable and relightable head avatars from casual video sequences would open up wide ranging applications in communication and entertainment. Current methods either build on explicit 3D morphable meshes (3DMM) or exploit neural implicit representations. The former are limited by fixed topology, while the latter are non-trivial to deform and inefficient to render. Furthermore, existing approaches entangle lighting in the color estimation, thus they are limited in re-rendering the avatar in new environments. In contrast, we propose PointAvatar, a deformable point-based representation that disentangles the source color into intrinsic albedo and normal-dependent shading. We demonstrate that PointAvatar bridges the gap between existing mesh- and implicit representations, combining high-quality geometry and appearance with topological flexibility, ease of deformation and rendering efficiency. We show that our method is able to generate animatable 3D avatars using monocular videos from multiple sources including hand-held smartphones, laptop webcams and internet videos, achieving state-of-the-art quality in challenging cases where previous methods fail, e.g., thin hair strands, while being significantly more efficient in training than competing methods.
1. Introduction
PointAvatar uses a deformable point-based representation to combine flexible topology, efficient rendering, controllable animation, and disentangled appearance. It achieves strong results across varied monocular video sources and challenging head-avatar cases.
- Representation: PointAvatar represents canonical head geometry and appearance with an explicit point cloud plus a continuous deformation field.The deformation uses learned blendshapes and skinning weights driven by expression and pose parameters.
- Representation: Point-based avatars retain efficient rasterization and deformation while accommodating flexible topologies and thin structures.These properties address limitations of fixed-topology meshes and computationally costly implicit representations.
- Appearance: PointAvatar disentangles apparent color into pose-agnostic albedo and normal-dependent, pose-dependent shading.Normals are estimated through a smooth SDF and transformed analytically with the deformation Jacobian.
- Results: PointAvatar surpasses mesh and implicit approaches across videos from DSLR, smartphone, laptop, and internet sources.The method handles challenging cases including voluminous curly hair and novel poses with large deformation.
- Results: State-of-the-art photo-realism is achieved with considerably greater efficiency than existing implicit 3D avatar methods.The method combines high-quality geometry and appearance with efficient rendering and deformation.
2. Related Work
Prior head-avatar work relies mainly on meshes, neural implicit representations, or point-based rendering systems. PointAvatar instead learns deformable point-based head avatars directly from monocular videos without explicitly relying on a 3DMM template or 3D scans.
- Point clouds for neural rendering: Point-based neural rendering projects point features through rasterization or volumetric rendering to produce perceptually high-quality images efficiently.These methods motivate point representations for neural rendering but do not describe PointAvatar’s head-avatar formulation.
- PointAvatar: The method pipeline deforms canonical points with learned offsets, blendshapes, and skinning, then combines albedo with normal-dependent shading for rasterization.FLAME regularization supports controllable and generalizable animations.
- Point-based human avatars: PointAvatar learns deformation, geometry, and appearance jointly from monocular videos, unlike prior point-based human-avatar methods requiring 3D scans.It also avoids explicit reliance on a 3DMM template.
3. Method
PointAvatar jointly learns a canonical point-based avatar, deformation, normals, albedo, shading, and rendering objectives from monocular RGB videos. Its coarse-to-fine point optimization, differentiable deformation, normal estimation, and albedo–shading factorization support efficient training and controllable rendering.
- Overall architecture: The model jointly learns canonical point geometry and appearance, a deformation network driven by FLAME parameters, and a normal-conditioned shading network.Training compares rendered shaded points in deformed space with target images.
- Optimization: Periodic point upsampling and radius reduction accelerate convergence by fitting coarse shape first and fine details later.Invisible points are pruned after each epoch to further accelerate training.
- Canonical representation: Canonical normals are estimated as spatial SDF gradients rather than directly from neighboring points, improving robustness when sampling is sparse, noisy, or irregular.An MLP represents the SDF and is fitted to canonical point locations with a data term and Eikonal regularization.
- Point color: An MLP maps canonical point locations to albedo, whose shared coordinate-based representation imposes a local smoothness prior on colors.The same shared MLP is used in practice to compute canonical normals and albedo.
- Point deformation: Two-stage deformation first maps canonical points to an intermediate FLAME space, then applies learned expression and pose blendshapes with skinning weights.The authors report that this design avoids bad local minima and yields more accurate geometries.
- Point color and rendering: Point colors factor into pose-agnostic albedo and pose-dependent shading inferred from deformed normals, enabling unsupervised disentanglement and rudimentary relighting.Differentiable rasterization splats points into images, allowing whole-image perceptual losses such as VGG feature loss.
4. Experiments
Experiments compare PointAvatar with mesh- and implicit-based baselines across diverse video sources and challenging geometries. Ablations evaluate lighting disentanglement, normal estimation, training efficiency, image losses, and canonical-space design.
- Comparison with SOTA Methods: Evaluations include DSLR, smartphone, internet, and laptop-webcam videos, with baselines spanning dynamic radiance fields, morphable meshes, and implicit surfaces.The datasets include subjects from prior benchmarks plus casual capture settings with limited pose variation, exposure changes, or low resolution.
- Comparison with SOTA Methods: PointAvatar achieves the best L1, LPIPS, SSIM, and PSNR metrics on both DSLR sequences and casual smartphone videos.The comparison excludes clothing for NHA because it models only the head region.
- Comparison with SOTA Methods: PointAvatar handles eyeglasses, thin frames, hair buns, volumetric curly hair, sharp skin details, and extreme head poses more effectively than the compared methods.Mesh topology prevents modeling eyeglass gaps, while implicit and volumetric baselines show failures on thin structures, silhouettes, geometry, or uncommon poses.
- Lighting Disentanglement: Disentangling albedo from normal-dependent shading improves facial geometry, while SDF normals are less noisy and Jacobian-based transformations capture blendshape-related changes.The ablations report smoother normals in detailed regions and correct nasal-line normals during smiling.
- Lighting Disentanglement: PointAvatar disentangles albedo and normal-dependent shading from fixed-lighting monocular video and supports faithful relighting by changing the shading component.The method can alter lighting directions after training, although supplementary material describes some alternative relighting as less faithful.
- Efficiency and Ablations: PointAvatar trains and renders substantially faster than implicit head-avatar methods, aided by point pruning and efficient full-image rendering for image- and patch-based losses.The VGG feature loss improves photo-realism relative to a pixel-only L1 objective.
- Efficiency and Ablations: A freely learned canonical space improves canonical geometry and generalization to novel poses compared with directly learning in predefined FLAME space.Without the canonical offset, the model overfits and learns incorrect canonical geometries.
5. Discussion
PointAvatar combines flexible point-based geometry, efficient rendering, and straightforward deformation for challenging head-avatar cases. Its lighting disentanglement supports detailed geometry and relighting, while several limitations remain for physically faithful rendering and fine structures.
- Discussion: PointAvatar uses deformable point-based avatars to handle eyeglasses, voluminous hair, skin details, and extreme head poses.The representation is trained from monocular video with fixed lighting while producing detailed facial and hair geometry.
- Discussion: The method disentangles lighting effects from intrinsic albedo despite training on a monocular video captured under fixed lighting.This enables faithful separation of detailed geometry and appearance components within the demonstrated scope.
- Limitations: The shading MLP does not separate lighting into environment maps and surface reflectance, limiting relighting capability.Future work proposes a more constrained physically based rendering model.
- Limitations: Uniform point radii can require unnecessarily many points for accurately modeling eyes and hair strands.Varying point sizes is proposed as a way to improve detail while reducing points and rendering cost.
- Limitations: PointAvatar cannot faithfully model eyeglass-lens reflections without explicitly modeling transparency and reflection.This is identified as a future improvement direction.
1. Additional Results
Additional evaluations show that PointAvatar matches state-of-the-art surface geometry reconstruction while retaining advantages for volumetric structures and mask-free learning.
- Results on the MakeHuman Dataset: PointAvatar performs on par with state-of-the-art methods for surface geometry reconstruction on the MakeHuman synthetic dataset.The evaluation reports normal consistency for reconstructed surface geometry, comparing against NHA and IMavatar.
- Results on the MakeHuman Dataset: PointAvatar reconstructs comparable head geometry to mesh-based NHA and implicit IMavatar representations.
- Self-supervised Foreground-Background Disentanglement: PointAvatar can roughly disentangle foreground and background contents without mask supervision or known backgrounds.The method is contrasted with NerFace, which cannot be learned without known backgrounds.
2. Implementation Details
The implementation uses separate canonical, deformation, and shading MLPs, staged point refinement, FLAME-based tracking, and post-processing for sparsity artifacts and relighting.
- Network Architecture: Canonical, deformation, and shading MLPs use weight normalization and nonlinear activations, with Softplus for the first two and ReLU for shading.
- Tracking Stabilization: A static bone reduces shoulder jitter caused by unstable upper-body FLAME tracking during preprocessing.The deformation MLP assigns large linear blend skinning weights to the static bone because the upper body remains mostly static in training videos.
- Point Upsampling and Pruning: Every five epochs, the method doubles point count and reduces rasterization radii; invisible points are pruned after each epoch.The practical radius reduction factor is 0.75, and points are pruned when their compositing weight falls below 0.5.
- Hole Filling: Sparse deformed point clouds can create white dots in stretched or infrequently observed regions, requiring erosion-dilation or per-subject neural-network post-processing.Erosion and dilation slightly blur images, while the neural-network approach avoids blur but requires cumbersome per-subject training.
- Relighting: Albedo is conditioned on canonical locations while shading is conditioned on deformed-space normals, enabling rudimentary relighting.Relighting can manipulate normal directions or replace learned shading with a default shading model using learned albedo and normals.
3. Proof of Eq. 6 (deformed normal formula)
The proof derives the deformed normal formula by imposing local planar consistency between neighboring deformed points and then normalizing the resulting vector.
- Local Surface Constraint: The proof assumes points locally form flat surfaces and defines the deformed normal through a constraint involving neighboring deformed points.
- Constraint Derivation: Linearizing neighboring deformed points and using the canonical normal constraint rewrites the local consistency condition into a requirement on the deformed normal.
- Final Formula: A normalizing scalar l enforces unit length, yielding the deformed-normal formulation in Eq. 6.
4. Ethics
The study uses consented recordings from five human subjects and states an ethical boundary against deceptive or reputation-harming synthetic media.
- Data Collection: Five human subjects were captured with smartphones or laptop cameras, and written consent was obtained for using their images.The authors plan to make the data public for research where subjects permit publication.
- Responsible Use: The authors reject using the method to generate fake images or videos intended to spread misinformation or tarnish a person’s reputation.