Source-linked AI summary
I M Avatar: Implicit Morphable Head Avatars from Videos
Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, Otmar Hilliges
TL;DR
Existing face models offer either fine-grained control or detailed appearance and geometry, while neural deformation formulations struggle to generalize beyond training expressions and poses. IMavatar learns implicit morphable head avatars from monocular videos using pose-independent blendshape and skinning fields with differentiable canonical-surface rendering. The method reports more accurate geometry and broader generalization to unseen poses and expressions than state-of-the-art methods.
Problem
Existing 3DMMs provide expression control but miss geometric and appearance detail, while neural volumetric avatars are difficult to animate and generalize poorly to unseen expressions.
Method
IMavatar uses three implicit fields for canonical geometry, texture, and pose- and expression-related deformations represented by learned blendshapes, skinning weights, and pose correctives.
Results
IMavatar produces more accurate geometry and generalizes better to unseen poses and expressions in synthetic and real-video evaluations than state-of-the-art methods.
Takeaways & Limitations
The model provides fine-grained, interpretable expression and pose control while retaining detailed facial geometry, appearance, hairstyles, and extrapolation beyond the training distribution.
Takeaways & Limitations
The method cannot model fine hair occlusions, trains slowly at approximately 2 GPU days, and depends on accurate face tracking because noisy 3DMM parameters degrade performance.
Abstract
from arXiv · showhide
Traditional 3D morphable face models (3DMMs) provide fine-grained control over expression but cannot easily capture geometric and appearance details. Neural volumetric representations approach photorealism but are hard to animate and do not generalize well to unseen expressions. To tackle this problem, we propose IMavatar (Implicit Morphable avatar), a novel method for learning implicit head avatars from monocular videos. Inspired by the fine-grained control mechanisms afforded by conventional 3DMMs, we represent the expression- and pose- related deformations via learned blendshapes and skinning fields. These attributes are pose-independent and can be used to morph the canonical geometry and texture fields given novel expression and pose parameters. We employ ray marching and iterative root-finding to locate the canonical surface intersection for each pixel. A key contribution is our novel analytical gradient formulation that enables end-to-end training of IMavatars from videos. We show quantitatively and qualitatively that our method improves geometry and covers a more complete expression space compared to state-of-the-art methods.
1. Introduction
IMavatar combines 3DMM-like expression control with detailed implicit geometry and appearance for personal avatars learned from monocular videos. Its pose- and expression-independent deformation formulation, differentiable rendering, and evaluation results target better generalization to unseen poses and expressions.
- Mesh-based morphable models trade resolution for practicality and cannot represent topological changes from hair, glasses, and accessories.
- Neural-field avatars produce high-quality images near the training distribution, but under-constrained deformation formulations limit generalization and require many training poses.
- IMavatar represents canonical geometry, texture, and pose- and expression-related deformations with three continuous implicit MLP fields.
- Learned expression blendshapes, skinning weights, and pose correctives warp canonical points under novel expression and pose conditions, providing fine-grained interpretable control.
- Ray marching and root-finding recover canonical surface correspondences for pixels, while an analytic implicit-differentiation gradient enables end-to-end training from videos.
- Evaluations on synthetic and real videos show more accurate geometry, better generalization to unseen poses and expressions, and more accurate reconstruction than state-of-the-art methods.The synthetic evaluation uses 10 subjects; real-video results report better pose and expression reconstruction and extrapolation.
2. Related work
Prior work spans statistical face models, personalized rigs, neural implicit surfaces, volumetric representations, and implicit deformation fields. IMavatar relates these strands through continuous morphable fields and canonical-surface correspondence search.
- 3D Face Models and Avatar Reconstruction: 3DMM variants provide statistical priors and overall facial estimates but typically lack detailed geometry, full-head coverage, and faithful modeling of eyes, teeth, hair, and accessories.
- 3D Face Models and Avatar Reconstruction: Personalized-rig methods recover blendshape controls and may add corrective layers, but the described approach introduces a new method for recovering detailed controllable representations.
- Neural Face Models: Neural implicit face models improve full-head and geometric detail, yet some estimated shapes cannot be animated.
- Neural Face Models: Neural volumetric representations encode thin structures and complex surface-light interactions, while some methods require complex multi-view video systems for training.
- Implicit deformation fields: IMavatar extends blendshape and skinning representations into continuous MLP fields that morph canonical points, using ray marching and correspondence search to find canonical surface locations.
- Implicit deformation fields: Implicit deformation methods condition fields on latent codes or use backward deformation fields, but backward formulations can require substantial correspondence modeling.
3. Method
IMavatar models personalized head avatars with continuous implicit geometry, texture, and deformation fields, then renders and trains them through canonical surface correspondences and analytical gradients.
- 3.2. IMavatar: The deformation network extends FLAME’s blendshape bases and skinning weights from mesh vertices to continuous canonical-space fields.It predicts expression offsets, pose correctives, and skinning weights for each canonical point.
- 3.2. IMavatar: IMavatar uses three neural implicit fields for canonical geometry, pose- and expression-related deformations, and texture.The fields are represented by multilayer perceptrons and combine morphable control with detailed implicit representations.
- 3.2. IMavatar: Deformed locations are obtained by applying pose and expression offsets to canonical points before linear blend skinning.This forward mapping uses the FLAME joint regressor and learned continuous deformation fields.
- 3.3. Differentiable Rendering: Non-rigid ray marching and iterative root finding locate the nearest canonical surface intersection for each image ray.Sampled deformed-space points are mapped to canonical correspondences and queried against the canonical occupancy field.
- 3.4. Training Objectives: Analytical gradients from implicit differentiation enable end-to-end optimization of geometry and deformation through surface, RGB, and mask supervision.The method also supports an optional FLAME loss using pseudo-ground-truth deformation values from nearby FLAME vertices.
4. Experiments
Experiments evaluate IMavatar on synthetic and real videos, testing geometry accuracy, image quality, expression fidelity, and generalization beyond training distributions. The method outperforms baselines broadly and preserves control under unseen expressions and poses.
- Experimental setup: Experiments use synthetic data with known geometry and real video sequences to evaluate geometry accuracy and expression generalization.The synthetic dataset contains 10 subjects with unseen, stronger test expressions; real videos contain mostly neutral training expressions and difficult unseen test expressions.
- Baselines: The evaluation compares IMavatar against alternative deformation modules, including conditioned networks, displacement warping, backward morphing, forward skinning, and an unsupervised variant.The unsupervised variant removes FLAME pseudo-ground-truth supervision and learns from images and masks.
- Synthetic results: IMavatar outperforms all baselines by a large margin across geometry, expression, and image-reconstruction metrics on the synthetic benchmark.The synthetic evaluation measures 10 identities across 12 COMA-rendered sequences.
- Synthetic results: Baseline geometric errors increase sharply for stronger expressions, whereas IMavatar shows only a slight increase as expression norm grows.Geometric error is measured as angular error of surface normals, with lower values preferred.
- Real-video results: On real videos with unseen expressions and poses, IMavatar reconstructs expressions more accurately while maintaining image quality comparable to competing methods.The method achieves a much lower facial keypoint error on novel test expressions and poses; the unsupervised variant has similar image and geometry quality but less accurate expression deformation.
- Synthetic results: IMavatar supports interpolation and extrapolation of expressions and poses beyond the training distribution, including jaw and neck pose variations.Figure 5 visualizes an expression component, jaw pitch, and neck yaw with five training-distribution samples.
5. Conclusion
IMavatar combines 3DMM-like expression and pose control with detailed implicit geometry and appearance, learning from RGB videos and extrapolating to strong unseen expressions. The method still has limitations involving hair occlusions, training speed, tracking accuracy, and mouth-interior appearance.
- IMavatar provides expression- and pose-controlled implicit head avatars that model diverse, detailed hairstyles and facial appearance.
- The method learns end-to-end from RGB videos, reconstructs deforming geometry accurately, and extrapolates to strong expressions beyond the training distribution.
- Surface representations cannot model fine occlusions produced by hair, motivating future combinations with volumetric representations.
- Iterative non-rigid ray marching makes IMavatar slow to train, requiring approximately 2 GPU days.
- Performance degenerates with noisy 3DMM parameters, while mouth-interior appearance can remain unrealistic.
Supplemental Materials
The supplemental document adds ablations, implementation and training details, broader-impact discussion, and an accompanying video of animated results.
- Supplementary materials provide additional ablation studies and results, implementation and training details, and discussion of broader impact.
- An accompanying video presents animated results and an explanation of the proposed method.
1. Additional Ablations and Results
Additional experiments compare IMavatar with state-of-the-art methods, test supervision and preprocessing choices, and examine generalization to diverse or unseen expressions and poses.
- IMavatar is evaluated against additional state-of-the-art baselines on real data using quantitative comparisons.
- The method achieves more accurate geometry than concurrent work on MakeHuman without test-time pose optimization.
- Without 3DMM supervision, expression error is higher with limited data, but TrainData+ yields comparable performance using only mask and RGB losses.
- More diverse training data improves learned expression and pose deformations, although neck geometry is not guaranteed to be correct.
- More diverse data and semantic segmentations improve mouth-interior geometry and texture, faithfully reproducing teeth when mouth-interior blendshape supervision is disabled.
- Foreground-mask degradation is tested at 10%, 50%, and 100%, while noisy 3DMM tracking and mask degradation are analyzed as preprocessing factors.
- IMavatar maintains low errors for strong jaw poses, whereas baseline errors increase drastically as jaw-pose magnitude grows.
2. Implementation Details
The implementation uses implicit geometry, texture, and deformation networks with ray-based canonical correspondence, iterative root finding, and video-based parameter preprocessing.
- Implicit morphing is illustrated as an alternative to mesh morphing for the avatar representation.
- The architecture contains separate geometry, texture, and deformation networks, with a 256-dimensional geometry latent feature conditioning texture prediction.
- The networks use positional encoding, geometric initialization, uniform skinning weights, zero blendshapes, and specified Softplus or ReLU activations.
- Baseline architectures differ in whether geometry or deformation networks are conditioned on expression and pose parameters or predict displacement distance.
- The renderer reconstructs an occupancy field rather than an SDF because SDF values may be unreliable after non-rigid deformation and cause ray-tracing overshoot.
- Canonical correspondences are found by initializing multiple locations and applying Broyden’s method with at most 10 updates and a 1e−5 convergence threshold.
- Training uses Adam for 60 epochs at η = 1e−4, with the learning rate halved after 40 epochs.
- Videos use one fixed camera, with head-rotation and talking sequences for training and strong unseen expressions for testing.
3. Broader Impact
IMavatars can reconstruct high-fidelity facial avatars from monocular videos, including extrapolation to unseen expressions from mild-deformation training footage. The authors also acknowledge risks of misuse, especially deep-fakes and identity dispossession.
- IMavatars can extrapolate to unseen expressions when trained on videos containing only mild deformations.
- This capability simplifies capture because users need not record overly restrictive expression sequences while retaining geometric quality.
- The method supports more accessible animatable-avatar creation from monocular videos without requiring expensive capture equipment.
- The authors identify potential abuse through deep-fakes and dispossession of identity when rendering individuals with controlled expressions and head pose.