Source-linked AI summary
MGAvatar: Mesh-Bound Gaussians for Head Avatar Geometry and Appearance Modeling
Lei Shi, Sen Peng, Zhiyang Deng, Zhonggui Chen, Xiaohu Guo, Baorong Yang, Xiao Dong
TL;DR
FLAME-based Gaussian head avatars have limited personalized geometry and struggle with structures such as hair and clothing, while appearance can be unstable across views and poses. MGAvatar uses vertex-bound Gaussians for geometry and face-bound Gaussians for appearance, with deformation and view-conditioned color modeling. It reports superior visual quality and generalization in novel-view synthesis and reenactment.
Problem
FLAME-based Gaussian avatars lack personalized geometric priors and struggle with beyond-template structures such as hair, while appearance may be unstable under novel views and poses.
Source: §1 · pp. 1–2Method
MGAvatar combines vertex-bound Gaussians for mesh geometry refinement with face-bound Gaussians for appearance, using pose-dependent deformation, Gaussian offsets, and a view-conditioned neural color field.
Source: §1 · p. 2, §3.1 · p. 4, §3.2 · pp. 4–5Results
Takeaways & Limitations
The optimized mesh remains compatible with FLAME driving for full-head animation, including hair and shoulders, within a unified representation.
Source: §1 · p. 2Because MGAvatar retains standard FLAME topology, its ability to model complex hair strands and clothing wrinkles is limited.
Source: §4.5 · p. 8
Abstract
from arXiv · showhide
Accurate head modeling requires a stable yet expressive geometric representation. Existing Gaussian-based head avatars commonly rely on parametric templates (e.g., FLAME) for Gaussian initialization and deformation, but these templates lack personalized priors and struggle to represent structures such as hair and clothing. To address this issue, we propose MGAvatar, a Gaussian-mesh hybrid representation that jointly models geometry and appearance through two Gaussian-mesh binding modes. Specifically, we introduce vertex-bound Gaussians and constrain their learnable parameters, enabling progressive mesh deformation to represent complex head geometry, while a pose-dependent offset module accounts for non-rigid deformations. Once geometry is stabilized, MGAvatar switches to face-bound Gaussians for appearance modeling. To improve appearance consistency across novel poses and viewpoints, we introduce a view-conditioned neural color field that alleviates artifacts caused by independently optimized Gaussian colors. In addition, we design a Gaussian offset network to predict Gaussian offset maps in the observation space, providing greater flexibility for face-bound Gaussians to capture dynamic facial textures. Extensive experiments on multi-view and monocular videos show that MGAvatar outperforms existing methods in rendering quality, producing high-fidelity head avatars with rich texture details.
1. Introduction
Head-avatar methods span mesh, NeRF, point, and Gaussian representations, but FLAME-guided Gaussian approaches still struggle with beyond-template geometry and appearance stability. MGAvatar combines mesh-bound Gaussians to model geometry and appearance, with a neural color field and Gaussian attribute offsets.
1. Introduction
- Head-avatar reconstruction approaches use mesh, NeRF, point-based, and 3D Gaussian representations, each trading off efficiency, fidelity, and animatability.Source: §1 · p. 1
- FLAME-guided Gaussian methods struggle with hair and shoulders beyond the template, non-rigid facial deformation, and unstable appearance under novel views and poses.Prior hair-Gaussian methods require complex preprocessing and cannot use FLAME to drive hair motion.Source: §1 · pp. 1–2
- MGAvatar’s mesh can be driven by FLAME parameters for full-head animation, including hair and shoulders, within a unified representation.Source: §1 · p. 2
- MGAvatar combines vertex-bound and face-bound Gaussians, using reduced-DoF geometric learning to refine identity-specific geometry for animatable avatars.The approach also uses Gaussian attribute offsets and a view-conditioned neural color field to improve consistency across novel viewpoints.Source: §1 · p. 2
- Experiments on multi-view and monocular datasets report improvements over state-of-the-art methods in novel-view synthesis and avatar reenactment.Source: §1 · p. 2
2. Related Work
Related work spans parametric and explicit meshes, implicit radiance fields, point-based models, and Gaussian avatars. Prior Gaussian methods combine 3DMM guidance with initialization, surface sampling, or mesh binding, while some optimize template parameters or displacements.
2. Related Work
- Earlier head-avatar approaches use 3DMMs and explicit meshes, with extensions to improve detail and methods that predict geometry and color or model deformation, materials, and illumination.Source: §2 · p. 2
- NeRF-based work models head appearance with implicit fields or expression-weighted radiance bases; point-based avatars offer topological flexibility, and DELTA separates face and hair modeling.Source: §2 · p. 2
- Gaussian-avatar methods include learnable deformable Gaussians, mesh-and-Gaussian hybrids, expression-blended models, and separate head-and-hair representations that require preprocessing and hair-motion estimation.Source: §2 · p. 2
- 3DMM guidance for Gaussian avatars includes template initialization, UV-guided sampling, and face-bound Gaussians that move with the mesh while allowing limited attribute adaptation.Source: §2 · p. 2
- Some methods optimize 3DMM parameters or predict pose- and expression-based displacements; MeGA’s smooth UV-space offsets limit complex hair modeling, unlike MGAvatar’s template adaptation and deformation modeling.MGAvatar’s overview depicts geometry refinement with vertex-bound Gaussians followed by face-bound Gaussian appearance modeling.Source: §2 · p. 2, Figure 1 · p. 3
3. Method
MGAvatar first refines a personalized, animatable mesh with vertex-bound Gaussians, then freezes geometry and switches to face-bound Gaussians for appearance modeling. Regularization stabilizes geometry, while view-conditioned color and attribute offsets add appearance flexibility.
3.1. Geometry Optimization of Animatable Head Avatar
- Vertex-bound Gaussians, initialized at subdivided-mesh vertices, support identity-specific mesh refinement and direct vertex offsets that better accommodate hair and shoulders.Gaussian attributes are constrained to reduce degrees of freedom during geometric learning.Source: §3.1 · p. 3
- A pose-dependent network predicts offsets from pose parameters and triplane features, combining them with identity-specific offsets to model non-rigid motion.Offset magnitudes and mesh Laplacian consistency are regularized to encourage stable deformations.Source: §3.1 · p. 4
3.2. Appearance Optimization Based on Face-Bound Gaussians
- After geometry stabilizes, MGAvatar freezes its geometry network and binds Gaussians to mesh faces, where triangle-local parameters allow appearance modeling to vary independently of geometry.Appearance learning uses color and SSIM supervision; adaptive cloning and splitting add Gaussians for locally complex details.Source: §3.2 · pp. 4–5
- ColorNet combines hash-grid features of canonical Gaussian positions with camera-relative view direction to predict view-dependent color.It addresses limitations of low-order spherical harmonics when views are limited and poses or expressions are challenging or unseen.Source: §3.2 · p. 4
- Gaussian Offset Net predicts per-Gaussian attribute-offset maps from canonical and posed mesh inputs, a learnable identity latent, and pose features.Sampling the maps supplies position, color, rotation, scale, and opacity offsets for the face-bound Gaussians used in rendering.Source: §3.2 · p. 5
4. Experiments
Experiments across multi-view and monocular videos compare MGAvatar with 3DGS-based avatar methods and ablate its geometry and appearance components. Results report stronger rendering quality, while optimized meshes and ablations illustrate the method’s geometry, appearance, and scope.
4.1. Datasets and Baselines
- Evaluation covers NeRSemble multi-view novel-view synthesis and self-reenactment, monocular self-reenactment, and cross-identity reenactment, against methods selected for each setting.NeRSemble has 11 sequences per subject captured from 16 viewpoints; monocular evaluation uses 10 videos from four datasets.Source: §4.1 · p. 6
4.2. Implementation Details
- Training uses 340k iterations, with the first 100k devoted to geometry before RefineNet is frozen for appearance learning.The implementation also specifies separate learning rates for RefineNet, ColorNet, and OffsetNet, and describes the identity- and pose-offset networks and appearance networks.Source: §4.2 · p. 6
4.3. Comparative Experiments
- MGAvatar outperforms the compared methods on NeRSemble novel-view synthesis and self-reenactment, with qualitative results showing detailed hair, clothing, teeth, and cross-identity expression transfer.The comparison also reports consistent geometry and appearance under novel poses; Figure 4 illustrates expression transfer while preserving identity-specific geometry and appearance.Source: §4.3 · p. 7, Figure 4 · p. 7
- Vertex-level optimization produces a personalized mesh spanning face, hair, ears, and shoulders while preserving FLAME topology for compatible control and animation.Geometry learning converges in approximately 30 minutes, compared with more than 3 hours of per-subject mesh preprocessing for GaussianAvatars; MeGA’s geometry optimization is described as even more time-consuming.Source: §4.3 · pp. 7–8, Figure 5 · p. 7
4.4. Ablation Study
- Removing geometry or appearance modules reduces performance on novel views and expressions; pose-dependent offsets preserve facial detail, Laplacian regularization stabilizes meshes, and OffsetNet adapts details to expressions and poses.Without Laplacian regularization, the mesh shows flipped triangles, irregular face scales, and broken connectivity; without OffsetNet, high-frequency details such as clothing wrinkles are less well recovered.Source: §4.4 · p. 8, Figure 7 · p. 8
4.5. Limitations and Future Works
- The fixed FLAME topology limits modeling of hair strands and clothing wrinkles, and incomplete oral structures plus weak monocular geometric constraints leave mouth reconstruction accuracy open to improvement.The authors identify more flexible 3D representations for hair, clothing, and oral structures as future work.Source: §4.5 · p. 8
5. Conclusion
MGAvatar uses mesh-bound Gaussians with distinct degrees of freedom to separate geometry and appearance optimization. Experiments report high-fidelity appearance and superior visual quality and generalization in novel-view synthesis and reenactment.
5. Conclusion
- MGAvatar combines two mesh–Gaussian binding strategies with different degrees of freedom to decouple geometry and appearance optimization.Joint optimization of parametric templates, mesh deformation, ColorNet, and OffsetNet learns personalized geometry compatible with FLAME-based driving while producing high-fidelity appearance.Source: §5 · p. 8
- The method learns personalized geometry compatible with FLAME-based driving while producing high-fidelity appearance.The conclusion attributes this to joint optimization of parametric templates, mesh deformation, ColorNet, and OffsetNet.Source: §5 · p. 8
- Experiments demonstrate superior visual quality and generalization in novel-view synthesis and reenactment tasks.Source: §5 · p. 8