Source-linked AI summary
SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting
Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, Zeyu Wang
TL;DR
Photorealistic, animatable avatars often face a trade-off between detailed appearance and efficient, portable rendering. SplattingAvatar combines explicit mesh-based motion with Gaussian Splatting and trainable embeddings, achieving state-of-the-art rendering quality while running in real time, including on mobile devices.
Problem
Realistic, animatable human avatars need detailed high-frequency geometry and appearance without the computational burden and motion-control limitations of conventional meshes and MLP-based methods.
Method
SplattingAvatar embeds trainable Gaussians on a triangle mesh and jointly optimizes Gaussian parameters and mesh embeddings with lifted optimization for explicit mesh-driven deformation.
Results
SplattingAvatar achieves the best rendering quality for head and full-body avatars reconstructed from monocular videos and runs at real-time frame rates on a mobile device.
Takeaways & Limitations
The hybrid representation separates mesh-controlled motion and deformation from Gaussian-based rendering, supporting real-time avatar modeling with mesh-based motion control.
Takeaways & Limitations
The method relies tightly on the underlying driving mesh, whose motion quality can matter more than fully reconstructing exact geometry.
Abstract
from arXiv · showhide
We present SplattingAvatar, a hybrid 3D representation of photorealistic human avatars with Gaussian Splatting embedded on a triangle mesh, which renders over 300 FPS on a modern GPU and 30 FPS on a mobile device. We disentangle the motion and appearance of a virtual human with explicit mesh geometry and implicit appearance modeling with Gaussian Splatting. The Gaussians are defined by barycentric coordinates and displacement on a triangle mesh as Phong surfaces. We extend lifted optimization to simultaneously optimize the parameters of the Gaussians while walking on the triangle mesh. SplattingAvatar is a hybrid representation of virtual humans where the mesh represents low-frequency motion and surface deformation, while the Gaussians take over the high-frequency geometry and detailed appearance. Unlike existing deformation methods that rely on an MLP-based linear blend skinning (LBS) field for motion, we control the rotation and translation of the Gaussians directly by mesh, which empowers its compatibility with various animation techniques, e.g., skeletal animation, blend shapes, and mesh editing. Trainable from monocular videos for both full-body and head avatars, SplattingAvatar shows state-of-the-art rendering quality across multiple datasets.
1. Introduction
SplattingAvatar addresses the tension between realistic, animatable avatars and efficient rendering by combining mesh-based motion control with Gaussian Splatting for detailed appearance. It introduces trainable mesh embeddings and lifted optimization to support explicit, adaptable Gaussian deformation.
- Increasing avatar realism through polygons, textures, and hair systems raises computational demands and obstructs efficient, portable rendering.
- Traditional meshes efficiently capture body motion and surface deformation but struggle with detailed geometry, topological changes, and complex appearance.
- NeRF-based avatars capture high-frequency details but require ambiguous backward mappings from posed samples to canonical space.
- SplattingAvatar embeds Gaussians on a mesh using trainable barycentric coordinates and displacement, with pose-dependent transformations responding to mesh warping.
- The hybrid representation can be trained from monocular video, ported to Unity, and rendered in real time while supporting editable mesh-driven motion.
- Unlike segmented hybrid models, SplattingAvatar uniformly renders avatar parts with Gaussian Splatting while disentangling motion from appearance.
- Jointly optimizing Gaussian parameters and mesh embeddings applies lifted optimization to avatar reconstruction.
2. Related Work
Related work spans mesh-based, implicit neural, and hybrid avatar representations. SplattingAvatar builds on these directions by combining forward Gaussian rendering with trainable mesh embeddings for efficient, detailed, and adaptable avatars.
- Mesh-based avatar: Mesh-based avatars provide expressive textured surfaces and controllable motion, often using shape priors and differentiable rendering to maintain consistency across large motions.
- Hybrid avatar representation: SplattingAvatar's pipeline optimizes Gaussian and embedding parameters on a canonical mesh, then uses mesh deformation for posed-space rasterization.
- Implicit neural avatar: NeRF avatars represent high-frequency details in canonical space but rely on non-trivial backward tracing from posed space.
- Implicit neural avatar: PointAvatar uses explicit point primitives and forward rasterization, while SplattingAvatar further improves efficiency and mesh-embedding compatibility over LBS-based deformation.
- Hybrid avatar representation: Trainable embeddings allow Gaussians to optimize their mesh locations and distribute unevenly according to texture complexity.
3. Method
SplattingAvatar embeds trainable 3D Gaussians on a canonical triangle mesh, combining mesh-driven motion and deformation with detailed Gaussian appearance. It updates embeddings across triangle boundaries during joint optimization and renders the deformed Gaussians differentiably.
- Gaussian embedding: Each Gaussian is embedded by triangle index, barycentric coordinates, and normal displacement, defining a continuous neighborhood around the mesh surface.The embedding E = {k, u, v, d} uses Phong-surface position and normal interpolation, with displacement d defining the Gaussian mean.
- Mesh-driven deformation: The mesh directly controls Gaussian positions, rotations, and scaling during deformation, while each Gaussian retains its own appearance and canonical parameters.Pose-dependent rotation is interpolated from per-vertex quaternions, and scaling follows the embedded triangle’s area change.
- Lifted optimization: Gaussian and embedding parameters are optimized simultaneously, with barycentric updates re-expressed in neighboring triangles when an embedding crosses a boundary.The walking procedure recursively transfers updates across adjacent triangles and supports clone, prune, and split operations for embeddings.
- Rendering: Differentiable Gaussian rendering projects the deformed 3D Gaussians into camera views using covariance-based splatting and alpha blending.The implementation uses CUDA pixel loops or Unity front-parallel quads with standard rasterization for mobile deployment.
- Training objectives: A scaling regularizer addresses long, thin Gaussians caused by limited viewing angles and pose variation in monocular video.The regularization activates when the maximum scale is both large and disproportionate to the minimum scale; random backgrounds also provide silhouette cues.
4. Experiments
Experiments evaluate SplattingAvatar on head and full-body avatar datasets, showing strong rendering quality, pose generalization, and artifact reduction through ablations. Results also reveal that driving-mesh quality and motion representation affect performance.
- Evaluation setup: The evaluation compares SplattingAvatar with state-of-the-art methods on separate head-avatar and full-body-avatar datasets.Head-avatar experiments use a combined dataset, while full-body evaluation uses PeopleSnapshot.
- Head avatars: Both SplattingAvatar variants outperform existing methods on average photometric errors, with Ours+NHA performing best by the metrics.Ours+FLAME nevertheless shows better visual quality in inner rendered-image regions.
- Full-body avatars: Figure 6 shows photorealistic full-body rendering, facial detail, thin wrist accessories, and qualitative generalization to novel poses.The comparison separates PeopleSnapshot results from novel-pose animation examples.
- Full-body avatars: SplattingAvatar achieves the lowest pixel-wise errors on PeopleSnapshot and produces better overall quality than InstantAvatar, especially in facial regions.All three methods have close LPIPS values, while the method shows slightly more artifacts under the shoulder because training poses are limited.
- Ablation study: Removing trainable embeddings causes irregular artifacts, while removing scaling regularization produces needle-like artifacts in novel poses.Using both components removes most of these artifacts in the ablation results.
- Driving mesh: NHA's better-aligned mesh improves generalization to large pose variations, while the driving mesh should primarily provide accurate motion.The discussion also notes that current FLAME and SMPL-X models lack separate motion representations for clothes and hair.
5. Conclusion
The paper concludes that SplattingAvatar combines mesh-based motion with Gaussian Splatting for high-quality human avatars. It reports state-of-the-art rendering for head and full-body avatars from monocular videos while running in real time on mobile devices.
- Conclusion: SplattingAvatar combines trainable Gaussian embeddings on a mesh with explicit motion representation and implicit rendering.The method jointly optimizes Gaussian parameters and their mesh embeddings through lifted optimization.
- Conclusion: The approach achieves the best rendering quality for head and full-body avatars reconstructed from monocular videos and runs at real-time frame rates on a mobile device.The authors position the method as a foundation for Gaussian Splatting manipulation with mesh-based motion control.
Supplementary Material
The supplementary material documents the head-avatar dataset, implementation details, and additional experimental comparisons.
- Head-avatar examples: Supplementary head-avatar examples show high-fidelity details in eyes, hair, and off-surface geometry such as glasses.The dataset includes subjects from publicly available sources.
- Supplementary scope: The supplement covers the head-avatar dataset, implementation details, and additional experimental comparisons.These topics are organized in Sections A, B, and C, respectively.
A. Dataset
The head-avatar evaluation uses 10 subjects collected from four publicly available datasets, with rendering results shown on testing samples.
- Dataset: The evaluation contains 10 subjects drawn from INSTA, NHA, IMAvatar, and NerFace.The rendering results use Ours+FLAME and show high-fidelity details especially in the eyes, hair, and glasses.
B. Implementation Details
Training uses fixed optimization settings and periodic Gaussian densification, pruning, and opacity resets, while Unity renders each Gaussian as a quad primitive. The walking-on-triangle step supports mesh-based Gaussian motion and takes around 3.5 ms.
- Training: Opacity for all Gaussians is reset every 3,000 iterations to remove redundant Gaussians.The opacity-reset process stops at iteration 15,000.
- Unity rendering: Unity rendering draws one quad primitive for each Gaussian and uses alpha blending for the final image.The implementation represents each Gaussian with a quad and illustrates its 2D covariance as an ellipse.
- Running time: 3.5 ms is required for walking on triangle, compared with 2.5 ms for densify-clone and 6 ms for densify-split.The walking step is implemented with pybind11 and runs after densifying and pruning.
- Unity rendering: Walking on triangle associates the Gaussians with the current driving mesh pose before rendering.The Unity pipeline shows the driving mesh, Gaussian quads, their 2D covariances, and the alpha-blended result.
C. Additional Results
Additional evaluations compare SplattingAvatar with FLARE, INSTA, PointAvatar, and NHA, including qualitative, error-map, and ablation results. The reported results emphasize photometric quality, efficiency, and the importance of walking on the triangle mesh for novel poses.
- Comparison with FLARE: SplattingAvatar emphasizes significant photometric-quality and rendering-efficiency improvements over FLARE, while FLARE reconstructs accurate geometry and materials.The comparison is conducted on the head avatar dataset, with qualitative results shown in Figure A4.
- Non-ambiguous motion control: INSTA produces significantly more noise than the embeddings-based motion control in complex regions such as the eye.The cited comparison attributes the noise to nearest-triangle deformation from posed space to canonical space.
- Error map: SplattingAvatar and INSTA show overall better quality in l1 RGB error maps than PointAvatar and NHA.The passage associates PointAvatar and NHA limitations with their point-based and mesh-based representations, respectively.
- Error map: Photometric-error metrics were affected mainly in the neck area by segmentation and head-tracking errors in preprocessing.The error maps are presented to show the evaluated methods despite this preprocessing limitation.
- Ablation on walking on triangle: Disabling walking on triangle causes Gaussians to stick and pile up at triangle boundaries, producing artifacts under novel poses.The ablation is reported for a full-body avatar and is also documented in the corresponding figure and table.