Source-linked AI summary
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians
Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, Matthias Nießner
TL;DR
Photorealistic head-avatar reconstruction must combine novel-view quality with control over expression, pose, and viewpoint. GaussianAvatars rigs 3D Gaussian splats to a parametric morphable face model, achieving strong image quality and expression accuracy while retaining flexible animation control. The authors also identify limits in relighting and modeling hair or accessories.
Problem
Creating head avatars that jointly capture photorealistic appearance, geometry, dynamics, and controllable novel expressions and poses remains challenging.
Method
GaussianAvatars rigs 3D Gaussian splats to a parametric morphable face model for dynamic, controllable head-avatar representation.
Results
GaussianAvatars outperforms existing works by a significant margin on novel-view rendering and driving-video reenactment.
Takeaways & Limitations
The representation enables flexible control and precise attribute transfer while modeling fine head details beyond the morphable mesh.
Takeaways & Limitations
The current approach cannot relight avatars and lacks control over FLAME-unmodeled areas such as hair and accessories.
Abstract
from arXiv · showhide
We introduce GaussianAvatars, a new method to create photorealistic head avatars that are fully controllable in terms of expression, pose, and viewpoint. The core idea is a dynamic 3D representation based on 3D Gaussian splats that are rigged to a parametric morphable face model. This combination facilitates photorealistic rendering while allowing for precise animation control via the underlying parametric model, e.g., through expression transfer from a driving sequence or by manually changing the morphable model parameters. We parameterize each splat by a local coordinate frame of a triangle and optimize for explicit displacement offset to obtain a more accurate geometric representation. During avatar reconstruction, we jointly optimize for the morphable model parameters and Gaussian splat parameters in an end-to-end fashion. We demonstrate the animation capabilities of our photorealistic avatar in several challenging scenarios. For instance, we show reenactments from a driving video, where our method outperforms existing works by a significant margin.
1. Introduction
Animatable photorealistic head avatars remain difficult because reconstruction must jointly represent appearance, geometry, and dynamics while supporting controllable novel poses and expressions. GaussianAvatars addresses this by rigging 3D Gaussian splats to a parametric morphable face model.
- Animatable human-head avatars remain a longstanding computer-vision and graphics problem.
- Photorealistic rendering from arbitrary viewpoints, controllable animation, and generalization to novel poses and expressions support applications from gaming to immersive telepresence.
- Reconstruction is under-constrained because it must capture appearance, geometry, and dynamics while preserving photorealism and expression controllability.
- NeRF-based dynamic methods achieve novel-view rendering quality but lack controllability and therefore generalize poorly to novel poses and expressions.
- Dynamic 3D Gaussian Splatting extensions build explicit temporal correspondences but do not support animation of reconstructed outputs.
- GaussianAvatars rigs 3D Gaussian splats to a parametric morphable face model and introduces binding inheritance to add or remove splats without losing controllability.
2. Related Work
Related work progresses from neural and voxelized radiance fields toward explicit, controllable representations for dynamic bodies and heads. GaussianAvatars builds on this trajectory by coupling triangle-based morphable-face deformation with optimizable 3D Gaussians.
- NeRF represents radiance fields with neural networks, while voxel grids, hashing, and tensor decomposition improve rendering efficiency.
- Dynamic-scene methods using 4D coordinates or tensors can replay realistic scenes but lack an explicit handle for content manipulation.
- Proxy geometries such as SMPL meshes provide more direct controllability than deformation MLPs for dynamic human representations.
- Head-avatar research has combined differentiable rendering, neural synthesis, face tracking, expression transfer, and FLAME-based geometry enhancement.
- Unlike INSTA’s nearest-triangle querying, GaussianAvatars maintains consistent Gaussian–triangle correspondences; unlike differential point splatting, Gaussian scales are optimizable.
3. Method
The method rigs 3D Gaussian splats to a FLAME mesh, optimizes them through differentiable rendering, and regularizes their geometry for controllable animation with fewer artifacts.
- Pipeline: The pipeline fits FLAME parameters from multi-view observations, renders Gaussian splats differentiably, and supervises the images with ground-truth views.It uses a photometric head tracker and known camera parameters before optimizing the rendered appearance.
- 3D Gaussian Rigging: Each Gaussian is paired with a mesh triangle and stored in that triangle’s local space, then transformed into global space as the mesh moves.Local position, rotation, and anisotropic scaling are converted for rendering using the parent triangle’s transformation.
- 3D Gaussian Rigging: Triangle scaling makes Gaussian position and scale relative to the parent triangle, enabling constant local-space learning rates with metric-space step sizes adapted to triangle size.Gaussians attached to smaller triangles move more slowly per iteration than those attached to larger triangles.
- Density Control: Adaptive density control splits or clones high-gradient Gaussians and prunes low-opacity ones while inheriting parent-triangle bindings for newly created splats.Binding inheritance preserves rigging during densification, while pruning suppresses floating artifacts but can over-remove frequently occluded regions.
- Optimization and Regularization: ϵscaling = 0.6 disables the scaling regularizer below 0.6× the parent-triangle scale, preventing excessive shrinkage that would slow rendering.Without this tolerance, camera rays may need to intersect more splats before reaching zero transmittance.
- Optimization and Regularization: The objective combines RGB reconstruction with position and scaling regularizers to keep splats aligned with their parent triangles during animation.The RGB term combines L1 and D-SSIM losses, while position and scale thresholds tolerate small local deviations.
4. Experiments
Experiments evaluate reconstruction and animation through novel-view synthesis, self-reenactment, and cross-identity reenactment, then test individual method components through ablations.
- 4.1. Setup: The evaluation covers novel-view synthesis, self-reenactment, and cross-identity reenactment using held-out views, unseen motions, and source parameters from another subject.Quantitative training uses 9 of 10 prescribed sequences and 15 of 16 cameras, with a free-performance sequence for visual cross-identity assessment.
- 4.2. Head Avatar Reconstruction and Animation: GaussianAvatars produces sharper novel-view renderings, reconstructs fine details, and shows more accurate self-reenactment expressions than baselines.The comparisons highlight reflective eye light, hair strands, teeth, and sharper outputs.
- 4.2. Head Avatar Reconstruction and Animation: Consistent triangle binding corrects tracked-mesh misalignment during optimization, avoiding the neck and shoulder noise observed with nearest-triangle warping.Positional gradients consistently back-propagate to the same triangle, allowing Gaussian splats to compensate for inaccurate FLAME tracking.
- 4.2. Head Avatar Reconstruction and Animation: The method outperforms alternatives on novel-view metrics and achieves significantly lower LPIPS in self-reenactment, although tracked FLAME misalignment disadvantages pixel-wise metrics such as PSNR.AvatarMAV struggles with novel-expression synthesis because its learned deformation bases must extrapolate beyond the training distribution.
- 4.2. Head Avatar Reconstruction and Animation: Cross-identity reenactment transfers expressions vividly and renders complex dynamics, while baselines exhibit artifacts, imprecise motion, or poor generalization to novel expressions.The evaluation drives reconstructed avatars with tracked FLAME expression and pose parameters from source actors.
- 4.3. Ablation Study: Ablations show that binding inheritance, scaling regularization with error tolerance, position regularization, and FLAME fine-tuning each support avatar fidelity or animation.Removing binding inheritance causes blurry renderings; removing position loss improves training-frame novel-view metrics but creates cracks and floating blobs under unseen motion.
5. Limitations and Potential Negative Impacts
The approach has technical and scope limitations, including infeasible relighting, missing control for hair and accessories, and risks of harmful misuse.
- Relighting is not feasible because the Gaussian-splat representation directly captures radiance without decoupling material and lighting.
- Position and scaling losses are used to prevent artifacts during animation with novel expressions and poses.
- The current approach lacks control over hair and other accessories because these areas are not modeled by FLAME.
- Photorealistic avatars raise risks including privacy violations, deceptive deepfakes, defamation, reputational harm, identity theft, and impersonation.
6. Conclusion
GaussianAvatars creates photorealistic, controllable head avatars by rigging 3D Gaussian splats to a parametric morphable face model. It reports large-margin gains in image quality and expression accuracy over state-of-the-art methods.
- GaussianAvatars creates photorealistic avatars from video sequences using a dynamic 3D representation of Gaussian splats rigged to a parametric morphable face model.
- The splats can deviate from the mesh surface to represent fine head details absent or inaccurately modeled by the morphable model.
- The method reports large-margin improvements over state-of-the-art methods in image quality and expression accuracy.
A. FLAME Tracking
FLAME tracking jointly optimizes per-frame and shared parameters using landmark, color, and regularization losses, with a staged optimization schedule. Revised eyes and rigidly attached tooth triangles improve avatar fidelity.
- FLAME Tracking: FLAME tracking optimizes per-frame translation, joint poses, and expression alongside shared shape, vertex-offset, and albedo parameters.
- FLAME Tracking: The optimization combines landmark loss, color loss, and regularization terms, with Laplacian and temporal smoothness used for regularization.
- FLAME Tracking: The staged schedule optimizes the first frame to convergence, subsequent per-frame parameters for 50 iterations, then all parameters for 30 global epochs.
- FLAME Tracking: The 2023 FLAME version revises eye regions, while 168 manually added teeth triangles are rigidly attached to the neck and jaw joints.
- FLAME Tracking: Rigidly moving teeth triangles help Gaussian splats capture teeth details.
B. Dataset Division
The dataset uses NeRSemble recordings divided across novel-view, self-reenactment, and cross-identity settings, with background and below-shoulder pixels removed during preprocessing.
- Dataset Division: The dataset contains 11 sequences per subject, with one emotion or expression sequence held out for self-reenactment and the remaining nine used for training.
- Dataset Division: Table 3 lists the sequence types and IDs assigned to the different evaluation settings.
- Dataset Division: Table 4 lists the held-out sequence for each subject in self-reenactment evaluation.
- Dataset Division: Background matting removes image backgrounds, and projected lines remove pixels below the shoulder to simplify Gaussian-splat optimization.
C. Computation Efficiency
GaussianAvatars maintains efficient optimization despite substantial Gaussian growth, while supporting real-time rendering at varied resolutions.
- Gaussian binding to triangles maintains high rendering and optimization speed despite active Gaussian addition and pruning.
- 10,144 Gaussians grow to around 100,000 on average, after which the count stops increasing during training.The stabilized count prevents ever-increasing memory requirements.
- 1000% Gaussian growth increases optimization-iteration runtime by less than 100% at the peak.
- The model can be trained on an NVIDIA RTX 2080 Ti with 12 gigabytes of VRAM.
- A scaling-loss threshold limits blending work, preventing substantially higher rendering time from many additional Gaussians.The differentiable tile rasterizer sorts splats before blending and stops ray marching at zero transmittance.
- The optimized model is rendered at arbitrary resolutions, with average FPS measured across variant resolutions for real-time applications.
D. Baselines
The evaluation compares GaussianAvatars with three state-of-the-art head-avatar methods using distinct representations and deformation or conditioning strategies.
- The comparison includes INSTA, PointAvatar, and AvatarMAV as state-of-the-art head-avatar creation methods.
- INSTA: INSTA warps points to the nearest FLAME triangle and enhances mouth quality using added triangles, expression conditioning, and region-specific losses.
- PointAvatar: PointAvatar uses a point-based representation with FLAME-conditioned deformation, coarse-to-fine optimization, and image-based hole filling.
- AvatarMAV: AvatarMAV represents canonical appearance and motion with voxel grids, blending motion bases with tracked morphable-model expression vectors.