Source-linked AI summary
GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians
Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, Liqiang Nie
TL;DR
GaussianAvatar addresses the challenge of reconstructing realistic, dynamic human avatars from a single monocular video despite inaccurate pose estimates and pose-dependent appearance changes. It uses animatable 3D Gaussians with dynamic properties and jointly optimizes motion and appearance, enabling realistic animation with real-time rendering speed.
Problem
Single-video avatar reconstruction must handle inaccurate monocular pose estimation and dynamic appearance changes in moving people.
Method
GaussianAvatar represents humans with animatable 3D Gaussians, augments them with pose-dependent properties, and jointly optimizes motion and appearance.
Results
GaussianAvatar reconstructs avatars with dynamic appearances, corrects initial motion misalignment, and maintains real-time rendering speed.
Takeaways & Limitations
The method supports realistic animation from monocular videos while improving reconstruction through motion refinement and dynamic appearance modeling.
Takeaways & Limitations
The method may produce artifacts from inaccurate foreground segmentation and struggles to model dynamic dresses because SMPL-derived skinning weights generalize poorly to new poses.
Abstract
from arXiv · showhide
We present GaussianAvatar, an efficient approach to creating realistic human avatars with dynamic 3D appearances from a single video. We start by introducing animatable 3D Gaussians to explicitly represent humans in various poses and clothing styles. Such an explicit and animatable representation can fuse 3D appearances more efficiently and consistently from 2D observations. Our representation is further augmented with dynamic properties to support pose-dependent appearance modeling, where a dynamic appearance network along with an optimizable feature tensor is designed to learn the motion-to-appearance mapping. Moreover, by leveraging the differentiable motion condition, our method enables a joint optimization of motions and appearances during avatar modeling, which helps to tackle the long-standing issue of inaccurate motion estimation in monocular settings. The efficacy of GaussianAvatar is validated on both the public dataset and our collected dataset, demonstrating its superior performances in terms of appearance quality and rendering efficiency.
1. Introduction
GaussianAvatar addresses the underdetermination, inaccurate motion estimates, and complex clothing deformations that make realistic avatar modeling from monocular video difficult. It uses animatable 3D Gaussians, dynamic appearance modeling, and joint motion–appearance optimization to improve consistency and realism.
- Monocular video leaves 3D animatable avatar modeling highly underdetermined, while faulty motion estimates and wrinkle deformations complicate realistic reconstruction.
- Existing implicit methods represent humans with deformable NeRF volumes and inverse skinning, which introduces ambiguous correspondences during canonicalization.
- GaussianAvatar explicitly models dynamic human surfaces with animatable 3D Gaussians and jointly optimizes motion and appearance end to end.
- A dynamic appearance network and optimizable feature tensor augment the representation with pose-dependent properties for learning motion-to-appearance mappings.
- Differentiable motion conditions allow motion refinement during avatar modeling, improving robustness to initial motion estimates and correcting motion-capture misalignment.
2. Related Work
Prior work reconstructs human avatars from monocular imagery using implicit neural fields, explicit representations, or direct regression. GaussianAvatar’s motivation is grounded in the limitations of implicit surface representation, fixed mesh resolution, and inaccurate monocular motion estimates.
- Neural Rendering for Human Reconstruction: Neural rendering methods reconstruct dynamic human appearances with approaches including latent vertex codes, deformable radiance fields, texture maps, pose-conditioned warping, and joint-structured embeddings.
- Neural Rendering for Human Reconstruction: NeRF-based methods can produce appealing rendering results but struggle to represent human surfaces with implicit 3D volumes.
- Explicit Representations: Explicit approaches use meshes, neural textures, dynamic surface deformation, or deformable point representations for avatar reconstruction.
- Avatar Modeling from Monocular Videos: Single-image regression methods recover clothed 3D humans but cannot recover dynamic appearance across an entire sequence.
- Avatar Modeling from Monocular Videos: Traditional preprocessing can require pre-scanning and manual rigging, while predefined-model-free methods may lose fine details because of fixed mesh resolution.
- Avatar Modeling from Monocular Videos: Monocular motion-estimation errors create pronounced artifacts, motivating methods that update inaccurate poses or learn error-correction networks.
3. Method
The method represents human surfaces with animatable 3D Gaussians, predicts pose-dependent Gaussian properties from UV surface features, and jointly optimizes motion and appearance. Its design includes isotropic Gaussian scaling and a two-stage training strategy for inaccurate monocular poses.
- 3.1. Animatable 3D Gaussians: Static 3D Gaussian Splatting represents scenes with centers, colors, opacities, quaternion rotations, and scaling factors while supporting high-quality real-time rendering.
- 3.1. Animatable 3D Gaussians: GaussianAvatar samples posed body-surface points into a UV positional map, combines pose features with an optimizable feature tensor, and decodes offsets, colors, and scales.
- 3.1. Animatable 3D Gaussians: Forward skinning reposes canonical Gaussians using propagated SMPL or SMPL-X skinning weights, avoiding inverse-skinning correspondence ambiguity.
- 3.2. Dynamic 3D Gaussian Property Estimation: The dynamic appearance network maps a 2D human manifold to pose-dependent Gaussian properties, while the feature tensor supplies coarse global appearance information.
- 3.2. Dynamic 3D Gaussian Property Estimation: Isotropic Gaussians are used because unbalanced monocular viewpoints can make anisotropic Gaussians fit frequent views and perform poorly in side views.
- 3.3. Joint Motion and Appearance Optimization: Motion updates are optimized as pose and translation corrections, then used in forward skinning to make rendered Gaussians differentiable with respect to motion conditions.
- 3.4. Training: Training uses two stages: first motion optimization and appearance fusion without pose-dependent information, then dynamic appearance learning from the refined motion and feature tensor.
4. Experiments
Experiments evaluate GaussianAvatar on People-Snapshot, NeuMan, and DynVideo using quantitative comparisons, qualitative novel-view synthesis, challenging poses, and ablations. The method outperforms baselines across reported metrics, improves motion estimation, and preserves pose-dependent appearance details.
- Datasets and comparisons: Evaluation covers People-Snapshot, NeuMan, and DynVideo, with comparisons against HumanNeRF and InstantAvatar.The study reports quantitative results and visualizes novel-view synthesis on test splits.
- Comparisons with the State of the Art: GaussianAvatar outperforms all baselines on all reported metrics for recovering dynamic-appearance details and correcting initialized-pose artifacts.The authors attribute boundary ghosting in HumanNeRF and InstantAvatar to their implicit representations.
- Comparisons with the State of the Art: GaussianAvatar generates realistic animation and consistent novel-view appearance for challenging out-of-distribution poses.Ground truth is unavailable for these poses, so the comparison is qualitative.
- Ablation Studies: Motion optimization dramatically improves the baseline on all metrics and corrects inaccurate pose estimates, including side and back views.The optimized motion can also enhance existing motion-capture methods.
- Ablation Studies: Dynamic appearance modeling preserves pose-dependent details such as cloth wrinkles, with stronger gains on NeuMan and DynVideo than on People-Snapshot.People-Snapshot contains limited pose variation relative to the other datasets.
5. Conclusion and Discussion
The conclusion presents GaussianAvatar as a monocular-video avatar reconstruction method that combines animatable 3D Gaussians, dynamic appearance modeling, and joint motion-appearance optimization. It reports realistic animation with real-time rendering while identifying segmentation and loose-clothing challenges.
- Conclusion: GaussianAvatar reconstructs human avatars from monocular videos using animatable 3D Gaussians with dynamic properties and joint motion-appearance optimization.The dynamic appearance network and optimizable feature tensor model pose-dependent appearance.
- Conclusion: The method enables realistic animation while maintaining real-time rendering speed.
- Limitations: Inaccurate foreground segmentation can cause artifacts, and loose outfits such as dresses remain challenging to model.
Supplementary Material
The supplementary material provides implementation details, dataset information, runtime comparisons, motion-optimization comparisons, challenging cases, and hand-animation results.
- Supplementary Material: The supplement organizes implementation details, dataset information, training and running-time comparisons, motion optimization, challenging cases, and hand animation across Sections 6–11.
6. Implementation Details
Implementation details describe the pose encoder, feature tensor, Gaussian decoder, training schedule, dataset distribution, runtime comparisons, and artifacts from inaccurate segmentation.
- Model Architecture: The pose encoder processes 128 × 128 × 3 SMPL UV maps using a five-block convolutional and five-block transposed-convolutional U-Net.
- Model Architecture: An 8-layer MLP with three prediction heads estimates Gaussian offsets, colors, and scales.
- Training: Training jointly optimizes the feature tensor, Gaussian decoder, and motion for 200 epochs, then fine-tunes the pose encoder and decoder for another 200 epochs.
- Dataset and runtime: Table 4 reports the number of frames assigned to training, validation, and testing for each sequence.
- Dataset and runtime: Table 5 compares training and running times, while inaccurate segmentation boundaries produce visible artifacts.
7. Dataset Details
The evaluation uses established video datasets and follows prior partitioning and protocol choices for comparison.
- Dataset Details: DynVideo uses the same partitioning settings as NeuMan.Dataset details are reported in Table 4.
- Dataset Details: The proposed DynVideo dataset is described through dataset details provided in Table 4.
- Dataset Details: NeuMan partitioning serves as the reference setting for the DynVideo dataset.
8. Training and Running Time Comparison
The paper compares GaussianAvatar’s inference speed with NeRF-based methods on People-Snapshot and presents related visual comparisons for motion and clothing cases.
- Training and Running Time Comparison: GaussianAvatar’s training and running time are compared with HumanNeRF and InstantAvatar on the People-Snapshot dataset.The measurements are reported in Table 5.
- Training and Running Time Comparison: Figure 9 compares ground-truth clothing with rendered images and Gaussian points for loose clothing.
- Training and Running Time Comparison: Figure 10 compares original, optimized, refined, and initial SMPL results for motion optimization.
9. Motion Optimization Comparison
GaussianAvatar’s pose-refinement capability is evaluated against InstantAvatar across sequences from the 3DPW and DNA-Rendering datasets.
- Motion Optimization Comparison: GaussianAvatar outperforms InstantAvatar in pose refinement on two 3DPW sequences and one DNA-Rendering sequence.The comparison is reported in Table 6 and Figure 10.
- Motion Optimization Comparison: The evaluation directly measures pose refinement for GaussianAvatar and the state-of-the-art InstantAvatar.
- Motion Optimization Comparison: The tested sequences come from the 3DPW and DNA-Rendering datasets.
10. Challenging Cases
The paper examines segmentation errors, loose clothing, and hand animation as challenging cases, while showing that hand animation can be extended without specific design changes.
- Segmentation: Foreground segmentation inaccuracies can produce black lines on the reconstructed surface.Automatic segmentation is not always satisfactory, while manual correction is time-consuming and inefficient.
- Hand Animation: Figure 11 presents reposed images alongside reference poses for hand-animation results.
- Loose Clothing: Dynamic dress appearance remains challenging because SMPL-derived skinning weights can create artifacts when generalized to new poses.The method may produce blurred clothing and incomplete point clouds.
- Hand Animation: GaussianAvatar generates plausible hand animation without specific design considerations when applied with an estimated SMPL-X model.The demonstration uses a sequence from the DynVideo dataset.