Source-linked AI summary
GauHuman: Articulated Gaussian Splatting from Monocular Human Videos
Shoukang Hu, Ziwei Liu
TL;DR
Existing implicit 3D human representations remain costly to train or render from monocular videos. GauHuman uses articulated Gaussian Splatting with canonical-space modeling, LBS-based posing, and human-prior-guided optimization; it reports state-of-the-art performance with 1∼2-minute training, real-time rendering, and roughly 13k Gaussians.
Problem
Existing NeRF-based human representations require substantial training and rendering time, while efficient alternatives still face rendering limitations for monocular 3D human modeling.
Method
GauHuman encodes Gaussians in canonical space, transforms them to posed space with LBS, refines pose and LBS weights, and optimizes Gaussians using human priors, KL-guided splitting or cloning, and merging.
Results
GauHuman achieves state-of-the-art novel view synthesis with fast training and real-time rendering, optimizing in around 1 or 2 minutes and reaching up to 189 FPS.
Takeaways & Limitations
GauHuman can fast model 3D human performers with approximately 13k 3D Gaussians without sacrificing rendering quality.
Takeaways & Limitations
The method assumes calibrated camera parameters, human region masks, and corresponding SMPL parameters are known.
Abstract
from arXiv · showhide
We present, GauHuman, a 3D human model with Gaussian Splatting for both fast training (1 ~ 2 minutes) and real-time rendering (up to 189 FPS), compared with existing NeRF-based implicit representation modelling frameworks demanding hours of training and seconds of rendering per frame. Specifically, GauHuman encodes Gaussian Splatting in the canonical space and transforms 3D Gaussians from canonical space to posed space with linear blend skinning (LBS), in which effective pose and LBS refinement modules are designed to learn fine details of 3D humans under negligible computational cost. Moreover, to enable fast optimization of GauHuman, we initialize and prune 3D Gaussians with 3D human prior, while splitting/cloning via KL divergence guidance, along with a novel merge operation for further speeding up. Extensive experiments on ZJU_Mocap and MonoCap datasets demonstrate that GauHuman achieves state-of-the-art performance quantitatively and qualitatively with fast training and real-time rendering speed. Notably, without sacrificing rendering quality, GauHuman can fast model the 3D human performer with ~13k 3D Gaussians.
1. Introduction
Existing 3D human methods are costly to train and render, motivating a faster articulated representation for monocular videos. GauHuman combines Gaussian Splatting with human-aware transformations and optimization to achieve fast training and real-time rendering.
- Motivation: NeRF-based 3D human methods can learn from sparse-view videos or a single image, but typically require about 10 GPU hours and render below 1 FPS.These costs hinder real-world applications.
- Motivation: Generalizable and hash-encoded alternatives reduce training time but still require substantial pre-training or fine-tuning and often retain limited rendering quality or speed.Generalizable methods may require several hours of pre-training plus one hour per human, while hash encoding reduces training time by a magnitude but leaves rendering as a bottleneck.
- Challenges: Gaussian Splatting offers a promising route to real-time articulated-human rendering, but monocular videos raise challenges in integrating human articulation and efficiently optimizing the representation.The paper frames these as two primary barriers to applying Gaussian Splatting to 3D humans.
- GauHuman: GauHuman encodes Gaussian Splatting in canonical space and transforms Gaussians into posed space using linear blend skinning, with pose and LBS refinement modules for fine details.The method starts from SMPL LBS weights and predicts corresponding weight offsets rather than estimating all weights directly with an MLP.
- GauHuman: GauHuman trains in 1∼2 minutes and renders at up to 189 FPS, while using human-prior initialization and pruning, KL-guided splitting or cloning, and Gaussian merging for faster optimization.Experiments on two monocular datasets report state-of-the-art novel view synthesis performance with fast training and real-time rendering.
2. Related Work
Prior work spans implicit neural representations for sparse-view or single-image human reconstruction and efficiency-oriented encodings. Although multi-hashing reduces training time to several minutes, rendering remains comparatively slow.
- Implicit Representations: Implicit neural representations reconstruct 3D human performers from sparse-view videos or even a single image.These methods extend novel-view synthesis to limited-input human capture settings.
- Implicit Representations: Implicit human reconstruction methods usually require hours of training and seconds of rendering per frame, restricting real-world applications.Their computational demands motivate more efficient representations.
- Efficient Representations: Multi-hashing encoding reduces training time to several minutes, but rendering remains comparatively slow; neural volumetric primitives have also been proposed for efficient 3D human rendering.The related work therefore separates training acceleration from rendering acceleration.
3. Our Approach
GauHuman represents articulated humans with canonical-space 3D Gaussians transformed into posed space through LBS, while refining pose and skinning weights. Its human-prior-guided initialization and adaptive Gaussian control support efficient optimization and fast inference.
- 3.1. Preliminary: GauHuman assumes calibrated cameras, human-region masks, and corresponding SMPL shape and pose parameters are known.
- 3.1. Preliminary: SMPL defines body shape and pose parameters, and its LBS algorithm transforms canonical-space points into posed space using joint transformations and blend weights.
- 3.1. Preliminary: Each Gaussian is represented by a world-space center and full covariance, with covariance parameterized by learnable rotation and scaling components.
- 3.2. Articulated 3D Gaussian Splatting: Canonical-space Gaussians are transformed to posed space by rotating and translating their positions and covariances with estimated LBS transformations.
- 3.2. Articulated 3D Gaussian Splatting: GauHuman predicts offsets to SMPL-derived LBS weights with an MLP and learns relative joint-angle updates through a pose correction module.
- 3.2. Articulated 3D Gaussian Splatting: Precomputed LBS weights and rectified poses reduce inference computation, enabling GauHuman to model high-quality humans at high rendering speed.
- 3.3. Fast Optimization of GauHuman: Human priors guide Gaussian initialization and pruning, while position gradients, scaling, and KL divergence regulate adaptive Gaussian control during optimization.
4. Experiments
Experiments on ZJU_MoCap and MonoCap evaluate GauHuman against subject-specific and generalizable baselines, including qualitative, quantitative, and ablation studies. GauHuman combines fast optimization with high-quality novel-view synthesis and substantially faster rendering.
- Experimental Setup: Experiments use ZJU_MoCap and MonoCap, with one training camera and remaining cameras for evaluation on six ZJU_MoCap subjects.Baselines include subject-specific optimization methods and generalizable methods; test-time SMPL optimization is omitted for InstantAvatar.
- Quantitative Results: GauHuman achieves photorealistic novel-view rendering after about 1–2 minutes of optimization, whereas several baselines require around 10 hours of training.Generalizable methods additionally require about 1 hour of per-human fine-tuning after 10 hours of pre-training.
- Qualitative Results: GauHuman provides better appearance and geometry details than DVA, NB, and NHP in qualitative comparisons.NB and NHP perform poorly with monocular videos, while DVA produces artifacts on garments and faces.
- Ablation Study: Pose refinement and the LBS weight field significantly improve all metrics with negligible computational cost, with models trained in around 1 minute.The ablation evaluates masked image regions on sequence 386 of ZJU_MoCap.
- Ablation Study: KL-guided split and clone operations achieve 60x faster convergence, while merging removes about 8.5% of Gaussians without degrading performance.The resulting model uses about 13k Gaussians, and articulated pruning removes about 0.5k Gaussians away from the body.
5. Discussion and Conclusion
GauHuman represents articulated humans with canonical-space Gaussian Splatting transformed by LBS, targeting fast training and real-time rendering. The paper reports state-of-the-art performance with compact Gaussian models, while identifying mesh extraction and monocular cloth-detail recovery as open limitations.
- Discussion and Conclusion: GauHuman targets fast training in 1–2 minutes and real-time rendering at 166 FPS for 3D human performers.The model also includes pose refinement and an LBS weight field to learn fine human details.
- Discussion and Conclusion: Canonical-space Gaussians are transformed to posed space with LBS, while human priors guide initialization and pruning and KL divergence guides splitting and cloning.A merge operation further speeds optimization.
- Discussion and Conclusion: Experiments on monocular videos report state-of-the-art performance with fast training, real-time rendering, and about 13k Gaussians.The reported compact model does not sacrifice rendering quality.
- Limitations: The current pipeline does not support extracting 3D meshes, and recovering cloth wrinkles from monocular videos remains challenging.The paper suggests garment physics as one possible future direction for cloth detail recovery.
A. Implementation Details
The implementation uses lightweight refinement networks and an efficient KL-divergence computation based on covariance factorization. Spherical harmonic rotation is omitted because it has little effect on final performance.
- Implementation Details: The pose refinement module uses four fully connected layers with two 128-dimensional hidden layers, while the LBS offset module uses five layers with three hidden layers.Both modules use ReLU activations; the pose module has input and output dimension 69.
- Implementation Details: KL-divergence computation is simplified by factoring each covariance as Σ = RSST RT, making diagonal scaling inverses and determinants inexpensive.The inverse of the orthogonal rotation matrix is its transpose, reducing matrix-operation cost.
- Implementation Details: Rotating spherical harmonic coefficients with a Wigner D-matrix has little effect on final performance and is therefore omitted.The rotation would otherwise account for view-dependent color effects when transforming Gaussians between spaces.
B. Details of Loss Functions
GauHuman optimizes rendered humans using photometric, mask, structural-similarity, and perceptual losses. These losses respectively constrain image agreement, human-region agreement, structural similarity, and perceptual quality.
- Photometric loss compares the predicted image with the ground-truth target image.
- Mask loss leverages human-region masks during optimization.
- SSIM loss enforces structural similarity between ground-truth and synthesized images.
- LPIPS loss is used to ensure the quality of rendered images.
C. Experiments on DNA-Rendering
On DNA-Rendering, GauHuman is evaluated against representative baselines using monocular training data and nearby camera views for testing. The experiments report photorealistic rendering quality together with training and rendering efficiency.
- Figure 5 compares novel-view synthesis results from GauHuman and baseline methods on the DNA-Rendering dataset.
- 100 frames from two DNA-Rendering sequences are used for training, with one camera for training and four nearby views for testing.
- GauHuman produces higher-quality human renderings than AS and InstantAvatar on complex clothing and fast-moving actors.
D. Comparison with concurrent work GART [57]
GauHuman is compared with concurrent work GART on ZJU-MoCap. It achieves slightly better novel-view synthesis while training and rendering faster, using an efficient Gaussian optimization strategy.
- GauHuman achieves slightly better novel-view synthesis performance than GART.
- Human-prior initialization and pruning, KL-guided splitting and cloning, and Gaussian merging enable GauHuman’s fast optimization.
- ∼13k 3D Gaussians model the performer without sacrificing rendering quality.