Source-linked AI summary
HUGS: Human Gaussian Splats
Muhammed Kocabas, Jen-Hao Rick Chang, James Gabriel, Oncel Tuzel, Anurag Ranjan
TL;DR
Existing human-rendering approaches struggle to combine detailed, animatable humans with efficient scene reconstruction from monocular video. HUGS uses deformable 3D Gaussians initialized from SMPL and coordinated by learned LBS weights, achieving fast training and rendering with state-of-the-art reconstruction quality.
Problem
Existing neural-rendering methods target static-scene photogrammetry, while parametric human models limit details such as clothing and hair and neural fields are less efficient to train and render.
Method
HUGS represents a human and static scene with 3D Gaussians, initializes human Gaussians from SMPL, permits deviations for details, and learns LBS weights for animation.
Results
HUGS reconstructs human and scene from 50-100-frame monocular videos in 30 minutes, renders at 60 FPS, trains approximately 100× faster than prior methods, and achieves state-of-the-art quality.
Takeaways & Limitations
HUGS enables novel-pose synthesis of the human and novel-view synthesis of both the human and scene from short in-the-wild monocular videos.
Takeaways & Limitations
HUGS is limited by SMPL and linear blend skinning for general deformable structures such as loose clothing, and its videos do not cover the human body's full pose space.
Abstract
from arXiv · showhide
Recent advances in neural rendering have improved both training and rendering times by orders of magnitude. While these methods demonstrate state-of-the-art quality and speed, they are designed for photogrammetry of static scenes and do not generalize well to freely moving humans in the environment. In this work, we introduce Human Gaussian Splats (HUGS) that represents an animatable human together with the scene using 3D Gaussian Splatting (3DGS). Our method takes only a monocular video with a small number of (50-100) frames, and it automatically learns to disentangle the static scene and a fully animatable human avatar within 30 minutes. We utilize the SMPL body model to initialize the human Gaussians. To capture details that are not modeled by SMPL (e.g. cloth, hairs), we allow the 3D Gaussians to deviate from the human body model. Utilizing 3D Gaussians for animated humans brings new challenges, including the artifacts created when articulating the Gaussians. We propose to jointly optimize the linear blend skinning weights to coordinate the movements of individual Gaussians during animation. Our approach enables novel-pose synthesis of human and novel view synthesis of both the human and the scene. We achieve state-of-the-art rendering quality with a rendering speed of 60 FPS while being ~100x faster to train over previous work. Our code will be announced here: https://github.com/apple/ml-hugs
1. Introduction
HUGS addresses the limits of parametric human models and neural fields by representing an animatable human and its scene with deformable 3D Gaussians. It learns this representation from short monocular videos while preserving details beyond SMPL and supporting fast synthesis.
- Parametric body models efficiently support rasterization and unseen deformations but limit clothing, hairstyles, and other geometric details through fixed topology.
- Neural fields capture clothing, accessories, and hair more effectively than textured parametric models, but are less efficient to train and render and are difficult to deform flexibly.
- HUGS represents the human and scene as 3D Gaussians, addressing dynamic-human challenges while enabling novel-pose human synthesis and novel-view synthesis of both human and scene.
- The method predicts Gaussian shifts, rotations, scales, and LBS weights from a canonical representation, initializes from SMPL, and allows deviations for hair and clothing.
- 50-100 frames and 30 minutes yield animatable avatars, with approximately 100× faster training, 60 FPS rendering, and state-of-the-art reconstruction quality.
2. Related Work
Prior work progressed from capture-heavy graphics pipelines to parametric models, neural fields, and Gaussian splatting. HUGS extends Gaussian splatting to dynamic humans while retaining novel-view and novel-pose capabilities.
- Traditional human capture required large multi-camera setups, detailed capture, artist-created templates, and manual rigging.
- Statistical body-shape models reduced manual template and rigging effort while enabling diverse human shapes and animation from a single model.
- NeRF-based methods enabled human capture from monocular video and joint human-scene representations, extending a framework originally designed for static objects.
- 3D Gaussian Splatting improves training and rendering times over NeRFs by representing scenes with explicit Gaussians and rasterizing them.
- HUGS adapts 3DGS for dynamic humans by combining SMPL priors with a learned deformation model and triplane-MLP coordination of Gaussian properties.
3. Method
HUGS disentangles a moving human from a static scene using Gaussian representations initialized from SMPL and structure-from-motion. Learned Gaussian properties and LBS weights support differentiable training and direct pose-controlled rendering.
- Given monocular video with camera motion, a moving human, and a static scene, HUGS represents both components as 3D Gaussians initialized from SMPL and COLMAP.
- Each Gaussian is modeled with opacity, mean, scale, rotation, and view-dependent radiance, then projected to the image plane for differentiable alpha-blended rendering.
- SMPL supplies pose, shape, and LBS initialization, while HUGS lets Gaussians deviate from the mesh to represent hair and clothing beyond fixed topology.
- A canonical-space triplane and three MLPs predict Gaussian appearance, geometry offsets, rotations, scales, and LBS weights for pose-dependent deformation.
- The learned LBS weights transform human Gaussians under joint configurations before combining them with scene Gaussians and splatting the result.
- Training optimizes Gaussian centers, the triplane, and MLPs using scene and human reconstruction losses plus LBS regularization, with pose parameters also adjusted during optimization.
- After optimization, Gaussians are explicitly constructed, so new human poses render directly through LBS without evaluating the triplane or MLPs.
4. Experiments
HUGS is evaluated on qualitative reconstruction, canonical human detail, human–scene disentanglement, benchmark metrics, and efficiency. Across these experiments, it generally outperforms prior methods while substantially reducing training and rendering time.
- Qualitative Results: HUGS preserves finer scene and human details than NeuMan and Vid2Avatar, especially around hands, feet, clothing wrinkles, and background objects.Qualitative comparisons report better reconstruction for both scene backgrounds and human regions, including house numbers, wooden planks, cupboards, hands, feet, wrinkles, and zippers.
- Canonical Human Shapes: HUGS captures fine canonical-space details around the feet, hands, and face, which supports high reconstruction quality during animation.NeuMan produces noisy feet and hands, whereas HUGS preserves richer facial detail in the canonical human shape.
- Disentanglement of the Human and the Scene: HUGS separately stores human and scene Gaussian features, allowing a captured human to be composed into a different scene.Figure 4 demonstrates transferring Human Gaussians from the original capture scene to another scene.
- Quantitative Results: HUGS achieves state-of-the-art performance across all scenes and metrics on ZJU-MoCap.The comparison includes NeuralBody, HumanNerf, and MonoHuman using the ZJU-MoCap benchmark.
- Quantitative Results: HUGS achieves state-of-the-art performance across NeuMan scenes and metrics except PSNR on Bike, where it remains competitive.Table 1 evaluates PSNR, SSIM, and LPIPS on NeuMan test images; the human-region evaluation similarly reports state-of-the-art results except PSNR on Jogging.
- Speed: HUGS trains 96× faster than Vid2Avatar and 336× faster than NeuMan within 30 minutes, while rendering at 60 FPS.Rendering is approximately 7600× faster than NeuMan and 3800× faster than Vid2Avatar; the benchmark uses one GeForce 3090Ti GPU.
- Ablation Study: Removing LBS introduces floating artifacts, disabling densification produces edge floaters, and removing human-pixel loss removes fine human details.Directly optimizing 3DGS parameters without the triplane+MLP produces noisy estimates.
5. Conclusion
HUGS brings deformable human modeling into Gaussian Splatting for novel-view and novel-pose synthesis from short monocular videos. It trains and renders quickly while improving rendering quality, but remains bounded by SMPL, LBS, limited pose coverage, and lighting assumptions.
- HUGS reconstructs a human embedded in a scene from 50-100-frame in-the-wild monocular videos for novel-view and novel-pose synthesis.
- 30 minutes and 60 FPS make HUGS 100× faster than previous methods while improving rendering quality on PSNR, SSIM, and LPIPS.
- HUGS is limited by SMPL and linear blend skinning for loose clothing, incomplete human pose coverage, and unmodeled environment lighting.
6. Appendix
The appendix details HUGS’s lightweight decoder architecture, LBS regularization, adaptive Gaussian optimization, and ablations demonstrating the importance of its design choices. Additional renderings show novel poses and transfer of animated humans across scenes.
- Model architecture: The triplane-MLP architecture uses lightweight decoders to predict Gaussian appearance, geometry, and deformation parameters efficiently.The appearance decoder predicts opacity and spherical harmonics; the geometry decoder predicts mean-shifts, rotations, and scales.
- Deformation and regularization: The deformation decoder predicts LBS weights, while low-temperature softmax and SMPL-based ℓ2 regularization encourage mostly single-bone assignments and reduce unseen-pose artifacts.The regularizer compares predicted weights with distance-weighted averages from the six nearest SMPL vertices.
- Gaussian optimization: Adaptive densification and pruning adjust Gaussian counts during optimization to represent high-frequency details that uniform SMPL initialization misses.The method clones, splits, and prunes Gaussians based on loss gradients and opacity.
- Ablation experiments: Replacing learned LBS with SMPL weights creates artifacts near joint intersections, while removing densification causes protruding Gaussians and poorer details such as boot laces.
- Ablation experiments: Joint human-scene optimization constrains scene reconstruction, mitigating floaters and producing cleaner rendered images than sequential optimization.
- Novel animation renderings: Novel-animation renderings include NeuMan subjects in new poses and multiple animated subjects transferred into different scenes using AMASS poses.