GauHuman: Articulated Gaussian Splatting from Monocular Human Videos

Introduction

• NeRF-based human methods can take about 10 GPU hours to train and render below 1 FPS, limiting real-world use.Source: §1 · p. 1
• Generalizable Human NeRF methods trade quality for shorter training, while hash-encoding methods speed training but remain limited by ray-cast rendering, especially at 1080p and 4k.Generalizable methods also require several hours of pre-training and an additional hour of per-human fine-tuning.Source: §1 · pp. 1–2
• Gaussian Splatting for articulated humans must integrate monocular human information and optimize efficiently; prior-based LBS can be inaccurate, while generic initialization and densification can create redundant Gaussians.The paper identifies closely positioned Gaussians as an opportunity for further acceleration.Source: §1 · p. 2
• GauHuman encodes Gaussians in canonical space and uses LBS to transform their positions and covariances into posed space, with learned weight offsets and pose refinement.It initializes LBS weights from SMPL rather than estimating them directly with an MLP.Source: §1 · p. 2
• Its optimization initializes and prunes Gaussians using human priors, guides splitting and cloning with KL divergence, and merges nearby Gaussians to accelerate optimization.Pruning removes Gaussians far from the human body, such as those far from SMPL vertices.Source: §1 · p. 2
• GauHuman trains in 1–2 minutes and renders at up to 189 FPS, modeling performers with ~13k Gaussians and 3.5MB storage.The paper reports these results for its final method.Source: §1 · p. 2

Rendering and Radiance Fields

• NeRF [72] and its acceleration methods advance radiance-field rendering, while point splatting and 3D Gaussians represent scenes efficiently; GauHuman extends splatting to articulated humans.Earlier approaches include light fields, photo blending, and MVS; 3D Gaussians have been applied to static scenes [52], dynamic scenes [68] [118] [124] [125], and other tasks [86] [108].Source: §2 · p. 3

3D Human Modelling

• 3D human methods use dense sensors, learned priors, or implicit representations, but face limited pose generalization or slow training and rendering; GauHuman targets monocular video with both efficiencies.Multi-hashing methods [30] [46] reduce training to several minutes but retain comparatively slow rendering; neural volumetric primitives [14] [66] [84] still require hours of training. Concurrent work [121] targets real-time rendering from dense-view videos.Source: §2 · p. 3

3. Our Approach

• The method assumes calibrated camera parameters, human-region masks, and corresponding SMPL shape and pose parameters are given.Source: §3 · p. 3

3.1. Preliminary

• SMPL supplies the shape and pose model for LBS, while Gaussian Splatting [52] represents scenes as discrete 3D Gaussians with learnable position, rotation, and scale.The Gaussian covariance is parameterized by rotation and scale to ensure positive semi-definiteness.Source: §3.1 · p. 3

3.2. Articulated 3D Gaussian Splatting

• Canonical-space Gaussians are forward-warped by LBS, with an MLP predicting offsets to nearest-SMPL-vertex weights and another correcting estimated joint poses; learned weights and corrected poses are cached for inference.The LBS-weight MLP uses positional encoding of each Gaussian’s canonical position; caching avoids recomputing weights and pose corrections during inference.Source: §3.2 · pp. 4–5

3.3. Fast Optimization of GauHuman

• SMPL vertices initialize Gaussians; k-NN-limited KL divergence guides split and clone at O(U) rather than O(U^2), while nearby redundant Gaussians are merged and off-body Gaussians pruned.Merging targets Gaussians with large position gradients, small scales, and KL divergence below 0.1; it averages their positions, opacity, and SH coefficients.Source: §3.3 · pp. 5–6
• GauHuman achieves state-of-the-art PSNR and LPIPS on ZJU_MoCap and MonoCap, with 1–2 minutes of training and rendering up to 189 FPS.FPS is measured on an RTX 3090.Source: Table 1 · p. 6

4.1. Experimental Setup

• Experiments use six ZJU_MoCap subjects and four MonoCap videos, with 100 training frames per subject sampled every five frames and the datasets’ masks, cameras, and SMPL parameters.ZJU_MoCap uses one camera for training and the remaining cameras for evaluation; MonoCap uses the camera views specified by [79]. Comparisons include subject-specific and generalizable methods, and InstantAvatar [46] is evaluated without test-time SMPL optimization.Source: §4.1 · p. 6

4.2. Quantitative Results

• GauHuman achieves slightly better performance than InstantNVR [30] and InstantAvatar [46] with 3 times faster training and 45 times faster rendering; it trains in around 1 or 2 minutes.Compared with baseline methods NB [78], AN [77], AS [79], and HumanNeRF [114], which take around 10 hours to train, generalizable methods require about 1 hour of fine-tuning after 10 hours of pre-training. DVA [84] takes about 90x more time to converge and 11.45x more time to render one image than GauHuman.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3. Qualitative Results

• GauHuman shows better appearance and geometry details than DVA [84], NB [78], and NHP [55], while NB and NHP perform poorly with monocular videos.HumanNeRF [114] and AS [79] achieve comparable visualizations but take 600x more training time and 472x more rendering time; InstantNVR [30] and InstantAvatar [46] show slightly worse results and require 45x more rendering time. DVA [84] produces garment and face artifacts.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4. Ablation Study

• Pose refinement and the LBS weight field improve all reported metrics at negligible computational cost and help remove artifacts and recover cloth wrinkles.Without either module, the visualization shows obvious hand artifacts and fewer cloth wrinkles; all ablation models train in around 1 minute.Source: §4.4 · p. 7, Table 2 · p. 8, Figure 4 · p. 8
• The ablation study uses sequence 386 and camera-plane bounding-box masks; sparse SfM initialization is not compared because it requires multi-view inputs unavailable in monocular videos.The setup discusses random or sparse-point initialization approaches, and the figure illustrates random initialization with 6,890 or 30,000 Gaussians.Source: §4.4 · pp. 7–8, Figure 4 · p. 8
• KL-based split and clone operations achieve 60x faster convergence than the original split and clone operations in [52], without sacrificing performance, while learning a human performer with about 13k 3D Gaussians.The novel merge operation merges about 8.5% of 3D Gaussians without degrading performance; articulated pruning with an SMPL prior removes about 0.5k Gaussians far from the 3D human body. Split and clone lead to up to 100k 3D Gaussians, which slows optimization.Source: §4.4 · p. 8

5. Discussion and Conclusion

• 1∼2 minutes training and 166 FPS rendering accompany GauHuman’s reported state-of-the-art performance on monocular videos, using ∼13k Gaussians to model performers.The conclusion also characterizes rendering as real-time.Source: §5 · pp. 8–9
• GauHuman represents Gaussians in canonical space and uses LBS, pose refinement, and LBS weight fields to learn human details; prior-guided initialization and pruning, KL-divergence-guided splitting/cloning, and merging speed optimization.Source: §5 · pp. 8–9
• The pipeline does not support extracting 3D meshes, and recovering cloth wrinkles from monocular videos remains challenging.The authors suggest incorporating garment physical simulation as a direction for addressing wrinkle recovery.Source: §5 · p. 9

A. Implementation Details

• The pose-refinement network uses four fully connected layers, while the LBS-offset network uses five; both use ReLU activations and Adam at 10−5.Hidden dimensions are 128 for pose refinement; its input and output dimensions are 69. The LBS-offset module takes 63 inputs and outputs 24 joint offsets, with positional encoding applied to Gaussian positions; KL-guided split/clone uses threshold 0.4.Source: A. Implementation Details · p. 15
• Novel-view and novel-pose image quality is evaluated with PSNR, SSIM, and LPIPS.Other 3D Gaussian training details follow [52].Source: A. Implementation Details · p. 15
• Comparisons include subject-specific neural-field, hashing, and volumetric-primitive methods, as well as generalizable pixel-feature methods.The described methods are Neural Body [78], Animatable NeRF [77], AS [79], HumanNeRF [114], InstantNVR [30], InstantAvatar [46], DVA [66], PixelNeRF [128], and Neural Human Performer [55]. For generalizable methods, each evaluated subject is fine-tuned after pre-training on the other dataset.Source: A. Implementation Details · p. 15
• Factoring Gaussian covariance into rotation and scaling matrices simplifies inverse and determinant calculations, reducing their computation time.The scaling matrix is diagonal and the rotation matrix orthogonal, enabling inverses and determinants to be derived from their components.Source: A. Implementation Details · p. 15
• GauHuman omits rotation of spherical-harmonic coefficients because it has little effect on final performance.The described rotation computes a Wigner D-matrix [116] to rotate coefficients for view-dependent color effects when transforming Gaussians from canonical to posed space.Source: A. Implementation Details · p. 15

• Photometric loss compares the ground-truth target image with the predicted image.Source: B. Details of Loss Functions · p. 15
• Mask loss leverages human-region masks, comparing accumulated volume density with the ground-truth binary mask label.Source: B. Details of Loss Functions · p. 15
• SSIM loss encourages structural similarity between ground-truth and synthesized images.Source: B. Details of Loss Functions · p. 15
• LPIPS perceptual loss is used to ensure rendered-image quality.Source: B. Details of Loss Functions · p. 15

C. Experiments on DNA-Rendering

• The evaluation uses sequences 0012_09 and 0025_11 from part 1, with 100 training frames, one training camera, and four nearby test views.Source: C. Experiments on DNA-Rendering · pp. 15–16
• The PyTorch implementation of the Wigner D-matrix is time-consuming because it uses a matrix exponential operation [76].Source: C. Experiments on DNA-Rendering · p. 15
• GauHuman produces high-quality 3D performers with fast training and rendering, while AS [79] and InstantAvatar [46] struggle with complex clothing and fast-moving actors.Figure 5 compares novel-view synthesis and shows training time and rendering speed for the methods on DNA-Rendering.Source: Figure 5 · p. 16, C. Experiments on DNA-Rendering · p. 16

D. Comparison with concurrent work GART [57]

• GART [57] achieves comparable novel-view synthesis to state-of-the-art baselines on ZJU_MoCap and renders at 77 FPS using Gaussian Splatting.The reported comparison reproduces GART using its released code and excludes test-time SMPL optimization on test images.Source: D. Comparison with concurrent work… · p. 16
• Training takes 1m vs. 3m and rendering runs at 189FPS vs. 77FPS for GauHuman and GART, respectively, with slightly better novel-view synthesis for GauHuman.The comparison omits test-time optimization of GART's SMPL parameters using test-set images for fairness.Source: D. Comparison with concurrent work… · p. 16
• GauHuman's fast optimization initializes and prunes Gaussians with a 3D human prior, guides splitting and cloning with KL divergence, and adds a merge operation.It models the performer with ∼13k Gaussians without sacrificing rendering quality.Source: D. Comparison with concurrent work… · p. 16