PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Gaussian Splatting: Real-time Rendering of Animatable Avatars

Arthur Moreau, Jifei Song, Helisa Dhamo, Richard Shaw, Yiren Zhou, Eduardo Pérez-Pellitero

arXiv:2311.17113v2cs.CVcs.GR

TL;DR

Neural human avatars can render slowly and generalize poorly to body poses not seen in training. HuGS uses canonical Gaussian primitives with forward skinning and local neural refinement; the authors report competitive or better results across three public datasets while rendering an order of magnitude faster than neural-based human models.

  • Problem

    Neural human representations can be difficult to render in real time and often struggle to maintain quality on body poses not observed during training.

    Source: §1 · p. 1

  • Method

    HuGS represents the body with canonical 3D Gaussians, using forward skinning for articulated motion and a pose-dependent MLP for local garment refinement.

    Source: §1 · p. 2

  • Results

    On three public datasets, HuGS achieves results on par with or better than state-of-the-art neural human models while rendering one order of magnitude faster.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that Gaussian splatting supports fast human-avatar rendering and good generalization, and may facilitate deployment as an intuitive baseline.

    Source: §6 · p. 8

    With sparse camera setups, reconstruction can overfit training observations and produce poor novel-pose synthesis; the MLP reproduces observed garment deformations but does not extrapolate novel ones, and Gaussians ignore local neighborhood relations.

    Source: §5 · p. 8

Abstract

from arXiv · show

This work addresses the problem of real-time rendering of photorealistic human body avatars learned from multi-view videos. While the classical approaches to model and render virtual humans generally use a textured mesh, recent research has developed neural body representations that achieve impressive visual quality. However, these models are difficult to render in real-time and their quality degrades when the character is animated with body poses different than the training observations. We propose an animatable human model based on 3D Gaussian Splatting, that has recently emerged as a very efficient alternative to neural radiance fields. The body is represented by a set of gaussian primitives in a canonical space which is deformed with a coarse to fine approach that combines forward skinning and local non-rigid refinement. We describe how to learn our Human Gaussian Splatting (HuGS) model in an end-to-end fashion from multi-view observations, and evaluate it against the state-of-the-art approaches for novel pose synthesis of clothed body. Our method achieves 1.5 dB PSNR improvement over the state-of-the-art on THuman4 dataset while being able to render in real-time (80 fps for 512x512 resolution).

1. Introduction

Neural human avatars can capture detailed clothed-body appearance, but remain difficult to render in real time and struggle to generalize to novel poses. HuGS combines Gaussian splatting, forward skinning, and local neural refinement for animatable human avatars.

1. Introduction

  • Neural avatars based on NeRF or SDF offer detailed clothed-body shape and texture, but are slow to train and render and struggle with novel body poses.The introduction contrasts their quality on training poses with weaker quality on unseen poses.Source: §1 · p. 1
  • HuGS represents the human body with 3D Gaussian Splatting and refines forward-skinned primitives with a shallow network for local garment movement.The authors report that learned Gaussian skinning weights generalize to new poses, while LBS alone does not capture local garment deformations.Source: §1 · p. 2
  • HuGS combines forward skinning for skeleton-based movement with a pose-dependent MLP for local garment deformation, and is reported to render one order of magnitude faster than neural-based human models.The authors compare the method on three public datasets and report results on par with or better than state of the art.Source: §1 · p. 2

2. Related Work

Related work spans implicit radiance fields and animatable human models. HuGS builds on explicit Gaussian primitives and forward skinning while addressing local non-rigid motion and loose clothing.

Differentiable rendering of radiance fields

  • NeRF-style methods evaluate networks along camera rays, and ray marching inherently limits rendering speed even when information is stored explicitly.3D Gaussian Splatting instead sorts and splats explicit primitives, enabling real-time rendering.Source: §2 · p. 2

Animatable human body models

  • Animatable body models map a canonical representation, often a reference pose, into posed space; SMPL templates commonly use linear blend skinning for this deformation.Avatars can be learned from scans, videos, or images, while this review focuses on multi-view RGB video capture.Source: §2 · p. 2
  • Some neural human models learn backward correspondences, but pose-dependent deformation fields can generalize poorly and joint root-finding approaches can be slow.Examples include Neural Actor, Animatable NeRF, and InstantNVR for pose-dependent fields, and ARAH, TAVA, and PoseVocab for root-finding.Source: §2 · p. 2
  • Forward-skinned primitives avoid backward correspondence, but methods built on the SMPL surface often struggle with loose clothing, while local neural refinements address garment and muscle motion.Rigid skeleton motion alone does not capture local wrinkles or contractions; learned local refinement is a common alternative to difficult physics-based modeling.Source: §2 · p. 3

3. Method

HuGS learns a canonical Gaussian representation from multi-view video and deforms it from canonical to posed space. Forward skinning handles articulated movement, an MLP refines local garment motion and shading, and differentiable rasterization renders the result.

3. Method

  • The model is trained from calibrated multi-view videos, body poses, and foreground masks to synthesize a character at target views and poses.The input poses use SMPL or SMPL-X parameters, including body shape and joint rotations.Source: §3 · p. 3

3.1. Canonical representation

  • Each canonical Gaussian has learnable position, orientation, scale, color, opacity, skinning weights, and a latent code for pose-dependent motion.Scale and opacity are considered consistent across novel views and poses.Source: §3.1 · p. 3

3.2. Deformation with forward skinning.

  • LBS weight-averages pose-specific joint transformations to move and rotate canonical Gaussians, with skinning weights learned per primitive.The authors apply forward skinning directly, rather than the backward formulations used by ray-marched NeRF and SDF approaches.Source: §3.2 · p. 3

3.3. Local non-rigid refinement

  • A pose-dependent MLP adds per-Gaussian residuals to LBS outputs, refining position, orientation, and color to represent local garment motion and pose-dependent shading.The local pose encoding uses skinning weights and an attention map; the MLP takes this pose encoding and a per-Gaussian latent code as input.Source: §3.3 · pp. 3–4

3.4. Image rendering

  • The resulting Gaussians are rendered from the target camera view with the fast, differentiable rasterizer from 3D-GS.The method overview depicts deformation followed by observation-space rendering.Source: §3.4 · p. 4, Figure 2 · p. 4

3.5. Training procedure

  • Training renders posed Gaussians against observed images and optimizes Gaussian and MLP parameters using reconstruction losses and regularization.The losses include L1, D-SSIM, and perceptual terms; regularizers constrain MLP residuals, keep canonical positions near the SMPL mesh within a clothing-distance threshold, and softly supervise skinning weights from SMPL.Source: §3.5 · pp. 4–5

4. Experiments

The experiments compare HuGS with prior methods on THuman4, DNA-Rendering, and ZJU-MoCap, then test its deformation and representation choices through ablations. Across these evaluations, the paper reports strong novel-pose results and real-time rendering.

4.3.1 Evaluation on THuman4 dataset

  • On THuman4, HuGS achieves the best performance on all metrics for novel-view and novel-pose synthesis and supports real-time inference.Rendering a 512x512 image takes approximately 12 ms (80 fps); training takes 5 to 20 hours on a Tesla V100, depending on dataset size.Source: Table 1 · p. 6, §4.3.1 · p. 6, §4.5 · pp. 7–8

4.3.2 Evaluation on DNA-Rendering dataset

  • On DNA-Rendering, HuGS outperforms DVA, preserving details and fitting unusual clothing topologies under fast motion and complex textures.The template is used only at initialization, allowing the canonical representation to fit arbitrary shapes.Source: §4.3.2 · p. 6, Table 2 · p. 6

4.3.3 Evaluation on ZJU-MoCap dataset

  • On ZJU-MoCap, HuGS ranks second on novel views and achieves the best novel-pose PSNR, with competitive SSIM and greater training and rendering efficiency than baselines.Training uses only 4 of 21 cameras and is limited to 50k iterations because further training degrades novel-pose textures under the sparse camera setup.Source: §4.3.3 · p. 6, Table 3 · p. 6

4.4. Ablation studies

  • LBS alone is quantitatively competitive but misses garment wrinkles and detail, while an MLP alone fails to represent large deformations such as arms.The full method combines both deformations.Source: §4.4 · pp. 6–7
  • >1 dB PSNR improvement over SMPL-template weights is obtained with learned per-Gaussian skinning weights in the LBS-only ablation.The authors attribute this to adapting weights to body shape and avoiding possible template-weight errors from pose-estimation misalignment.Source: §4.4 · p. 7, Table 4 · p. 7
  • Learned latent codes perform slightly better than canonical position encoding for the MLP while using 16 rather than 63 floats.The authors suggest learned features help small MLPs decode information; with more MLP layers, positional encodings may work similarly.Source: §4.4 · p. 7, Table 5 · p. 7
  • Without shading, novel-pose synthesis scores 29.72dB PSNR, 0.978 SSIM, and 0.027 LPIPS, while duplicate colored Gaussians lead to overfitting.These scores are directly comparable with those in Table 1.Source: §4.4 · p. 7

4.5. Efficiency

  • 80 fps at 512x512 is reported for rendering on a single Tesla V100, with transformation computation taking 10 ms and rasterization 2 ms.Training takes 5 to 20 hours on a Tesla V100, depending on dataset size.Source: §4.5 · pp. 7–8

5. Discussion

The discussion identifies limits in sparse-camera reconstruction, garment-deformation generalization, and independent Gaussian deformation. It also notes potential misuse and the growth of concurrent Gaussian-splatting work.

5. Discussion

  • Sparse camera setups can lead to overfitting on training observations, degraded 3D reconstruction, and poor novel-pose synthesis.The authors also note that the MLP replicates observed garment deformations but does not extrapolate novel ones, and that independently optimized Gaussians ignore local neighborhood relations.Source: §5 · p. 8
  • Figure 7 compares the full LBS-plus-posed-MLP model with versions using only LBS or only MLP, alongside ground truth.The caption specifies the image order but does not state the outcome of the visual comparison.Source: Figure 7 · p. 8
  • The authors warn that HuGS could be used to synthesize malicious deepfake videos through reenactment.They say this potential needs careful mitigation.Source: §5 · p. 8
  • Many preprints on human modeling with Gaussian splatting appeared during the paper’s review process.The paper discusses key similarities and differences in supplementary materials.Source: §5 · p. 8

6. Conclusion

The conclusion presents HuGS as an animatable human-avatar approach built on Gaussian Splatting and coarse-to-fine deformation. It reports state-of-the-art neural-rendering performance and argues that fast rendering may facilitate deployment.

6. Conclusion

  • HuGS combines forward skinning with local learning-based refinement in a coarse-to-fine deformation algorithm for animatable human avatars.The authors describe it as a first approach for creating and animating virtual human avatars based on Gaussian Splatting.Source: §6 · p. 8
  • HuGS’s Gaussian Splatting representation accelerates rendering and bypasses difficult inverse-skinning approaches used in NeRF formulations.The conclusion also says HuGS can fit body shapes with loose clothing, unlike other forward approaches.Source: §6 · p. 8
  • The authors report state-of-the-art human neural-rendering performance with good generalization.They say fast rendering should facilitate deployment and hope HuGS serves as a baseline for follow-up research on Gaussian-based avatars.Source: §6 · p. 8

Supplementary Material

The supplementary material presents further analysis, implementation details, and additional THuman4 results. It also directs readers to a video demonstrating real-time rendering.

Supplementary Material

7. Real-time video rendering

The supplementary video demonstrates real-time novel-pose synthesis on THuman4. It uses a viewer extended from 3D-GS for dynamic-scene video rendering.

7. Real-time video rendering

  • The supplementary video demonstrates real-time novel-pose synthesis on the THuman4 dataset.Source: §7 · p. 12
  • The video runs at 60 fps.Source: §7 · p. 12
  • Dynamic-scene rendering is implemented by extending the 3D-GS viewer to render videos.Source: §7 · p. 12

8. Reproducibility details

The appendix supplements the main manuscript with pipeline hyperparameters and implementation details. It describes how HuGS applies LBS to canonical Gaussians and sets learning rates for learnable components.

8. Reproducibility details

  • HuGS applies LBS to canonical Gaussians, with per-Gaussian skinning weights ReLU-clipped and normalized to sum to 1.The weights are optimized by gradient descent; the implementation omits pose blend shapes and uses an MLP for pose-dependent deformations.Source: §8 · p. 12
  • The MLP and skinning weights use a constant learning rate of 1e−4, while latent codes use 2.5e−3.Learning rates for the original Gaussian parameters—position, orientation, scaling, colors, and opacity—are left unchanged from 3D-GS.Source: §8 · p. 12
  • Table 6 lists hyperparameter values for the pipeline.The appendix identifies it as the location of the pipeline hyperparameters but does not state their values in the supplied prose.Source: Table 6 · p. 12, §8 · p. 12

9. Concurrent works

The discussion situates HuGS among concurrent Gaussian-based avatars, noting shared use of forward LBS and neural local refinement alongside differences in refinement, skinning weights, and preprocessing. HuGS’s per-Gaussian latent code and shading components are identified as advantages over the approaches discussed.

9. Concurrent works

  • The authors describe Gaussian splatting as an active area and expect follow-up research to build on the growing set of proposals.They refer readers to Awesome 3D Gaussian Splatting for a broader list of related work.Source: §9 · p. 12
  • Most concurrent Gaussian avatars use forward LBS deformation, and neural-network local refinement is also common.The cited refinement designs include canonical Gaussian inputs in SplatArmor [12] and 2D CNNs in Animatable Gaussians [27] and ASH [39].Source: §9 · p. 12
  • HuGS’s per-Gaussian latent code and shading components are identified as its main advantages over these Gaussian-avatar approaches.GART [19] also optimizes skinning weights, but in a voxel grid rather than per Gaussian as in HuGS.Source: §9 · p. 12
  • Most compared methods use template-based skinning weights, while Animatable Gaussians [27] and ASH [39] import a fitted subject template.The discussion says this design adds an expensive preprocessing step and appears to yield very good results.Source: §9 · p. 12

10. Qualitative analysis

HuGS produces detailed, high-fidelity novel-pose renderings for the shown THuman4 subjects, while training-image segmentation errors and motion blur can leave artifacts. These subjects are presented qualitatively because the dataset evaluation setup was unavailable.

10. Qualitative analysis

  • No quantitative comparison is reported for these subjects because the THuman4 evaluation setup was not released.Figure 8 presents their novel-pose synthesis qualitatively.Source: §10 · p. 12, Figure 8 · p. 13
  • HuGS fits the shown THuman4 subjects with precise details and renders the target poses with high fidelity.Examples include a black hood button and shoes.Source: §10 · pp. 12–13, Figure 8 · p. 13
  • Segmentation-mask inaccuracies and motion blur in training images create artifacts that degrade rendering quality on these subjects.The authors describe these issues as recurring in the training images.Source: §10 · p. 13
Loading 2311.17113v2…