PaperlayerView exact revision on arXiv

Source-linked AI summary

ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering

Haokai Pang, Heming Zhu, Adam Kortylewski, Christian Theobalt, Marc Habermann

arXiv:2312.05941v2cs.CV

TL;DR

Dynamic, controllable human avatars remain difficult to render in real time, as Gaussian-based avatar methods are non-trivial to adapt from static scenes. ASH attaches animatable 3D Gaussians to a deformable mesh and learns their parameters in 2D texture space, outperforming state-of-the-art real-time methods and even state-of-the-art offline methods.

  • Problem

    Gaussian splatting is primarily demonstrated for static scenes, and adapting it to controllable, detailed, dynamic human avatars is non-trivial.

    Source: §3.1 · p. 4

  • Method

    ASH attaches 3D Gaussians to an animatable template mesh and learns their parameters in 2D texture space.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    ASH performs significantly better than state-of-the-art real-time methods on animatable human rendering and even better than state-of-the-art offline methods.

    Source: §5 · p. 8

  • Takeaways & Limitations

    ASH produces real-time photorealistic renderings from skeletal motion and unseen camera views, including motion-aware appearance on novel poses.

    Source: §1 · p. 2, §4.1 · pp. 5–6

    The setup assumes segmented multi-view video recorded in a studio with synchronized, calibrated cameras and skeletal pose annotations.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Real-time rendering of photorealistic and controllable human avatars stands as a cornerstone in Computer Vision and Graphics. While recent advances in neural implicit rendering have unlocked unprecedented photorealism for digital avatars, real-time performance has mostly been demonstrated for static scenes only. To address this, we propose ASH, an animatable Gaussian splatting approach for photorealistic rendering of dynamic humans in real-time. We parameterize the clothed human as animatable 3D Gaussians, which can be efficiently splatted into image space to generate the final rendering. However, naively learning the Gaussian parameters in 3D space poses a severe challenge in terms of compute. Instead, we attach the Gaussians onto a deformable character model, and learn their parameters in 2D texture space, which allows leveraging efficient 2D convolutional architectures that easily scale with the required number of Gaussians. We benchmark ASH with competing methods on pose-controllable avatars, demonstrating that our method outperforms existing real-time methods by a large margin and shows comparable or even better results than offline methods.

1. Introduction

Photorealistic, controllable human avatars are difficult to render in real time: neural methods can capture fine appearance but often require costly volume rendering, while Gaussian splatting was developed for static scenes. ASH attaches animatable Gaussians to a deformable template and learns their parameters in texture space, targeting real-time, high-quality rendering from multi-view video.

1. Introduction

  • Hybrid NeRF methods capture delicate appearance details but are unsuitable for real-time use because volume rendering requires intensive sampling and MLP evaluations.Source: §1 · pp. 1–2
  • Static-scene Gaussian splatting offers rendering quality and speed, but extending it to motion-dependent clothed humans with pose control remains challenging.Source: §1 · p. 2
  • ASH generates photorealistic, motion-dependent clothed-human renderings from skeletal motion and a virtual camera, using only multi-view videos for training supervision.Source: §1 · p. 2
  • ASH attaches dynamic Gaussian splats to a deformable human template and learns their parameters as a 2D texture-translation task for efficient, real-time avatar rendering.The paper frames this representation as avoiding 3D architectures that do not easily scale to the large number of Gaussians required.Source: §1 · p. 2

2. Related Work

Prior work balances rendering quality, speed, and pose control in different ways: mesh-based methods are efficient but can lack detail, while hybrid implicit methods can be slow. ASH uses animatable Gaussian splats to target real-time rendering quality comparable to or better than leading offline methods.

2. Related Work

  • Scene-representation advances accelerate NeRF rendering and 3D Gaussian splatting enables real-time static-scene rendering, but these approaches are not designed for dynamic clothed humans.Source: §2 · p. 2
  • Dynamic NeRF extensions often support replay of a captured sequence under novel views, rather than user-controlled, pose-dependent clothing appearance.Source: §2 · p. 2
  • Mesh-based methods provide skeletal control and rasterization integration, but coarse geometry proxies or template-mesh resolution can limit rendering quality.DDC supports novel poses and views with real-time photorealistic rendering, and ASH reports better rendering quality than DDC.Source: §2 · pp. 2–3
  • ASH reports significantly better rendering quality than DDC while retaining real-time capability, and quality on par with or superior to HDHumans while rendering in real time.HDHumans and Neural Actor are described as taking roughly 5 seconds per frame.Source: §2 · p. 3
  • Hybrid human NeRFs address mesh-resolution limits with implicit fields, but closely related methods Neural Actor and HDHumans are slow; Neural Actor also struggles with loose outfits.Source: §2 · p. 3

3. Method

ASH maps skeletal motion and camera input to real-time human renderings by representing a deformable character with animatable 3D Gaussian splats. It stores and predicts Gaussian parameters in the template’s 2D texture space, then trains the motion-aware decoders in two stages.

3.1. Problem Setting and Background

  • ASH takes skeletal motion and camera parameters as input, renders animatable Gaussians, and uses multi-view video for supervision; at inference it accepts arbitrary poses and virtual cameras.The method targets real-time rendering, reported at 29.64 fps.Source: §3.1 · p. 3, §3 · p. 3

3.2. Animatable Gaussian Splats

  • Each covered template texel stores a 3D Gaussian, letting ASH learn motion-to-Gaussian mappings in 2D texture space instead of directly modeling more than 20,000 splats in 3D.Dual Quaternion skinning poses the splats, while learned per-texel canonical offsets capture fine motion-dependent deformations.Source: §3.2 · p. 4
  • Motion-aware normal and position textures encode skeletal motion for 2D convolutional geometry and appearance networks that predict Gaussian parameters.The geometry network predicts offsets, scale, rotation, and opacity; the appearance network predicts spherical-harmonic appearance parameters.Source: §3.2 · pp. 4–5

3.3. Training Strategy

  • Training first warms up the motion-aware decoders because training them from scratch on long, varied sequences does not converge, then optimizes the full sequence with pixel L1 and SSIM losses.Source: §3.3 · p. 5

4. Results

The section evaluates ASH on held-out views and poses, compares it with competing methods, and ablates its design choices. Reported comparisons favor ASH, while the ablations examine rendering quality and computational cost.

4. Results

  • Evaluation uses DynaCap subjects with held-out camera views and test motions, plus two newly recorded sequences captured with 120 cameras at 25 fps.The DynaCap evaluation includes subjects wearing loose and tight apparel.Source: §4 · p. 5

4.1. Qualitative Results

  • ASH renders photorealistic held-out views with sharp wrinkles and view-dependent appearance, and handles substantially shifted poses with motion-aware appearances.The qualitative results also show generalization to loose apparel and clothing dynamics such as skirt swing.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2. Comparisons

  • Across novel-view and novel-pose tests, ASH outperforms DDC and prior methods on reported metrics, while matching or exceeding offline HDHumans in quality at real-time speed.For tight outfits in novel-pose synthesis, ASH achieves the highest PSNR and second-best LPIPS; for loose clothing, it outperforms other methods.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7

4.3. Ablations

  • Removing motion conditioning loses clothing dynamics, removing offsets adds blur, and 128 resolution harms perceptual metrics; 512 adds U-Net and rasterization cost without comparable gains.The 256 texture resolution is the reference setting; doubling it increases computational complexity enough to prevent real-time compatibility.Source: §4.3 · pp. 7–8, Table 3 · p. 7

5. Conclusion

ASH attaches Gaussian splats to a deformable mesh and learns them in 2D texture space, outperforming state-of-the-art real-time methods and even offline methods. Its current template mesh is not updated.

5. Conclusion

  • ASH is a real-time method for high-quality animated-human rendering, learned solely from multi-view videos.It attaches 3D Gaussian splats to a deformable mesh template and learns them in 2D texture space as an image-to-image translation task.Source: §5 · p. 8
  • ASH demonstrates significantly better quantitative and qualitative performance than state-of-the-art real-time methods, and even better performance than state-of-the-art offline methods.The method attaches Gaussian splats to a deformable mesh template and learns them in 2D texture space.Source: §5 · p. 8
  • ASH currently does not update the underlying deformable template mesh; improving mesh geometry with Gaussian splatting remains future work.Source: §5 · p. 8
  • ASH has compatible rendering quality with doubled texel resolution and sharper rendering than with halved texel resolution.Figure 5 compares design choices on the image synthesis task.Source: Figure 5 · p. 9

A. Overview

The appendix collects further implementation details, ablations, results, runtime analysis, applications, and discussion of limitations and future directions.

A. Overview

  • The appendix provides further implementation details and ablative studies.Source: A. Overview · p. 13
  • It also includes results with driven poses from novel datasets and runtime analysis of major components.Source: A. Overview · p. 13
  • The appendix describes real-time applications built upon ASH and gives a more detailed discussion of limitations and future directions.Source: A. Overview · p. 13

B. Implementation Details

The implementation deforms a template mesh through an embedded graph and uses motion-aware 2D convolutional networks to predict Gaussian geometry and appearance parameters. The appendix also identifies the ablation table's evaluation scope.

B. Implementation Details

  • Embedded-graph parameters and per-vertex displacements derived from skeletal motion deform the template mesh, with graph connectivity and weights linking mesh vertices to graph nodes.The deformation includes node rotations and translations as well as per-vertex displacement for finer geometry details.Source: B. Implementation Details · p. 13
  • Motion-aware 2D U-Nets predict Gaussian geometry and appearance parameters from motion-aware textures, with global appearance features added to account for studio lighting variations.The global feature is derived from positional-encoded skeleton root translation using a shallow MLP.Source: B. Implementation Details · p. 13
  • Table 1 assesses design choices on image synthesis for a DynaCap subject wearing loose outfits.Source: Table 1 · p. 13

C. Ablations

With SMPL body meshes despite large template-to-surface deviations, ASH produces plausible results and significantly better quantitative performance than state-of-the-art real-time methods. Ablations also examine sparse camera views and direct 3D Gaussian prediction.

C. Ablations

  • ASH still accurately synthesizes animatable characters when trained with evenly distributed views from 12, 30, or 60 cameras.Figure 1 illustrates robustness to these sparser camera-view settings.Source: C. Ablations · p. 13, Figure 1 · p. 14
  • Directly predicting Gaussian parameters from 3D produces blurry rendering and fails to preserve motion-dependent wrinkle details, whereas 2D texel-space learning delivers detailed rendering.The direct-prediction ablation uses an 8-layer MLP with hidden-layer width 256.Source: C. Ablations · p. 13, Figure 1 · p. 14
  • With SMPL body meshes despite large deviations from the real surface, ASH produces visually plausible results and significantly better quantitative performance than state-of-the-art real-time methods.The comparison is reported in Table 2, and Figure 2 shows the qualitative results.Source: C. Ablations · p. 14, Table 2 · p. 14, Figure 2 · p. 14

D. More Results

The section reports evaluation on the DynaCap test set and illustrates rendering for motion from AMASS DanceDB, alongside an ASH runtime analysis.

D. More Results

  • DynaCap’s test set contains more than 7,000 frames with strongly varying poses.The section describes it as an established and challenging benchmark.Source: D. More Results · p. 14
  • ASH generates photorealistic renderings when driven by motion from the separate AMASS DanceDB dataset.The result is illustrated in Figure 3.Source: Figure 3 · p. 14
  • Table 3 reports runtime in milliseconds for ASH’s major components and compares halved and doubled texel resolutions with the standard model.Its caption notes that ASH renders high-quality animatable humans at a real-time frame rate.Source: Table 3 · p. 14

E. Runtime Analysis

The runtime analysis breaks ASH into four rendering stages and compares texel resolutions, showing a quality–speed trade-off while retaining real-time rendering at the standard resolution.

E. Runtime Analysis

  • The analysis measures a 1K image on one Nvidia Tesla A100 across three texture resolutions and four pipeline stages.The stages cover mesh creation, motion-aware texture computation, Gaussian prediction, and tile-based rasterization.Source: E. Runtime Analysis · p. 14
  • A one-to-one mapping between skeletal motion and dynamic clothed-human appearance is infeasible, motivating future fine-grained controls such as external physical forces.Source: E. Runtime Analysis · p. 15
  • ASH Player streams real-time renderings computed on a GPU cluster server to a browser interface on a personal computer.Users specify skeletal poses and virtual camera views, and can inspect characters with spiral camera views.Source: Figure 4 · p. 15
  • Doubling texel resolution substantially increases computational complexity and prevents real-time compatibility, while ASH produces high-fidelity real-time renderings.Halving resolution speeds synthesis but may blur rendered details; doubled resolution gives comparable quality.Source: E. Runtime Analysis · pp. 14–15

F. Application

ASH Player is a browser-based application for viewing ASH’s real-time renderings of animatable characters.

F. Application

  • ASH Player is a real-time application built on ASH.Source: F. Application · p. 15
  • The browser interface runs on a personal computer while ASH computes renderings on a GPU cluster server and streams them to the user.Users specify skeletal poses and virtual camera views.Source: F. Application · p. 15
  • Users can inspect animatable characters using spiral camera views.Source: F. Application · p. 15

G. Limitations

ASH has limitations in explicit geometry, topology changes, and representing motion-dependent clothing appearance.

G. Limitations

  • ASH does not extract detailed explicit geometry from its Gaussian splats; refining the template mesh through image-space gradients is proposed for future work.Source: G. Limitations · p. 15
  • ASH does not model topological changes such as opening a jacket, with adaptive Gaussian addition and removal suggested as future work.The proposed direction refers to the original 3D Gaussian splatting paper [26].Source: G. Limitations · p. 15
  • A one-to-one correspondence between skeletal motion and dynamic clothed-human appearance is infeasible; future work may explore finer controls such as external physical forces.Source: G. Limitations · p. 15
Loading 2312.05941v2…