PaperlayerView exact revision on arXiv

Source-linked AI summary

3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting

Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, Siyu Tang

arXiv:2312.09228v3cs.CV

TL;DR

Existing avatar methods struggle to combine high-quality monocular reconstruction with both efficient training and real-time inference. 3DGS-Avatar uses deformable 3D Gaussian Splatting with geometric regularization to model animatable clothed humans. It reports comparable or better rendering quality, 400× faster training, and 250× faster inference, while acknowledging limitations in training speed, fine textures, and geometry quality.

  • Problem

    High-quality NeRF-based avatars often require days of training and slow inference, while fast-training alternatives provide only limited rendering speed or do not jointly satisfy both goals.

  • Method

    The method optimizes canonical 3D Gaussians, deforms them using rigid articulation and a learned non-rigid deformation field, and regularizes Gaussian means and covariances geometrically.

  • Results

    The method achieves rendering quality comparable to or better than state of the art, with training 400× faster and inference 250× faster.

  • Takeaways & Limitations

    3DGS-Avatar combines photorealistic rendering, pose-dependent cloth deformation, unseen-pose generalization, fast training, and real-time rendering in one monocular-avatar system.

  • Takeaways & Limitations

    Training remains slower than fast grid-based methods; high-frequency or repetitive textures may blur, and geometry reconstruction and surface normals are inaccurate.

Abstract

from arXiv · show

We introduce an approach that creates animatable human avatars from monocular videos using 3D Gaussian Splatting (3DGS). Existing methods based on neural radiance fields (NeRFs) achieve high-quality novel-view/novel-pose image synthesis but often require days of training, and are extremely slow at inference time. Recently, the community has explored fast grid structures for efficient training of clothed avatars. Albeit being extremely fast at training, these methods can barely achieve an interactive rendering frame rate with around 15 FPS. In this paper, we use 3D Gaussian Splatting and learn a non-rigid deformation network to reconstruct animatable clothed human avatars that can be trained within 30 minutes and rendered at real-time frame rates (50+ FPS). Given the explicit nature of our representation, we further introduce as-isometric-as-possible regularizations on both the Gaussian mean vectors and the covariance matrices, enhancing the generalization of our model on highly articulated unseen poses. Experimental results show that our method achieves comparable and even better performance compared to state-of-the-art approaches on animatable avatar creation from a monocular input, while being 400x and 250x faster in training and inference, respectively.

1. Introduction

Animatable clothed-avatar reconstruction from monocular video remains difficult because high-quality methods are computationally costly, while fast methods do not simultaneously provide real-time inference. 3DGS-Avatar addresses this gap with explicit Gaussian representations, deformation modeling, and geometric regularization.

  • NeRF-based avatar methods achieve high-quality rendering but often require days of training and seconds for inference.
  • No prior method simultaneously provides fast training and real-time inference for animatable avatars from monocular videos.
  • 3DGS-Avatar combines rigid articulation, non-rigid deformation, and a color MLP within a 3DGS framework.
  • As-isometric-as-possible regularization on Gaussian means and covariances promotes geometric consistency across dynamic and varied poses.
  • 400× faster training and 250× faster inference are achieved while matching or surpassing state-of-the-art monocular avatar methods.
  • The method targets less than 30 minutes of training and real-time rendering above 50 FPS while supporting unseen articulated poses.

2. Related Works

Related work has accelerated either training or inference, but existing avatar methods retain limitations in deformation modeling, input requirements, generalization, or speed. The paper positions its approach as combining high-quality rendering with fast monocular-avatar reconstruction.

  • NeRF avatar methods condition fields on body encodings or warp rays from observation space into a canonical representation.
  • InstantAvatar enables training in under 5 minutes and 15 FPS rendering but ignores pose-dependent non-rigid deformation.
  • Some real-time dynamic-human methods require dense multi-view inputs, while Fourier representations do not generalize beyond their training poses.
  • Dynamic 3DGS methods use frame-specific optimization or time-dependent deformation fields, but can incur model-growth or pipeline-complexity issues.
  • Pose-dependent Gaussian deformation methods have required 2 hours of training and lacked monocular-input results, while other methods neglect cloth deformation.

3. Preliminaries

The preliminaries describe canonical-space human articulation through Linear Blend Skinning and image formation through depth-sorted, alpha-blended 3D Gaussian splats. The representation is optimized photometrically with adaptive Gaussian densification and pruning.

  • Linear Blend Skinning: Linear Blend Skinning maps a canonical point to observation space using learned skinning weights and rigid bone transformations.
  • Linear Blend Skinning: The formulation uses 24 SMPL bone transformations, each represented by a 4 × 4 rotation-translation matrix.
  • Linear Blend Skinning: Unlike inverse-skinning approaches, the method directly computes observation-space correspondences from canonical points, avoiding difficult and potentially ambiguous inversion.
  • 3D Gaussian Splatting: Each 3D Gaussian is represented by a mean, covariance, opacity, and view-dependent color.
  • 3D Gaussian Splatting: Projected Gaussians are depth-sorted and accumulated through alpha blending to compute pixel colors.
  • 3D Gaussian Splatting: Photometric optimization uses periodic densification and pruning to adapt the number of Gaussians toward a suitable scene density.

4. Methods

The method represents an avatar with canonical 3D Gaussians, deforms them through pose-conditioned non-rigid and skeleton-controlled rigid transformations, and renders their learned appearance. It further canonicalizes viewing direction, models deformation-dependent color, and regularizes positional and covariance changes during optimization.

  • Pose-dependent Non-rigid Deformation: 50k canonical 3D Gaussians sampled from an SMPL mesh are decomposed into pose-dependent non-rigid deformation and skeleton-controlled rigid transformation.Each Gaussian stores position, scale, rotation, opacity, and a color feature vector.
  • Pose-dependent Non-rigid Deformation: The non-rigid deformation network takes canonical position and a pose-and-shape latent code, outputting position, scale, rotation offsets, and a latent feature.A two-hidden-layer MLP with width 128 uses multi-level hash-grid spatial features concatenated with the pose latent code.
  • Rigid Transformation: Learned neural skinning transforms deformed Gaussians into observation space through forward linear blend skinning using predicted skinning weights.The position and rotation matrix are transformed using the learned weights and bone transformations.
  • Color MLP: Canonicalized viewing directions, local deformation features, per-frame latent codes, and Gaussian features are fed to a small MLP that predicts view-dependent color.A one-layer 64-dimensional hidden MLP was sufficient; larger networks caused overfitting and performance degradation.
  • Optimization: Optimization jointly updates canonical Gaussians and deformation, skinning, and color networks while combining RGB, mask, skinning, and as-isometric-as-possible losses.The isometric regularization constrains both Gaussian positions and covariance matrices, using five-neighbor neighborhoods and L2 distances.

5. Experiments

Experiments show that the method delivers strong monocular avatar rendering quality while combining fast training, real-time inference, and improved generalization to unseen poses.

  • Comparison with Baselines: 400x faster training and 250x faster inference than the most competitive baseline, while achieving fast training and real-time rendering.The reported comparison is 30 GPU minutes versus 8 GPU days for training, and 50 FPS versus 0.1 FPS for ARAH or 0.2 FPS for HumanNeRF.
  • Comparison with Baselines: The method matches ARAH on PSNR and SSIM and significantly outperforms all baselines on LPIPS on ZJU-MoCap.LPIPS is emphasized as more informative for monocular novel-view evaluation because exact ground-truth appearance is difficult to reproduce with stochastic cloth deformation.
  • Qualitative Comparison: The method preserves sharper details than ARAH, avoids HumanNeRF's fluctuating artifacts, and generalizes to extreme out-of-distribution poses.Instant-NVR is described as oversmooth with noisy limbs, while AMASS and AIST++ pose sequences are used for qualitative animation.
  • Comparison with Baselines: On PeopleSnapshot, the approach significantly outperforms InstantAvatar on PSNR and LPIPS while rendering more than 3x faster.
  • Ablation Study: All proposed components are required for optimal performance, with the effect best reflected by LPIPS in monocular novel-view evaluation.The ablation studies include the color MLP, as-isometric-as-possible regularization, and pose correction module.
  • Ablation Study: As-isometric-as-possible regularization constrains Gaussian movement consistently during deformation, improving generalization on novel poses and producing more plausible animation.

6. Conclusion

The conclusion presents 3DGS-Avatar as an efficient approach for reconstructing clothed human avatars from monocular videos. It combines photorealistic rendering, pose-dependent deformation, unseen-pose generalization, fast training, and real-time rendering, with experiments showing comparable or better quality and two-orders-of-magnitude speedups.

  • The method combines photorealistic rendering, pose-dependent cloth deformation, unseen-pose generalization, fast training, and real-time rendering.
  • Two orders of magnitude faster training and inference accompany rendering quality comparable to or better than state-of-the-art methods.The paper also identifies a shallow MLP for Gaussian color decoding and geometric deformation constraints as effective for enhancing rendering quality.

Supplementary Material

The supplementary material defines the training objectives, combining reconstruction, mask, skinning, and as-isometric-as-possible regularization losses. It also describes whole-image rasterization and explicit foreground supervision for the monocular setting.

  • RGB Loss: RGB supervision uses pixel-wise L1 and VGG-based LPIPS perceptual loss to tolerate local misalignments.The method renders the whole image by rasterization rather than sampling random NeRF rays.
  • Mask Loss: Mask supervision compares rendered opacity with the ground-truth foreground mask using L1 loss.The opacity is computed by summing sample weights in the rendering equation, and L1 converges faster than BCE in experiments.
  • Skinning Loss: Skinning regularization samples 1024 canonical SMPL-surface points and matches predicted weights to barycentrically interpolated weights.The corresponding loss expression regularizes the forward skinning network toward the target weights.
  • As-isometric-as-possible Loss: The as-isometric-as-possible loss regularizes Gaussian deformation, with λskin scheduled from 10 during the first 1k iterations to 0.1 afterward.The schedule encourages fast initial convergence followed by softer skinning regularization.

B. Implementation Details

Implementation initializes and optimizes canonical Gaussians with learned skinning, non-rigid deformation, and color components. Training uses staged optimization, pose and view augmentation, and dataset-specific iteration budgets.

  • Gaussian initialization: Canonical Gaussians start from 50k random samples on the canonical SMPL surface and are densified or pruned using view-space position gradients.The gradients are computed from transformed Gaussians in observation space.
  • Network architectures: The forward skinning network maps canonical 3D coordinates to hierarchical-softmax skinning weights that sum to 1.It uses a four-hidden-layer MLP and outputs a 25-dimensional vector before hierarchical normalization.
  • Network architectures: The non-rigid network combines multi-level hash-grid features with a pose latent code to decode pose-dependent local deformation.A shallow MLP with three hidden layers performs the decoding.
  • Data augmentation: Training noise perturbs SMPL poses with N(0, 0.1) at probability 0.5 and augments viewing directions with random rotations in [0, 45).The stated purpose is reducing overfitting and improving novel-pose generalization.
  • Optimization: Training takes 30 minutes for 15k iterations on ZJU-MoCap and 45 minutes for 30k iterations on PeopleSnapshot using one RTX 3090.Adam is used for the model and per-frame latent codes.
  • Optimization: Coarse-to-fine optimization first trains only the forward skinning network, then enables Gaussian and deformation optimization in stages.The initial 1k iterations prevent noisy gradients from moving Gaussians away from initialization.

C. Implementation Details for Baselines

The supplementary material documents baseline implementations, evaluation alignment, and component ablations. These studies examine color modeling, pose correction, AIAP regularization, and mask supervision under the reported evaluation setup.

  • Evaluation setup: Baseline evaluation uses official repositories, matched data splits where possible, and reduced 512 × 512 images for the HumanNeRF setup.Different split conventions are explicitly corrected for MonoHuman, while Anim-NeRF uses its original optimized poses.
  • Ablation design: The additional ablation study reports average metrics over six ZJU-MoCap sequences.The study also evaluates network-free variants of color, non-rigid deformation, and skinning components.
  • Color MLP: The learned color MLP produces higher-quality rendering than spherical harmonics, including more realistic cloth wrinkles and sharper textures.It uses pose-dependent features and per-frame latent codes as additional inputs.
  • Pose correction: Pose correction refines inaccurate SMPL fittings by optimizing sequence shape and per-frame motion parameters, improving avatar modeling quality.Figure 6 reports better alignment between the SMPL mesh and ground-truth image after pose optimization.
  • AIAP regularization: AIAP regularization removes scattered Gaussian artifacts and stabilizes Gaussian position and shape on unseen poses.Its improvement on novel-view synthesis benchmarks is described as marginal.
  • Mask supervision: Mask loss yields only slight metric improvement but removes floating Gaussian blobs that can occlude the subject in novel views.Figure 8 illustrates the floating-background-color Gaussian without mask loss.

D.6. Ablation on Gaussian Initialization

The initialization ablation tests random Gaussian placement against SMPL-surface initialization. Random placement can still converge with little performance drop, while SMPL initialization remains the chosen default.

  • Gaussian initialization: Random initialization samples 50k points uniformly in the bounding box surrounding the canonical SMPL mesh.This is compared with initialization from the canonical SMPL mesh surface.
  • Gaussian initialization: Random initialization converges with little performance drop compared with SMPL initialization.The authors retain SMPL initialization because it is more intuitive and adds no overhead.
  • Training efficiency: The method reaches high-quality results after 7k iterations, requiring around 10 minutes of training.The full 15k-iteration ZJU-MoCap training run takes around 24 minutes.

E.1. Qualitative Results of Novel View Synthesis on ZJU-MoCap

The qualitative comparisons show that 3DGS-Avatar preserves detail, avoids common artifacts, and produces realistic cloth deformations across novel views and unseen poses.

  • The method generates high-quality novel-view images with realistic cloth deformations, while baselines exhibit distortions, blur, or limb artifacts.
  • 3DGS-Avatar produces no obvious artifacts in extreme out-of-distribution pose animation, demonstrating generalization to unseen poses.
  • On PeopleSnapshot, 3DGS-Avatar produces sharper results than InstantAvatar, especially in the face region.

F. Limitations

The method has practical advantages in rendering quality and speed but remains limited in training efficiency, high-frequency texture rendering, and geometry reconstruction.

  • Training remains slower than fast grid-based methods despite state-of-the-art rendering quality and interactive rendering.
  • High-frequency textures or repetitive patterns, such as striped shirts, may produce blurry results.
  • The approach does not provide accurate avatar geometry reconstruction and produces noisy surface normals from inconsistent Gaussian-splat depth.
Loading 2312.09228v3…