Source-linked AI summary
GART: Gaussian Articulated Template Models
Jiahui Lei, Yufu Wang, Georgios Pavlakos, Lingjie Liu, Kostas Daniilidis
TL;DR
Monocular reconstruction of articulated humans and animals needs representations that preserve template structure while modeling detailed, non-rigid appearance efficiently. GART uses canonical-space Gaussian mixtures with learnable forward skinning, latent bones, and smoothness priors, achieving efficient high-quality reconstruction and rendering. The method reports state-of-the-art performance across human and animal settings, but depends on an available template pose estimator and currently fits single monocular videos rather than category-level video collections.
Problem
Template models cannot capture detailed geometry and appearance across deformations, while neural representations are slow to render and difficult to animate.
Method
GART explicitly approximates canonical geometry and appearance with 3D Gaussian mixtures, using learnable forward skinning, latent bones, and smoothness priors.
Results
GART achieves state-of-the-art monocular human reconstruction and rendering with high training and inference efficiency, while producing high-quality animal reconstructions.
Takeaways & Limitations
GART provides a general explicit representation that captures challenging non-rigid articulated deformations while supporting efficient reconstruction and rendering.
Takeaways & Limitations
GART relies on a template pose estimator that may not exist for more general animal species and currently fits a single monocular video rather than learning category-level priors from video collections.
Abstract
from arXiv · showhide
We introduce Gaussian Articulated Template Model GART, an explicit, efficient, and expressive representation for non-rigid articulated subject capturing and rendering from monocular videos. GART utilizes a mixture of moving 3D Gaussians to explicitly approximate a deformable subject's geometry and appearance. It takes advantage of a categorical template model prior (SMPL, SMAL, etc.) with learnable forward skinning while further generalizing to more complex non-rigid deformations with novel latent bones. GART can be reconstructed via differentiable rendering from monocular videos in seconds or minutes and rendered in novel poses faster than 150fps.
1 University of Pennsylvania 2 UC Berkeley 3 Archimedes, Athena RC
GART is presented as an explicit, efficient, and expressive model for articulated object capture and rendering from monocular videos.
- GART models articulated objects for capture and rendering from monocular videos.
1. Introduction
The introduction identifies a gap between template models that lack detailed deformation and appearance and neural representations that are expressive but inefficient. GART addresses this gap with Gaussian mixtures, learnable skinning, latent bones, and smoothness priors for efficient monocular reconstruction and rendering.
- Categorical templates such as SMPL and SMAL provide structure but cannot capture detailed appearance and geometry across varied deformations.
- Neural-field extensions improve quality but suffer from slow rendering, difficult animation, and sensitivity to template pose errors.
- GART uses a canonical-space Gaussian mixture to explicitly approximate the radiance field while retaining template kinematics, simplicity, and interpretability.
- Learnable forward skinning, latent bones, and smoothness priors extend GART to loose-clothing motion, noisy poses, and sparse monocular observations.
- 30 seconds reconstructs a human avatar, while 540 × 540 rendering exceeds 150 FPS on a laptop.
- Experiments report state-of-the-art monocular human reconstruction and rendering with high training and inference efficiency, plus high-quality animal reconstruction in the wild.
2. Related Work
Related work combines template priors, neural representations, and differentiable rendering to reconstruct humans and animals, but these approaches face fidelity, generality, or efficiency limitations. GART builds on this landscape with explicit Gaussian-based rendering.
- 3D Human Reconstruction: Parametric human templates provide strong pose and shape priors but cannot represent personalized details such as texture, hair, and clothing.
- 3D Human Reconstruction: Hybrid neural representations enable high-fidelity dynamic-human modeling and re-animation, but their querying and rendering are inefficient.
- 3D Animal Reconstruction: Animal reconstruction methods use species-specific templates, image-guided deformation, or model-free sphere deformation, with limited fidelity or costly template creation.
- 3D Gaussian Splatting: 3D Gaussian Splatting uses differentiable splatting of Gaussians to approximate radiance fields with high efficiency and fidelity.
3. Method
GART represents articulated subjects with canonical-space Gaussian mixtures, then deforms them using learnable forward skinning, latent bones, and smoothness regularization before efficient Gaussian-splat rendering.
- Shape and appearance: GART replaces the template mesh with a canonical-space Gaussian mixture that explicitly represents subject geometry and appearance.Each Gaussian stores position, rotation, anisotropic scale, opacity, and view-dependent color parameters.
- Shape and appearance: Each Gaussian is independently movable rather than constrained by fixed mesh topology, increasing representation flexibility.This flexibility allows the mixture to approximate deformable subjects without a fixed-topology mesh.
- Forward skinning: GART uses learnable forward skinning to correct template weights and transform Gaussian centers and rotations from canonical to articulated space.The articulation combines template-bone transformations with learnable per-Gaussian skinning corrections.
- Latent bones: Learned latent bones and their skinning weights extend the predefined skeleton to capture deformations such as loose clothing.The latent-bone transformations are globally shared across Gaussians, keeping the deformation model compact and computationally efficient.
- Regularization: Smoothness regularization constrains skinning weights and Gaussian attributes when observations are sparse or poses are noisy.Additional regularizers encourage small deviations from the template motion structure and small Gaussian scales.
- Rendering: During inference, GART stores Gaussian attributes explicitly and renders articulated subjects at more than 150 FPS at 540 × 540 resolution.No voxel-grid query is needed during inference, and rendering is described as fast as rendering a static scene.
4. Experiments
Experiments evaluate GART for monocular human and animal reconstruction and view synthesis, emphasizing quality, efficiency, and deformation expressiveness. Across datasets and ablations, GART delivers fast training and rendering, handles challenging clothing and animal appearance, and benefits from latent bones and smoothness regularization.
- Human rendering: GART is evaluated with SMPL-based monocular RGB videos and estimated poses using PSNR, SSIM, and LPIPS for novel-view synthesis.The experiments target effectiveness, efficiency, and expressiveness in monocular human reconstruction and view synthesis.
- Human rendering: GART surpasses competing methods on ZJU-MoCap while achieving similar quantitative performance after less than 30 seconds of training.The authors attribute the efficiency to rendering and forward skinning, with qualitative results capturing more details than Instant-NVR.
- Human rendering: GART achieves comparable People-Snapshot performance with shorter training and renders at more than 150FPS versus Instant-Avatar’s 15FPS at 540 × 540.The comparison uses a single RTX-3080-Laptop GPU for the reported GART inference speed.
- Challenging clothing: On UBC-Fashion’s dynamic-dress sequences, GART captures challenging clothing that Instant-Avatar fails to model and avoids artifacts around highly deformed garments.The paper links this behavior to forward skinning, latent bones, and deformation-based optimization with 3D Gaussian Splatting.
- Animals: On in-the-wild dog sequences, GART captures different breeds and reconstructs breed-specific appearance such as tails and ears better than D-SMAL.The evaluation uses eight sequences selected for corrected pose estimates and limited occlusion.
- Ablation study: Ablations show that latent bones improve independent skirt motion, while removing KNN regularization or voxel-distilled skinning introduces strong or noisy artifacts.The full model performs best, and the version without latent bones remains more robust than Instant-Avatar under noisy poses and large deformations.
- Text-to-GART: Text-to-GART replaces image reconstruction losses with SDS loss and uses sampled AMASS poses, with typical generation taking around 10 minutes per subject on one GPU.The computation bottleneck is mainly 2D diffusion forwarding.
5. Conclusions
GART is a simple, general representation for non-rigid articulated subjects that combines Gaussian mixtures with novel skinning deformation. It achieves strong human and animal reconstruction and rendering performance with efficient training and inference, while relying on template pose estimators and fitting single monocular videos.
- GART represents non-rigid articulated subjects using Gaussian mixtures and novel skinning deformation.
- The method relies on a template pose estimator, which may not exist for more general animal species.
- GART currently fits a single monocular video efficiently, leaving category-level prior learning from in-the-wild video collections as future work.