Source-linked AI summary
Drivable 3D Gaussian Avatars
Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, Javier Romero
TL;DR
Controllable human avatars must represent body regions and garments driven by different signals while avoiding the limitations of single-layer or densely conditioned systems. D3GA embeds 3D Gaussians in tetrahedral cages and composes separate body, face, and garment layers with localized conditioning. The resulting avatars achieve high-quality results against comparable state-of-the-art methods while remaining lightweight, real-time, and compact.
Problem
Existing avatar methods struggle with single-layer garment modeling, limited layered conditioning, and dense test-time inputs for holistic control.
Method
D3GA combines 3D Gaussian primitives with tetrahedral cage deformations and separate body, face, and garment cages driven by localized signals.
Results
D3GA achieves high-quality results that surpass state-of-the-art methods with similar model architectures while maintaining a lightweight, real-time, compact representation.
Takeaways & Limitations
The compositional model supports localized conditioning such as facial keypoints and can extend to regions including hair, hands, and shoes.
Takeaways & Limitations
Single-layer training struggles to model proper garment sliding, while high-frequency patterns can blur and loose-garment self-collisions remain challenging.
Abstract
from arXiv · showhide
We present Drivable 3D Gaussian Avatars (D3GA), a multi-layered 3D controllable model for human bodies that utilizes 3D Gaussian primitives embedded into tetrahedral cages. The advantage of using cages compared to commonly employed linear blend skinning (LBS) is that primitives like 3D Gaussians are naturally re-oriented and their kernels are stretched via the deformation gradients of the encapsulating tetrahedron. Additional offsets are modeled for the tetrahedron vertices, effectively decoupling the low-dimensional driving poses from the extensive set of primitives to be rendered. This separation is achieved through the localized influence of each tetrahedron on 3D Gaussians, resulting in improved optimization. Using the cage-based deformation model, we introduce a compositional pipeline that decomposes an avatar into layers, such as garments, hands, or faces, improving the modeling of phenomena like garment sliding. These parts can be conditioned on different driving signals, such as keypoints for facial expressions or joint-angle vectors for garments and the body. Our experiments on two multi-view datasets with varied body shapes, clothes, and motions show higher-quality results. They surpass PSNR and SSIM metrics of other SOTA methods using the same data while offering greater flexibility and compactness.
1. Introduction
D3GA addresses the difficulty of modeling avatars whose body regions and garments respond to distinct signals. It combines tetrahedral cage deformation with compositional layers and localized conditioning to support flexible, compact avatar control.
- Distinct facial, body, and garment motions make single-layer avatar modeling difficult, motivating separate layers for different regions.Garment sliding is a specific challenge requiring separate clothing-piece modeling.
- Existing methods can struggle with imprecise meshes, fixed primitive counts, limited layered conditioning, and dense test-time inputs.These constraints reduce flexibility for holistic avatars that combine speech, facial expression, gestures, and garment motion.
- D3GA embeds 3D Gaussians in tetrahedral cages, whose deformation gradients transform Gaussian covariance matrices during motion.The cage representation models stretching, rotation, and scaling while avoiding reliance on LBS alone.
- Cage-based deformation decouples representation resolution from model degrees of freedom, enabling localized and regularized deformation control.Each cage locally influences its Gaussians, separating low-dimensional driving poses from the rendered primitive set.
- The models are trained on nine high-quality multi-view sequences covering varied body shapes, motions, and clothing, including loose-fitting garments.The trained avatars can later be driven with new poses from any subject.
- D3GA uses separate body, face, and garment cages with localized conditioning, including facial-expression control.The model is designed to use minimal inputs while remaining competitive with methods requiring more information.
2. Related Work
Prior controllable-avatar methods use neural, point-based, hybrid, or time-conditioned representations, but face rendering speed, garment disentanglement, pose generalization, or arbitrary control limitations. D3GA builds on 3D Gaussian Splatting and cage-based deformation to target these gaps.
- Dynamic Neural Radiance Fields: Dynamic NeRF avatar methods can produce high-quality results but often treat avatars as single layers, complicating sliding and loose-garment modeling.Hybrid methods improve garment reconstruction but can struggle with novel-pose prediction.
- Point-based Rendering: Point-based methods may incur lengthy nearest-neighbor searches during training, limiting practicality for dense multi-view datasets.NPC is reported to require 12 hours compared with 30 minutes for D3GA.
- Cage-based Deformations: Cage-based deformation uses sparse proxy nodes to control interior points efficiently, supporting deformation through manipulation of the cage.Prior tetrahedral-cage work is described as real-time, high-quality, and controllable but limited to objects with local deformations.
- Time-conditioned Methods: Time-conditioned methods support novel-view synthesis while traversing time but cannot be arbitrarily controlled.They therefore belong to a different class from pose-conditioned avatar methods.
- Dynamic Gaussian Splatting: D3GA extends 3D Gaussian Splatting, which offers real-time capabilities and high-quality results, to dynamic avatar modeling.The related work positions 3DGS as an alternative to NeRF for neural-scene modeling.
3. Method
D3GA combines 3D Gaussian Splatting with tetrahedral cages to model dynamic avatar parts, transferring cage deformations to Gaussian geometry and appearance. Separate networks and layered conditioning support controllable garments, bodies, and faces.
- Core representation: D3GA extends 3D Gaussian Splatting with tetrahedral cages that model the geometry and color of each dynamic avatar part.The framework uses cages for deformation and neural representations for appearance.
- Cage construction: Cages are created from segmented body and garment regions, using a fitted low-resolution LBS model to obtain canonical part templates.Image segmentation is projected onto a fitted body mesh, with garment meshes additionally inflated.
- Rendering and training: D3GA renders color and segmentation outputs from transformed Gaussians and trains with weighted color losses, segmentation supervision, and tetrahedral regularization.The primary color objective combines L1, D-SSIM, and VGG losses; garment separation uses a part-rendering loss, with Neo-Hookean regularization in weakly supervised regions.
- Cage deformation transfer: Gaussian means are represented with barycentric coordinates inside tetrahedra, while deformation gradients transform their covariance matrices between canonical and posed spaces.This transfers tetrahedral rotation, stretching, and shearing to Gaussian kernels, including effects such as garment stretching.
- Part-specific conditioning: Each avatar part uses specialized MLPs for cage-node offsets, Gaussian corrections, and shading, conditioned on joint angles or face embeddings.The shading network predicts pose-dependent color and opacity, while face embeddings can be regressed from 150 normalized 3D keypoints.
4. Dataset
The dataset contains diverse multi-view human motion data, with separate training and testing splits. Qualitative comparisons emphasize facial-expression and garment modeling, especially for loose clothing.
- The dataset includes nine subjects performing various motions, observed by 200 cameras.
- Training uses 12,000 frames at 10 FPS, while testing uses 1,500 frames at 30 FPS.
- Images were captured at 4096×2668 resolution and downsampled to 1024 × 667 for computational efficiency.
- Qualitative comparisons focus on facial expressions and garments, particularly loose regions such as skirts and sweatpants.
- Training uses segmentation masks, RGB images, keypoints, 3D joint angles, and a single registered mesh template.
5. Results
D3GA achieves strong image quality and compactness while supporting layered avatar control. Its ablations show that tetrahedral deformation and garment-specific modeling are important for preserving sharpness and reducing sliding artifacts.
- Model Size: D3GA uses up to 10 million parameters, whereas CNN-based MVP, BD, and AG use approximately 230 million.
- Image Quality: D3GA achieves the best PSNR and SSIM on the authors’ dataset compared with MVP and BD.
- Image Quality: On ActorsHQ, D3GA outperforms other Gaussian Avatar methods in PSNR and SSIM but is less sharp than AG.
- Ablation Studies: Triangle-based cages fail to stretch Gaussian primitives correctly, producing holes and artifacts that demonstrate the importance of cage deformation.
- Ablation Studies: The garment loss improves garment separation and reduces translucent regions, while single-layer training produces over-smoothed garment boundaries and sliding artifacts.
- Model Flexibility: D3GA offers a favorable quality–size tradeoff while enabling motion transfer and preserving avatar details.
6. Discussion
D3GA remains lightweight and competitive, but challenging garment appearance and deformation cases expose limitations in its current control and appearance models.
- Image Quality: High-frequency patterns such as stripes may produce blurry regions.
- Garment Modeling: Loose-garment self-collisions remain challenging despite the garment loss, while sparse control signals cannot capture complex wrinkles or self-shadowing.
- Scope: D3GA currently models photorealistic avatars only for a few consenting subjects captured with a dense multi-view device.
7. Conclusion
The paper presents D3GA as a lightweight, real-time framework for multi-layered animatable avatars. Its cage-based deformation and compositional conditioning support high-quality control across different avatar regions and input signals.
- Method: D3GA reconstructs multi-layered animatable avatars using 3D Gaussians embedded in tetrahedral cages.
- Method: Applying deformation gradients to Gaussian parameters improves transformation from canonical to deformed space.
- Compositional Control: Localized conditioning supports signals such as facial keypoints and can extend to hair, hands, and shoes.
- Conclusion: The method reports high-quality results while remaining lightweight, real-time, and compact.
– Supplemental Document –
D3GA supports motion transfer while preserving the detailed appearance of each avatar.
- Motion transfer demonstrates good generalizability while preserving each avatar’s high-quality details.
A. Appendix
The appendix examines compositional control, regularization, corrective networks, failure cases, and the speed–quality trade-off in D3GA.
- Compositionality: D3GA independently models avatar segments, allowing each layer to receive specialized conditioning and be controlled separately.Examples include decomposed garment parts and conditioning through expression codes, keypoints, or motion vectors.
- Regularization Effects: LNeo regularization reduces geometric artifacts by preventing tetrahedra from losing volume, flipping, or diverging from their canonical size.It is especially useful in garment regions that are permanently or temporarily covered.
- Corrective Networks: Tetrahedral cages and MLP-based correctives improve Gaussian orientation compared with using LBS alone, which can produce test-time artifacts.The ablation attributes the artifacts to incorrect 3D Gaussian orientation.
- Failure Cases: Pose-conditioned methods commonly struggle with long garments when primitives lack a specialized garment template.MLP-based methods are more stable with a coarse SMPL mesh, whereas AG fails without its specialized template.
- Efficiency: 100k and 200k primitives provide the best compromise between rendering speed and quality.Table 3 reports that 100k Gaussians provide the best rendering-time-to-quality ratio at 1024 × 667 resolution.
B. Cage Deformation Gradient
The cage deformation gradient approximates volumetric deformation within tetrahedra and applies the resulting transformation to the Gaussian kernels.
- Deformation Gradient: The deformation gradient specifies how an infinitesimal vector changes under the solid’s deformation.It is the Jacobian of the mapping from canonical to deformed space.
- Discretization: For general shapes, the method approximates the Jacobian by discretizing the shape with tetrahedra.This replaces a continuous deformation field with a piecewise representation.
- Tetrahedral Mapping: For each cage tetrahedron, the deformation gradient maps its three edge vectors from canonical coordinates to their deformed counterparts.The mapping is defined using the four tetrahedron vertices and their corresponding deformed positions.
- Kernel Transformation: The resulting deformation gradient is applied to each encapsulated Gaussian primitive to transform its kernel between canonical and deformed tetrahedra.The formulation uses 3 × 3 matrices associated with the canonical and deformed tetrahedra.
C. Broader Impact
The appendix documents broader practical considerations, including synthetic-media risks, dynamic-sequence limitations, garment failures, and layer-specific corrections.
- Broader Impact: The authors recognize that reconstructing controllable avatars for telepresence or mixed reality can be misused and advocate stronger digital-media forensics.The stated goal is to aid detection of synthetic media.
- Garment Decomposition: Single-layer avatars incorrectly model sliding garments, whereas garment loss improves separation between layers.This comparison is reported in the evaluation table for the authors’ dataset.
- Garment Failure Cases: Long garments remain a common failure case for methods without specialized garment templates.AG performs best with a tracked garment template but fails without one; MLP-based methods are more stable but remain incorrect.
- Geometric Corrections: Neo-Hookean regularization is most visible in poorly supervised or sliding regions, where it helps prevent tetrahedral deformation artifacts.Geometry corrective fields alter the input tetrahedra before pose correction, while garment supervision reduces boundary semitransparency.
- Layered Composition: Independent avatar components can be arranged and manipulated as separate layers, including upper and lower garment segments.The layer arrangement is not restricted to garments.