Source-linked AI summary
TAVA: Template-free Animatable Volumetric Actors
Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhofer, Jurgen Gall, Angjoo Kanazawa, Christoph Lassner
TL;DR
Dynamic neural avatars must remain controllable at novel poses while retaining realistic appearance and editability, despite the limitations of template-based techniques and the limited exploration of deforming articulated actors. TAVA builds a template-free volumetric actor from multi-view data and a tracked skeleton using radiance fields and skeleton-driven deformation. Experiments report strong generalization to novel poses and unseen views, accurate dense correspondences, and applications to humans and animals, while the method has a documented root-finding failure rate and trains substantially slower than baselines.
Problem
Neural representations capture visual detail but remain underexplored for dynamic articulated actors that must generalize to unseen poses and remain editable.
Method
TAVA jointly optimizes a coordinate-based radiance field and skeleton-driven deformation model using multi-view photometric data and a tracked skeleton, without a body template.
Results
TAVA outperforms state-of-the-art approaches for animating and rendering human actors on ZJU-Mocap and is evaluated on synthetic animals, including editing applications.
Takeaways & Limitations
The template-free representation supports novel-pose animation, unseen-view rendering, dense cross-pose correspondences, and use with creatures beyond humans.
Takeaways & Limitations
Inverse root finding fails for 1% to 8% of view-space points, requiring density suppression or neighbor interpolation.
Abstract
from arXiv · showhide
Coordinate-based volumetric representations have the potential to generate photo-realistic virtual avatars from images. However, virtual avatars also need to be controllable even to a novel pose that may not have been observed. Traditional techniques, such as LBS, provide such a function; yet it usually requires a hand-designed body template, 3D scan data, and limited appearance models. On the other hand, neural representation has been shown to be powerful in representing visual details, but are under explored on deforming dynamic articulated actors. In this paper, we propose TAVA, a method to create T emplate-free Animatable Volumetric Actors, based on neural representations. We rely solely on multi-view data and a tracked skeleton to create a volumetric model of an actor, which can be animated at the test time given novel pose. Since TAVA does not require a body template, it is applicable to humans as well as other creatures such as animals. Furthermore, TAVA is designed such that it can recover accurate dense correspondences, making it amenable to content-creation and editing tasks. Through extensive experiments, we demonstrate that the proposed method generalizes well to novel poses as well as unseen views and showcase basic editing capabilities.
1 Introduction
TAVA addresses the challenge of building editable neural representations of articulated actors that can generalize beyond captured poses. It uses coordinate-based radiance fields and a skeleton-driven deformation model without requiring a body template.
- Motivation: Neural representations capture realistic appearance, but dynamic articulated actors require meaningful animation and generalization to out-of-distribution poses.The high-dimensional pose space makes capturing every pose impractical, while editing dynamic neural representations remains unclear.
- Approach: TAVA combines coordinate-based radiance fields with pose-dependent deformation anchored in forward LBS for stable generalization to unseen poses.The radiance field and deformation model are optimized jointly and end-to-end.
- Approach: TAVA requires only tracked skeleton and multi-view photometric data, without a template mesh or artist-designed rigging.This design supports both human and non-human articulated actors.
- Capabilities: The resulting representation supports rendering, posing, correspondence-based matching, content creation, and editing.TAVA establishes correspondences across poses through its canonical representation and deformation model.
- Results: Experiments report state-of-the-art performance for animating and rendering human actors on ZJU-Mocap, alongside evaluations on two synthetic animals.The animal experiments include evaluation, ablation, content-creation, and editing applications.
2 Related Work
Prior work represents deformable scenes and animatable actors using deformation fields, templates, meshes, or canonical spaces. TAVA’s design combines volumetric rendering with LBS-based deformation and a canonical representation for broader controllability and editing.
- Deformable Neural Scene Representations: Coordinate-based neural scene representations model properties such as occupancy, surface distance, density, and color, but deformable animatable representations remain challenging.
- Deformable Neural Scene Representations: Several dynamic neural methods optimize deformation fields or latent codes, but are designed to memorize scenes and cannot control them beyond interpolations.
- Animatable Neural Radiance Fields: Animatable neural radiance-field methods use template bodies, deformable meshes, inverse deformation fields, or related mechanisms to map observations into canonical space.
- Animatable Shapes: Traditional animatable-shape methods extract articulated or rigged meshes and may optimize blend weights and rigs from data.
- TAVA Design: TAVA uses volumetric rendering, LBS-based non-linear deformation, blending weights, and canonical-space appearance models for actor representation.Its canonical-space models parameterize color, density, and ambient occlusion.
3 Method
TAVA builds a template-free volumetric neural actor from a canonical representation, learned skinning, and neural rendering. Its deformation and rendering design supports novel-pose animation, inverse canonical querying, pose-dependent shading, and cross-view correspondence tracing.
- 3.2 Canonical Neural Actor Representation: TAVA combines a canonical neural radiance field with neural blend skinning to represent an articulated actor and animate it without a body template.The radiance field models canonical geometry and appearance, while learned skinning weights describe animation.
- 3.2 Canonical Neural Actor Representation: The canonical representation removes pose effects from geometry and appearance, while its skinning weights remain valid across a large range of novel poses.This design also supports matching in pose-independent canonical space.
- 3.3 Skinning-based Deformation: Forward LBS maps canonical points to view space, while a learned residual deformation models non-linear effects such as muscles and clothing dynamics.An additional background weight lets empty space remain unaffected by skeleton deformation.
- 3.3 Skinning-based Deformation: Because the deformation has no analytical inverse, TAVA formulates canonical correspondence recovery as root finding and solves it numerically with Newton’s method.Multiple inverse-rigid initializations are used for contact cases, while restricting initialization to the K = 5 nearest bones reduces computation.
- 3.4 Deformation-based Neural Rendering: A pose-conditioned coordinate-based MLP estimates ambient-occlusion shading, and canonical coordinates are accumulated during rendering to trace correspondences across views and poses.Inverse root finding fails for 1% to 8% of view-space points; the full model interpolates attributes from nearest valid neighbors along the ray.
4 Experiments
TAVA is evaluated on human and synthetic-animal subjects using splits designed to test novel views and poses, including out-of-distribution poses. It is compared with template-free and template-based baselines on rendering, correspondence, and editing, alongside ablations of deformation and shading components.
- Datasets: Experiments use four ZJU-Mocap human subjects and two synthetic animal subjects, Hare and Wolf, rendered from multiple views.The animal subjects are rendered using Blender.
- Data Splits: The evaluation separates novel-view synthesis from in-distribution and out-of-distribution novel-pose synthesis, with four human training views and 17 testing views.Animals use 10 random training views and 10 testing views; valview retains training poses while changing viewpoints.
- Baselines: TAVA is compared with template-free methods NARF and A-NeRF, SMPL-based methods Animatable-NeRF and NeuralBody, and a pose-conditioned Pose-NeRF baseline.Animatable-NeRF and NeuralBody are excluded for animals because they require a template 3D model; NARF uses a re-implementation for the reported comparison.
- Novel-view Synthesis: TAVA outperforms other template-free methods on novel-view synthesis and is comparable to SMPL-based methods on ZJU-Mocap, while remaining applicable to animal subjects.The SMPL-based methods benefit from the SMPL body template but do not work on other creatures such as animals.
- Novel-pose Synthesis: On out-of-distribution poses, TAVA outperforms template-free and template-based methods on ZJU-Mocap and drops only 1.68 dB on animals versus approximately 5 dB for other methods.Across InD-to-OOD poses, reported drops range from 0.67 dB to 2.66 dB on ZJU-Mocap and 1.68 dB to 5.73 dB on animals. Forward LBS is identified as more reliable than inverse deformation driven by pose-conditioned MLPs for the synthetic animals.
- Limitations and Ablations: The method trains 5 to 8 times slower than baselines because inverse deformation requires a root-finding process, and unseen-pose deformation and shading remain challenging to estimate.The authors suggest invertible neural networks as a possible way to avoid root finding; lighting-dependent shading is especially challenging.
A Inverse Skinning Gradients.
TAVA finds each view-space sample’s canonical correspondence by solving an inverse skinning constraint, then differentiates this implicit solution to optimize deformation parameters.
- Correspondence recovery: Canonical correspondence x_c^* is found for each view-space sample x_v by solving the deformation equation constrained to equal x_v.The constraint combines LBS with a learned pose-dependent displacement Δ_w.
- Gradient computation: Gradients of the overall loss with respect to skinning and deformation parameters require derivatives through x_c^*.The loss gradients factor through ∂x_c^*/∂Θ_s and ∂x_c^*/∂Θ_Δ.
- Gradient computation: Back-propagation computes the loss derivative with respect to canonical coordinates, while implicit differentiation computes the parameter sensitivity of the root-finding solution.The second derivative terms are obtained analytically via implicit differentiation.
B Dataset Splits and Pose Clustering.
The dataset is split by pose similarity rather than consecutive frames, using K-Medoids clustering to separate similar motions and construct an out-of-distribution validation set.
- Dataset splitting: Consecutive-frame splits can place repetitive or similar poses in both training and validation, making evaluation unsuitable.A pose similarity matrix for ZJU-Mocap subject 313 illustrates the repetitive motion pattern.
- Pose clustering: Pose differences are measured after disabling global root transformations, using Euclidean distances between corresponding mesh vertices.SMPL mesh information is used for ZJU-Mocap pose clustering.
- Pose clustering: K-Medoids partitions the poses into K = 10 clusters, after which the most different cluster is selected as the OOD validation set.The clustering procedure is illustrated for ZJU-Mocap subject 315 and the Wolf subject.
C Implementation Details.
TAVA uses multiple MLPs for radiance, deformation, and appearance, with positional encodings and hierarchical volume rendering following Mip-NeRF settings.
- Network architecture: TAVA employs four MLPs, with radiance and deformation networks using four layers, 128 hidden units, and 4-degree positional encoding.The radiance network is additionally described as an 8-layer, 256-hidden-unit MLP with 10-degree integrated positional encoding.
- Volume rendering: Volume rendering draws 64 samples per ray at both coarse and fine levels using Mip-NeRF hyper-parameters.
D Baseline Implementation Details.
The evaluation uses official implementations for template-based baselines and re-implementations for template-free baselines, with implementation checks on ZJU-Mocap.
- Baseline implementations: Animatable-NeRF and NeuralBody are evaluated using their official implementations, while NARF and A-NeRF are re-implemented in TAVA’s codebase.The authors adapt the implementations to ZJU-Mocap for fair comparison.
- Verification: The re-implementation verification is reported on ZJU-Mocap subject 313 in Table 5.
E Visualizations for Skinning Weights.
The visualizations show learned skinning weights and canonical geometry for animal subjects and ZJU-Mocap data, with colors identifying each point’s top-1 bone assignment.
- Visualizations for Skinning Weights: Learned skinning weights and canonical geometry are visualized for both animal subjects and ZJU-Mocap data.The surfaces are extracted from the density field using marching cubes with threshold 5.0.
- Visualizations for Skinning Weights: Color denotes the top-1 bone from the learned skinning weights.
F Challenges in ZJU-Mocap Dataset.
The ZJU-Mocap dataset contains imperfect camera calibrations and varying camera exposures, producing visual inconsistencies that affect evaluation.
- Challenges in ZJU-Mocap Dataset: ZJU-Mocap includes imperfect camera calibrations and various camera exposures that previous works did not address or mention.
- Challenges in ZJU-Mocap Dataset: Imperfect camera calibrations can cause ghost effects in views rendered by a standard NeRF trained on a single frame.The figure contrasts this artifact with ground-truth images of the same subject captured by different cameras.
G Per-subject Breakdown Comparisons.
The paper reports per-subject quantitative comparisons against baseline methods for human subjects in ZJU-Mocap and for the animal subjects Hare and Wolf.
- Per-subject Breakdown Comparisons: Per-subject quantitative metrics are reported against all baseline methods on the ZJU-Mocap dataset.
- Per-subject Breakdown Comparisons: Per-subject quantitative metrics are also reported for the animal subjects Hare and Wolf.