Source-linked AI summary
NASA: Neural Articulated Shape Approximation
Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, Andrea Tagliasacchi
TL;DR
Articulated objects are difficult to represent and query efficiently with polygonal meshes, especially when differentiability and deformation are required. NASA addresses this with pose-conditioned neural occupancy built from structured per-part functions, and reports strong deformation generalization, direct queries, and differentiable tracking. The approach remains dependent on pose-related quantities and is limited to individual subjects and lower-frequency detail.
Problem
Articulated-object modeling needs representations that support deformation, occupancy queries, and differentiable processing beyond mesh-based approaches.
Method
NASA is a structured neural occupancy model that conditions indicator functions on pose and exploits quasi-rigid articulated parts.
Results
NASA outperforms previous geometric learning algorithms, approaches mesh-based surface reconstruction accuracy, and enables .06 ms/query occupancy evaluation.
Takeaways & Limitations
The representation supports differentiable occupancy queries and nearly trivial implementation of tracking realistic articulated bodies.
Takeaways & Limitations
NASA relies on pose transforms and skinning weights, currently models individual subjects, and fails to capture some high-frequency features.
Abstract
from arXiv · showhide
Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics. To efficiently simulate deformation, existing approaches represent 3D objects using polygonal meshes and deform them using skinning techniques. This paper introduces neural articulated shape approximation (NASA), an alternative framework that enables efficient representation of articulated deformable objects using neural indicator functions that are conditioned on pose. Occupancy testing using NASA is straightforward, circumventing the complexity of meshes and the issue of water-tightness. We demonstrate the effectiveness of NASA for 3D tracking applications, and discuss other potential extensions.
1 Introduction
NASA introduces a pose-conditioned neural occupancy representation for articulated objects, replacing mesh-based querying with direct differentiable occupancy evaluation. The framework supports articulated deformation modeling, spatial queries, and differentiable tracking.
- Motivation: Neural implicit representations are flexible, continuous, and differentiable, but articulated-object models have received comparatively little attention.Articulated representations matter for humans and animals in applications including games, animation, augmented reality, and virtual reality.
- Limitations of existing representations: Polygonal meshes require expert supervision, handle topology variation poorly, and complicate simultaneous surface modeling and volume-based intersection testing.Mesh-based articulated models also require additional acceleration structures for occupancy queries.
- Limitations of existing representations: Prior neural articulated-deformation approaches still rely on classical acceleration data structures for queries, sacrificing full differentiability.NASA instead learns an implicit neural occupancy that can be queried directly.
- NASA: NASA combines per-part learnable indicator functions with per-part pose encoders to model localized non-rigid deformations.The architecture exploits the relative stability of local body-part geometry across poses and improves generalization to unseen poses.
- Capabilities: NASA supports differentiable occupancy, continuous surfaces, direct spatial queries, and a fully differentiable point-cloud tracker.The tracker estimates pose by maximizing likelihood under NASA’s occupancy model rather than requiring mesh-based tracking machinery.
- Efficiency: .06 ms/query on an NVIDIA GTX 1080 is reported for differentiable occupancy queries.This avoids converting the representation or dynamically updating spatial acceleration data structures.
2 Related work
Related work spans mesh skinning, accelerated intersection queries, part-based approximations, and neural implicit representations. NASA addresses articulated deformation and occupancy querying within one neural framework.
- Skinning algorithms: Skinning algorithms deform mesh vertices as skeleton joints change, but linear blend skinning can produce artifacts and requires manually painted weights for best results.Reported artifacts include collapsing elbows, candy-wrapper effects, and volume loss.
- Object intersection queries: Polygonal meshes make inside/outside queries inefficient because each query may require testing thousands of triangles.Voxel grids, octrees, and kd-trees are among the spatial structures developed to accelerate such queries.
- Notation: Animation notation associates T frames with pose parameters, with each θ_t encoding transformations of B bones.The figure contrasts a rest-frame occupancy and pose with an animation sequence of posed frames.
- Part-based representations: Part-based representations approximate articulated shapes with moving rigid primitives, enabling efficient queries but introducing a second representation that may not match the desired deformation exactly.Automatic creation of such part-based representations remains a core challenge.
- Neural implicit object representation: Recent neural implicit object methods focus on static shapes in aligned canonical frames rather than modeling transformations.NASA positions its contribution as adapting neural implicit architectures to complex articulated objects.
3 Neural Articulated Shape Approximation
NASA models occupancy conditioned on articulated pose by learning a neural function over points and bone transformations. The formulation combines conditional occupancy modeling with pose-related structure and sampled training objectives.
- Problem formulation: The problem is to model the joint distribution of pose and occupancy, decomposed into a conditional occupancy term and a pose-prior term.The pose θ represents transformations of B bones, while O defines whether points lie inside or outside the articulated body.
- Problem formulation: NASA focuses on learning p(O|θ), the occupancy distribution conditioned on pose.Pose priors can be combined with this conditional model to obtain the joint distribution p(θ, O).
- Neural occupancy: The pose-conditioned occupancy function Oω(x|θ) maps a 3D point and pose to a real-valued occupancy prediction approximating ground-truth occupancy.The model is trained from T ground-truth occupancies associated with T poses.
- Pose and part structure: Each pose contains B posed bone transformations, and available vertex-to-part association weights encode the part-whole relationship.The weights lie in [0,1]^B and sum to one for each vertex.
- Occupancy queries: Occupancy queries on general meshes can require generalized winding numbers because of self-intersections and non-watertight geometry.NASA’s neural occupancy formulation is designed to provide direct queries without this mesh-querying complication.
- Training objective: Training samples points from the volume surrounding a posed character and adds samples near the deformed surface.The sampling density p(x) defines how points in R^d contribute to the objective.
4 Pose conditioned occupancy O(x|θ)
The paper compares unstructured, rigid, and deformable pose-conditioned occupancy architectures, culminating in a part-based model that captures localized non-rigid deformation through pose-dependent components.
- Architectures: The unstructured baseline uses a global MLP conditioned on pose, while structured variants represent geometry as compositions of rigid or deformable elements.Figure 3 distinguishes the three architectures by whether they encode articulated structure explicitly and whether components are rigid or deformable.
- Unstructured model – “U”: Pose can be supplied by concatenating affine bone transformations with the query point, but a compact representation uses relative bone transforms and the root translation.The full concatenation has input size 3+16×B, whereas the proposed representation reduces this to 3+3×B.
- Piecewise rigid model – “R”: The piecewise-rigid model queries learned rest-pose indicator functions after transforming points into corresponding body-part coordinate frames.This formulation assumes each learned component retains a constant local shape across poses.
- Piecewise deformable model – “D”: The deformable model extends the rigid formulation by adjusting each component’s shape according to pose using per-part pose conditionals.NASA combines learnable rest-pose indicator functions with part-specific pose encoding to model localized deformation.
- Piecewise deformable model – “D”: Each part uses a learnable projection from pose information into a low-dimensional subspace, with D=4; experiments report that this bottleneck greatly improves generalization.The projection is one per bone and is motivated by the intuition that only a small subset of coordinate frames affects each part’s deformation.
- Technical details: An auxiliary skinning-weight loss assigns vertices to their highest-weight part, discouraging one component from modeling the entire object while others collapse to zero.The target uses occupancy value 0.5 for the assigned part, matching the surface level set convention.
5 Dense articulated tracking
NASA is applied to dense articulated 3D tracking by minimizing fitting and pose-prior energies directly over implicit occupancy functions. The formulation avoids explicit signed-distance, closest-point, and convolution computations while retaining differentiable optimization.
- Tracking formulation: Dense articulated tracking applies NASA's neural occupancy functions to estimate pose from point-cloud sequences.The pose at frame t is initialized from the minimizer computed at frame t−1; the initial pose θ(0) is assumed to be ground truth.
- Fitting energy: The fitting energy evaluates how well occupancy matches input data, using a Gaussian-smoothed occupancy approximation to signed distance.The Gaussian kernel has zero mean and variance σ2, and the approximation is intended for tracking where large distances can be treated as outliers.
- Fitting energy: Stochastic sampling re-expresses the occupancy convolution, avoiding voxelized representations and GPU 3D convolutions.The number of samples provides direct control over the cost–accuracy trade-off.
- Fitting energy: Equation 17 enables tracking directly on occupancy functions without computing signed distances, closest points, or 3D convolutions.Gradients are made available by applying the re-parameterization trick.
- Pose prior energy: A pose prior constrains transformations because optimization can lose its corrective gradient when the model is too far from the target.The chosen prior aligns runtime bone-direction vectors with their rest-pose counterparts, while more sophisticated priors remain possible.
- Iterative optimization: Pose optimization uses iterative updates centered at the current state, with inverse coordinate frames optimized to avoid back-propagation through matrix inversion.The pose differential is represented through transformations, and rotation-related optimization is centered on the current state.
6 Results and discussion
NASA is evaluated on reconstruction and tracking across DFaust and Transitions, with structured representations generalizing better than unstructured ones and supporting test-time tracking. The method remains constrained by required body-part transformations and skinning weights, subject-specific modeling, and limited high-frequency detail.
- Evaluation setup: The evaluation covers sampled occupancy training data, reconstruction on DFaust and Transitions, qualitative comparisons, and tracking on held-out DFaust sequences.Training samples combine indicator values, transformation frames, and skinning weights; tracking uses test sequences not seen during training.
- Datasets and splits: 100 DFaust models average metrics across subject-wise leave-one-sequence-out tests, while Transitions trains on 80 sequences and tests on 30 sequences from one subject.DFaust contains 10 subjects and 10 sequences per subject; Transitions provides 110 sequences with roughly 1000+ frames per sequence.
- Reconstruction: +49% in F-score is achieved by the rigid structured baseline over the unstructured baseline on DFaust, while the deformable model adds another +5% over the rigid model.The rigid baseline is consistently better than the unstructured baseline under every reported metric.
- Reconstruction: +70% in F-score separates the rigid and unstructured models on Transitions, while moving from DFaust to Transitions improves the deformable model by +1% in F-score.The Transitions dataset contains more varied and complex motions, and the unstructured model struggles more strongly there.
- Tracking: NASA tracks noisy point clouds on two held-out DFaust sequences, with structured representations producing significantly better tracking than unstructured ones.The tracking demonstration uses complete noisy panoptic point clouds without a discriminative per-frame re-initializer and is not intended to outperform traditional baselines.
- Discussion and limitations: The method relies on body-part transformations and skinning weights, currently models individual subjects, and fails to capture high-frequency features such as facial detail.Automatically obtaining the required quantities from raw in-the-wild observations remains an open problem; identity parameters would be needed for competitiveness with mesh-based models.
7 Conclusions
NASA is a structured neural occupancy representation for articulated bodies that supports differentiable occupancy queries and deformable surfaces. It achieves better quality and generalization by encoding the bodies’ quasi-rigid part structure.
- Conclusion: NASA represents articulated bodies with structured neural occupancy, enabling direct occupancy queries and deformable surface representations competitive with classic hand-crafted meshes.The representation is fully differentiable and supports tracking of realistic articulated bodies.
- Conclusion: Encoding quasi-rigid part structure improves representation quality and generalization for articulated bodies.The conclusion identifies task-appropriate inductive bias as central to this improvement.
9 Supplementary material
The supplementary studies analyze NASA’s tracking components, losses, pose representations, projection bottlenecks, and model capacity. They show that the proposed structured and deformable models improve generalization and tracking when key design choices are enabled.
- Loss ablations: L2 loss slightly outperforms binary cross-entropy, while including the skinning-weights loss produces a large improvement.These ablations motivate using L2 and retaining Lweights in the reported experiments.
- Linear subspace projection: Removing the learned linear projection causes the rigid model to outperform the deformable model on the test set, indicating that the projection improves generalization.The projection enforces a sparse pose representation, and D=4 is selected through a dimensionality ablation.
- Pose representations: Encoding query points in part coordinate frames substantially improves the unstructured model, while fixed-point pose encoding provides a compact effective representation.The query-point encoding reduces the need for the network to learn coordinate changes directly.
- Tracking ablations: The pose prior and random perturbations together achieve the best tracking results across all metrics, especially on hard sequences.The pose prior improves IoU from 44.31% to 86.15%, while random perturbations reduce Chamfer error from .00258 to .00006.
- Dataset-wide performance: The deformable model achieves consistent IoU performance across AMASS/DFaust, with only two sequences favoring the rigid model.The comparison covers all 100 DFaust experiments, sorted by deformable-model performance.