Source-linked AI summary
SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes
Xu Chen, Yufeng Zheng, Michael J. Black, Otmar Hilliges, Andreas Geiger
TL;DR
Articulated neural implicit surfaces are difficult to learn because backward warp fields are pose dependent and correspondences may be non-unique. SNARF learns a differentiable forward skinning field in canonical space from posed meshes, achieving better generalization to unseen poses while preserving accuracy.
Problem
Articulating neural implicit surfaces is challenging because backward deformation fields are pose dependent, while deformed-point correspondences may be implicit and non-unique.
Method
SNARF learns pose-independent forward skinning and pose-conditioned neural implicit shapes from posed watertight meshes and bone transformations without predefined skinning supervision.
Results
SNARF generalizes better to unseen poses while preserving accuracy and generates high-quality shapes under bone transformations far beyond training poses.
Takeaways & Limitations
The method supports plausible continuous articulation of diverse 3D human shapes, including clothed bodies, in nearly arbitrary poses.
Takeaways & Limitations
The canonical pose is a free hyper-parameter; the experiments on human shapes use one empirically selected canonical pose.
Abstract
from arXiv · showhide
Neural implicit surface representations have emerged as a promising paradigm to capture 3D shapes in a continuous and resolution-independent manner. However, adapting them to articulated shapes is non-trivial. Existing approaches learn a backward warp field that maps deformed to canonical points. However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn. To address this, we introduce SNARF, which combines the advantages of linear blend skinning (LBS) for polygonal meshes with those of neural implicit surfaces by learning a forward deformation field without direct supervision. This deformation field is defined in canonical, pose-independent space, allowing for generalization to unseen poses. Learning the deformation field from posed meshes alone is challenging since the correspondences of deformed points are defined implicitly and may not be unique under changes of topology. We propose a forward skinning model that finds all canonical correspondences of any deformed point using iterative root finding. We derive analytical gradients via implicit differentiation, enabling end-to-end training from 3D meshes with bone transformations. Compared to state-of-the-art neural implicit representations, our approach generalizes better to unseen poses while preserving accuracy. We demonstrate our method in challenging scenarios on (clothed) 3D humans in diverse and unseen poses.
1. Introduction
SNARF learns articulated neural implicit shapes and forward skinning directly from posed meshes, using canonical-space deformation to generalize beyond training poses. Iterative correspondence finding and differentiable gradients enable unsupervised end-to-end learning while preserving pose-dependent detail.
- Unlike pose-dependent backward fields, SNARF defines forward skinning in canonical space, supporting generalization to unseen poses.
- Iterative root finding retrieves all canonical correspondences for deformed points, allowing the model to handle topology changes.
- Analytical gradients make forward skinning differentiable, enabling joint end-to-end learning of canonical shape and skinning weights from deformed observations.
- SNARF generates high-quality shapes under arbitrary bone transformations, including poses far beyond training, while modeling fine geometric and local pose-dependent deformations.
2. Related Work
Prior articulated-shape methods rely on mesh-based, part-based, or backward-warp formulations with limitations in topology, continuity, supervision, or unseen-pose generalization. SNARF extends skeletal deformation to continuous neural implicit representations with learned forward skinning and pose-dependent correctives.
- Linear blend skinning deforms mesh vertices using convex combinations of bone transformations, while pose correctives reduce articulation artifacts.
- Many learned rigging methods require large datasets of rigged models with hand-painted skinning weights and do not address implicit surfaces.
- Neural implicit representations offer continuous shapes with arbitrary topology, but deforming them requires transforming continuous space rather than fixed mesh vertices.
- Backward deformation fields map deformed points to canonical space but can generalize poorly because their learned fields depend on pose.
- NASA combines independent occupancy-based parts, but rigid part transformations violate surface continuity at intersections and require ground-truth surface skinning weights.
- SNARF learns pose-independent forward weights and pose-dependent canonical-space correctives, producing continuous articulated shapes beyond the training distribution.
3. Method
SNARF represents articulated shapes with a canonical neural occupancy field and pose-independent forward skinning weights, while modeling pose-dependent local deformations. It finds all canonical correspondences of deformed points through iterative root finding and supports end-to-end learning via differentiable gradients.
- Representation: SNARF represents an articulated object using a canonical neural occupancy field, a canonical LBS weight field, and pose-dependent conditioning for local deformations.LBS captures broad shape changes, while the pose-conditioned neural function models effects such as wrinkles or soft tissue.
- Differentiable Forward Skinning: Iterative root finding retrieves multiple correspondences by using bone-transformation-derived initializations, retaining converged roots and discarding divergent ones by thresholding.This procedure handles topology changes that create multiple canonical points for one deformed point.
- Representation: The canonical LBS weight field uses nonnegative weights summing to one and remains independent of pose.A softmax activation enforces the blend-weight constraints.
- Differentiable Forward Skinning: For a deformed query point, SNARF searches for canonical correspondences whose forward-skinned locations match the query, because the mapping has no closed-form inverse and may be one-to-many.The method evaluates occupancy at the recovered canonical points and composites their predictions to obtain the deformed-point occupancy.
- Differentiable Forward Skinning: SNARF aggregates occupancy across canonical correspondences with a softmax-based approximation to the maximum, allowing gradients to reach all correspondence branches.The union-style aggregation treats a deformed point as occupied when any canonical correspondence is occupied.
- Gradients: Training minimizes binary cross entropy on posed meshes and backpropagates through the occupancy network and LBS weight field using implicit differentiation.The analytically derived correspondence gradients enable joint end-to-end learning of canonical shape and skinning weights.
4. Experiments
Experiments evaluate SNARF on synthetic 2D articulated shapes, minimally clothed humans, and clothed humans, emphasizing topology changes, unseen poses, and local deformation. Across these settings, SNARF generally produces accurate, detailed shapes and degrades more gracefully than baselines outside the training distribution.
- Evaluation settings: Experiments cover 2D sticks, minimally clothed humans, and clothed humans, using controlled articulation and diverse pose and clothing settings.The evaluation includes topology changes, within-distribution poses, out-of-distribution poses, and novel clothing configurations.
- 2D Stick Dataset: For 2D pose extrapolation, SNARF outperforms all baselines with and without topological changes, while interpolation gaps widen as training poses become sparser.Piecewise remains insensitive to training sample density but exhibits artifacts at part intersections.
- 2D Stick Dataset: SNARF handles topology changes gracefully, whereas Back-LBS cannot represent one-to-many backward correspondences and distorts the occupancy field.The topology-change setting adds a rigid object that can intersect the articulated stick in posed space.
- Minimally Clothed Humans: On minimally clothed humans, SNARF improves IoU over NASA by 1.2% for uniformly sampled points and 4.6% for near-surface points.All methods perform well within the training distribution, but baselines show missing parts or discontinuities near joints.
- Minimally Clothed Humans: For out-of-distribution human poses, the IoU surface gap between SNARF and NASA increases from 4.6% to 20.4%, while SNARF degrades gracefully.Other baselines produce distorted, meaningless, or artifact-laden shapes under substantially different test poses.
- Clothed Humans: On clothed humans, SNARF models varied clothing with wrinkles and plausible novel-pose details, but cloth may fall unnaturally for extreme unseen poses.The reported failure case still degrades gracefully.
5. Conclusion
SNARF introduces a differentiable forward skinning model for articulating neural implicit surfaces, learning continuous pose-conditioned shapes and skinning weights from meshes. It achieves state-of-the-art results on articulated neural implicit representations and better generalization to unseen poses than baselines.
- SNARF learns continuous pose-conditioned shapes and skinning weights from meshes using a differentiable forward skinning model.
- The method generates plausible shapes in nearly arbitrary poses and achieves state-of-the-art results for articulated neural implicit representations of 3D human bodies.
- SNARF generalizes significantly better to unseen poses than the baselines on challenging clothed 3D humans with diverse shapes and poses.
- Future work will extend the method across subjects and train from images only using differentiable rendering.