Source-linked AI summary
BANMo: Building Animatable 3D Neural Models from Many Casual Videos
Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, Hanbyul Joo
TL;DR
Existing articulated 3D reconstruction methods often depend on specialized sensors or predefined deformable templates, limiting their applicability to diverse objects in the wild. BANMo combines canonical neural representations, neural blend skinning, and self-supervised embeddings to build animatable models from casual videos, achieving higher-fidelity reconstructions than prior works for humans and animals.
Problem
Articulated 3D reconstruction often relies on specialized sensors or pre-built deformable models, which do not scale to diverse objects in the wild.
Method
BANMo consolidates multiple casual videos in a canonical space using neural implicit functions, differentiable invertible blend-skinning deformations, and self-supervised canonical embeddings.
Results
BANMo produces higher-fidelity 3D reconstructions than prior works for humans and animals and renders realistic images from novel viewpoints and poses.
Takeaways & Limitations
BANMo can register multiple unsynchronized videos into one canonical model, improve reconstruction completeness with more videos, and retarget driving-video articulations.
Abstract
from arXiv · showhide
Prior work for articulated 3D shape reconstruction often relies on specialized sensors (e.g., synchronized multi-camera systems), or pre-built 3D deformable models (e.g., SMAL or SMPL). Such methods are not able to scale to diverse sets of objects in the wild. We present BANMo, a method that requires neither a specialized sensor nor a pre-defined template shape. BANMo builds high-fidelity, articulated 3D models (including shape and animatable skinning weights) from many monocular casual videos in a differentiable rendering framework. While the use of many videos provides more coverage of camera views and object articulations, they introduce significant challenges in establishing correspondence across scenes with different backgrounds, illumination conditions, etc. Our key insight is to merge three schools of thought; (1) classic deformable shape models that make use of articulated bones and blend skinning, (2) volumetric neural radiance fields (NeRFs) that are amenable to gradient-based optimization, and (3) canonical embeddings that generate correspondences between pixels and an articulated model. We introduce neural blend skinning models that allow for differentiable and invertible articulated deformations. When combined with canonical embeddings, such models allow us to establish dense correspondences across videos that can be self-supervised with cycle consistency. On real and synthetic datasets, BANMo shows higher-fidelity 3D reconstructions than prior works for humans and animals, with the ability to render realistic images from novel viewpoints and poses. Project webpage: banmo-www.github.io .
1. Introduction
BANMo targets accurate, animatable 3D reconstruction of deformable objects from casual videos, combining canonical neural representations, articulated deformation, and dense correspondence learning.
- Casual-video reconstruction supports turning user-captured images and videos into 3D content or animatable assets, but rigid-scene methods do not generalize to deformable people and pets.
- BANMo consolidates cues from multiple videos into a fixed canonical space to learn a high-fidelity model of a subject’s appearance, shape, and articulation.
- The method addresses canonical representation, time-dependent deformation, and correspondence across differing viewpoints, lighting, backgrounds, and object deformations.
- Neural implicit functions represent canonical color and surface, while neural blend skinning constrains deformation and supports unknown-camera pose variation.
2. Related work
Related work spans parametric body models and category-agnostic non-rigid reconstruction, but existing approaches face limitations with diverse categories, fine detail, large deformations, or long-range correspondence.
- Parametric human and animal models rely on registered 3D scans and can reconstruct shapes from single images or videos, but applying them broadly across categories is challenging.
- Coarse template-based methods are not able to recover fine-grained details or large deformations.
- Non-rigid structure-from-motion reconstructs class-agnostic shapes from 2D point trajectories, yet inaccurate long-range correspondences limit performance on videos in the wild.
3. Method
BANMo represents deformable objects in a canonical space and learns their appearance, shape, articulated deformation, and cross-video correspondences through differentiable rendering.
- 3.1. Shape, Appearance, and Warping Model: BANMo represents 3D shape, color, and dense semantic embeddings with implicit functions in a canonical, time-invariant rest-pose space.The model deforms and renders these representations at each time instance using differentiable volume rendering.
- 3.1. Shape, Appearance, and Warping Model: BANMo models color with view direction and environment codes while converting SDF-based geometry into density for volume rendering.The learnable parameter β controls object solidness, and the SDF zero level-set provides a principled surface representation.
- 3.3. Registration via Canonical Embeddings: Canonical embeddings map 3D canonical points to features that pixels can match across viewpoints, lighting conditions, and video times.The embedding is jointly optimized with shape and articulation through consistency between feature matching and geometric warping.
- 3.1. Shape, Appearance, and Warping Model: The rendered model supports color, opacity, expected surface intersection, and optical-flow prediction from warped ray samples.Flow is obtained by forward-warping sampled canonical points to another time and projecting them with a video-specific camera matrix.
- 3.2. Deformation Model via Neural Blend Skinning: Neural blend skinning defines differentiable forward and backward mappings between canonical space and camera-space poses.These mappings compose neural-weighted rigid-bone transformations, making the articulated deformations invertible and suitable for large deformations.
- 3.2. Deformation Model via Neural Blend Skinning: Fine geometry is modeled by adding coordinated-MLP delta skinning weights to coarse weights before softmax normalization.Pose codes use latent root and body representations, with sinusoidal basis functions across frame indices and separate matrices for each video.
4. Experiments
BANMo is evaluated on casual-video reconstruction, component ablations, multi-video scaling, and motion retargeting. Across these experiments, its neural implicit and articulated modeling choices support detailed reconstructions, registration across videos, and pose transfer.
- Dataset and evaluation: BANMo reconstructs casual-video animals and humans at larger scale, with collections containing around 1k images per video.Silhouettes and optical flow are extracted using PointRend and VCNrobust.
- Dataset and evaluation: Table 1 reports 3D Chamfer distance and F-score averaged over all frames, with models normalized and aligned through a 3D similarity transformation.Eagle and hands models are resized to a 2m bounding-box edge; some results use ground-truth root poses and S denotes single-video results.
- Reconstruction results: BANMo produces more accurate geometry than Nerfies across all sequences and recovers finer geometry and articulation than ViSER.Nerfies fails on large articulations such as fast cat-head motion, while ViSER blurs details and misses head rotation and leg switching.
- Reconstruction results: BANMo’s neural implicit representation can recover from topology errors during gradient updates, unlike ViSER’s sub-optimal topology that limits improvement with more iterations.Figure 5 shows an incorrectly reconstructed dog rear leg being automatically corrected during optimization.
- Diagnostics: Removing PoseNet causes root poses to collapse to a degenerate solution, while removing canonical embeddings or flow loss prevents registration and produces ghosting.The ablations identify root-pose initialization and registration cues as crucial components.
- Diagnostics: Replacing neural blend skinning with MLP-SE(3) yields less regular deformation, while MLP-translation produces ghosting wings under large motion.BANMo models large articulations and complex deformation such as close hand contact.
- Diagnostics: With more videos, BANMo registers observations into the same canonical space, improving reconstruction completeness and reducing shape ambiguities; Nerfies does not improve similarly.The comparison uses one versus eight AMA samba videos.
- Motion retargeting: BANMo retargets articulations from a tiger driving video to a pre-optimized cat model by optimizing only video-specific and frame-specific pose codes.Shared cat-model parameters, including shape, skinning, and canonical features, remain frozen during retargeting.
5. Discussion
BANMo reconstructs high-fidelity animatable 3D models from casual videos without predefined shape templates or preregistered cameras, but depends on pose initialization and substantial computation.
- 5. Discussion: BANMo reconstructs high-fidelity animatable 3D models from casual videos without a predefined shape template or preregistered cameras.It registers unsynchronized frames into a shared canonical space using feature-metric consistency and 2D–3D correspondence matching.
- 5. Discussion: A hybrid neural blend-skinning model supports reconstruction of large deformations and fine geometry.
- Limitations: BANMo uses pretrained DensePose-CSE to provide rough root body-pose registration, creating a dependency on a generic relative root-pose estimator.
- Limitations: Optimization requires substantial computation, with cost increasing linearly with the number of input images.
A. Notations
This material points readers to notation and parameter references while comparing BANMo with Nerfies and ViSER.
- A. Notations: The paper provides a dedicated list of notations in Table 6.
- A. Notations: The paper provides a dedicated list of learnable parameters in Table 7.
- A. Notations: Table 2 summarizes differences between Nerfies, ViSER, and BANMo.
B.1. Root Pose Initialization
BANMo initializes root poses with PoseNet, trained on rendered DensePose-CSE embeddings and robustified against occlusions, then uses staged optimization and adaptive sampling.
- B.1. Root Pose Initialization: PoseNet predicts root poses from single images using DensePose-CNN pixel features decoded by a ResNet-18.The inference pipeline applies PoseNet to each video frame to obtain initial root poses.
- B.1. Root Pose Initialization: PoseNet training renders 16-channel surface-embedding images from random spherical viewpoints and augments them with adversarial masks.The feature images are derived from pretrained DensePose-CSE embeddings for humans and quadruped animals.
- B.1. Root Pose Initialization: PoseNet is trained by comparing predicted rotations with ground-truth rotations using a geometric rotation loss.Learning rotation prediction is sufficient for initializing the root body pose; translation is initialized as T = (0, 0, 3)^T.
- B.1. Root Pose Initialization: The optimization uses an uncertainty MLP trained from current color-reconstruction errors to identify active samples.After initial iterations, half of the samples are replaced with high-uncertainty pixels, which improves reconstruction fidelity.
- B.1. Root Pose Initialization: The final optimization has three stages, including a second stage that improves articulations using coordinate gradient descent and 2D cycle consistency.
C.1. SFM root pose initialization
SFM-based root-pose initialization is unreliable for deformable objects, whereas the evaluation compares rotation errors against BANMo-optimized poses.
- C.1. SFM root pose initialization: COLMAP registered only 18 of 811 casual-cat images because deformable-object motion violates rigidity assumptions.
- C.1. SFM root pose initialization: DROID-SLAM registered all images, but its pose accuracy was lower than PoseNet's in Table 4.
- C.1. SFM root pose initialization: Table 4 evaluates root-pose prediction using mean and standard deviation of per-frame rotation error, with BANMo-optimized poses as ground truth.Rotations are aligned to ground truth by a global rotation under chordal L2 distance.
C.2. More ablation study
BANMo’s ablations identify practical choices that affect reconstruction quality, stability, and efficiency, including bone configuration, initialization, delta skinning weights, and active sampling.
- Number and location of bones: Too few bones fail to recover all body parts through over-regularization, while more than 16 bones produce good reconstructions at higher memory cost.Constraining bones near the surface with a Sinkhorn divergence loss improves results.
- Sensitivity to incorrect initial pose: BANMo remains stable with initial pose errors up to 50° rotation error.The evaluation injects Gaussian noise producing average rotation errors of 20°, 50°, and 90°.
- Bone re-initialization: Re-initializing rest bone parameters after improving the shape estimate avoids bad local optima and yields more accurate final reconstructions.The re-initialization updates bone parameters according to the current shape estimate.
- Delta skinning weights: Point-specific delta skinning weights prevent the reconstructed shape and motion from being over-regularized by the 3D Gaussians.The ablation evaluates reconstruction without learning these point-specific corrections.
- Importance sampling: Active sampling improves reconstruction by focusing on informative frames and pixels, leading to faster convergence and more accurate geometry.Without active sampling, the method converges more slowly and misses details such as ears and eyes; active samples focus on high-error face and boundary pixels.
C.3. Qualitative results
The supplementary material provides additional qualitative results and reference tables documenting notation and learnable parameters.
- Qualitative results: Complete qualitative results are provided on the supplementary webpage.
- Notation: A supplementary table documents the notation used throughout the method.
- Learnable parameters: A supplementary table lists the learnable parameters of the model.