LASR: Learning Articulated Shape Reconstruction from a Monocular Video

1. Introduction

• Limited 2D measurements admit multiple interpretations of 3D geometry and motion, making nonrigid reconstruction under-constrained.The challenge matters for perceiving and modeling the geometry and dynamics of 3D entities.Source: §1 · p. 1
• Category-specific shape priors can fail on novel or unknown objects, where methods may hallucinate an average 3D structure.Obtaining the 3D annotations needed to train such priors is costly; SMPL uses thousands of registered human scans, while SMAL uses animal-toy scans and a manually rigged mesh.Source: §1 · p. 1
• Video provides multiview constraints on the same instance, while two-frame optical flow supplies dense correspondences that are more reliable to obtain than long-range point tracks.The paper motivates optical flow as providing finer-grained information than keypoints or semantic parts.Source: §1 · pp. 1–2
• LASR reconstructs articulated shapes from monocular video without category templates, using rigid-bone constraints, coarse-to-fine remeshing, and soft-symmetric constraints.The authors report state-of-the-art performance on BADJA animal videos, strong results against model-based human methods, and higher accuracy than A-CSM and SMALify on two animated animals.Source: §1 · p. 2

2. Related Work

• Model-based methods use parametric shape models learned from 3D scans, but are challenging to apply to unknown categories or categories with limited 3D data.Regression methods predict model-specific parameters from images or video and are usually trained with 3D data generated by parametric models.Source: §2 · pp. 2–3
• LASR renders silhouette, texture, and two-frame optical flow from a monocular video, then compares them with observed pixels, masks, and flow to update shape, motion, and camera parameters.The method overview describes this as gradient-based optimization of an inverse graphics problem.Source: Figure 2 · p. 3
• Category mesh methods use 2D annotations and category priors, but single-view reconstructions are usually coarse and lack instance-specific detail.Adapting category-specific models to test-time video still does not handle unknown object classes.Source: §2 · p. 3
• Template-free methods include PIFu, which requires ground-truth 3D shapes [41, 42], and A3DC, which requires user stroke interactions.NRSfM avoids templates but depends on 2D keypoints or point trajectories and low-rank shape or deformation assumptions [2, 12, 19].Source: §2 · p. 3

3. Approach

• LASR optimizes rest shape, time-varying articulations, and camera intrinsics by matching rendered measurements to those observed in a monocular video.The method frames this as analysis-by-synthesis and hypothesizes that low-rank shape and motion can be solved up to unknown scale.Source: §3 · p. 3

3.1. Forward-synthesis model

• The forward model renders color images, silhouettes, and optical flow, using perspective projection and differentiable rendering for color and silhouette.Forward flow is computed from the projected displacement of surface positions between consecutive frames.Source: §3.1 · pp. 3–4

3.2. Articulation Modeling

• Linear-blend skinning constrains vertex motion with rigid bone transformations; LASR jointly learns skinning weights and bone transformations, representing weights with Gaussian mixtures.The Gaussian mixture reduces skinning-weight parameters from NB to 9B and guarantees smoothness.Source: §3.2 · p. 4

3.3. Self-supervised Learning from a Video

• Self-supervision combines silhouette, optical-flow, texture, and perceptual losses with mesh and motion regularizers, including soft symmetry of the rest shape.The symmetry plane is initialized by visual inspection and jointly optimized; regularizers also encourage smooth surfaces, natural deformation, and small deformation from rest.Source: §3.3 · pp. 4–5

3.4. Implementation Details

• A ResNet-18 predicts time-varying camera and pose parameters, while coarse-to-fine reconstruction remeshes the object and jointly optimizes its shape, articulation, and cameras.The process starts with rigid reconstruction at S0, then repeats remeshing and joint optimization for S1–S3.Source: Figure 3 · p. 4, §3.4 · p. 5
• On near-rigid DAVIS sequences, LASR reconstructs the rigid object and near-rigid person, whereas COLMAP recovers only the visible rigid part.The comparison covers scooter, soapbox, and car sequences.Source: Figure 4 · p. 5

4. Experiments

• Real-video 3D quality is approximated by keypoint-transfer accuracy, while mesh accuracy is evaluated on five articulated videos with ground-truth meshes using Chamfer distance.The mesh evaluation aligns predicted and ground-truth meshes by a 3D similarity transformation before computing bidirectional Chamfer distance.Source: §4 · pp. 5–6, §4.2 · p. 7

4.1. 2D Keypoint Transfer on Animal Videos

• PCK-T reaches 49.3 vs UMR: 32.4 on horsejump-high; LASR beats reconstruction baselines across categories, though auto-masks reduce performance.LASR also improves on initial optical flow for camel transfer (81.9% vs 47.9%).Source: Table 2 · p. 6, §4.1 · p. 6

4.2. Mesh Reconstruction on Articulated Objects

• Qualitatively, LASR recovers finer geometry and instance-specific details, including the dog’s fluffy tail and a more natural pose, and reconstructs all four horse legs.The experiments also compare arbitrary-object DAVIS videos with COLMAP.Source: Figure 6 · p. 7, Figure 7 · p. 8, §4.2 · p. 8
• On animated-object meshes, LASR scores 0.28 vs A-CSM: 0.38 on dog and slightly beats A-CSM on horse, but trails VIBE on dancer (0.35 vs 0.22).For the stone golem, LASR is the only method to reconstruct a meaningful shape.Source: Table 3 · p. 7, §4.2 · p. 7
• Ablations identify optical-flow supervision, symmetry canonicalization, CNN camera parameters, linear-blend skinning, coarse-to-fine re-meshing, and parametric skinning as beneficial or critical design choices.The study uses rendered color, silhouette, and flow supervision on a rigid object and an animated dog.Source: Table 4 · p. 7, §4.2 · p. 8
• LASR struggles with surfaces unseen in every input view and heavy occlusions missed by mask annotations; runtime is under an hour for rigid objects and a few hours for nonrigid shapes on one GPU.Source: §4.2 · p. 8

• LASR reconstructs articulated shapes from monocular video without category-specific shape templates.Source: §5 · p. 8
• The method reconstructs individual objects across diverse categories, including humans, camels, dogs, and bears.Source: §5 · p. 8
• The authors conclude that LASR applies to a wide range of scenarios and express hope that it will enable further progress in articulated shape reconstruction.Source: §5 · p. 8

6.1. Implementation details

• Video preprocessing skips the next frame when average measured flow inside the object silhouette is below 0.05 in clip space.Source: §6.1 · p. 11

6.2. Notations

• Table 6 defines camera, mesh, bone, skinning, and motion symbols, as well as the parameter sets optimized in stages S0 and S1–3.Source: Table 6 · p. 12, §6.2 · p. 12