Source-linked AI summary
Self-supervised Learning of Motion Capture
Hsiao-Yu Fish Tung, Hsiao-Wei Tung, Ersin Yumer, Katerina Fragkiadaki
TL;DR
Monocular motion capture is limited by optimization sensitivity and the scarcity of realistic dense 3D annotations. This paper trains a neural model with synthetic supervision and differentiable-rendering self-supervision, finding that test-time adaptation improves reconstruction over pretrained and direct-optimization alternatives.
Problem
Monocular motion capture is difficult because optimization-driven systems are sensitive to clutter and initialization, while realistic video lacks large-scale dense 3D mesh annotations.
Method
A neural network predicts SMPL 3D meshes from monocular image sequences and adapts at test time using differentiable reprojection of keypoints, segmentation, and mesh motion.
Results
The model outperforms pretrained non-adapted and direct-optimization baselines, and all three self-supervised losses show complementary value in ablation.
Takeaways & Limitations
Combining supervised pretraining with unsupervised adaptation lets the model improve using unlabelled video data for dense human 3D body tracking.
Abstract
from arXiv · showhide
Current state-of-the-art solutions for motion capture from a single camera are optimization driven: they optimize the parameters of a 3D human model so that its re-projection matches measurements in the video (e.g. person segmentation, optical flow, keypoint detections etc.). Optimization models are susceptible to local minima. This has been the bottleneck that forced using clean green-screen like backgrounds at capture time, manual initialization, or switching to multiple cameras as input resource. In this work, we propose a learning based motion capture model for single camera input. Instead of optimizing mesh and skeleton parameters directly, our model optimizes neural network weights that predict 3D shape and skeleton configurations given a monocular RGB video. Our model is trained using a combination of strong supervision from synthetic data, and self-supervision from differentiable rendering of (a) skeletal keypoints, (b) dense 3D mesh motion, and (c) human-background segmentation, in an end-to-end framework. Empirically we show our model combines the best of both worlds of supervised learning and test-time optimization: supervised learning initializes the model parameters in the right regime, ensuring good pose and surface initialization at test time, without manual effort. Self-supervision by back-propagating through differentiable rendering allows (unsupervised) adaptation of the model to the test data, and offers much tighter fit than a pretrained fixed model. We show that the proposed model improves with experience and converges to low-error solutions where previous optimization methods fail.
1 Introduction
Monocular motion capture remains difficult because optimization-driven systems are sensitive to clutter, initialization, and local minima, while realistic 3D mesh annotations are scarce. The proposed model combines synthetic supervision with differentiable-rendering self-supervision to predict and adapt 3D human shape and motion.
- Motivation: Detailed monocular human-motion understanding could support applications including rehabilitation guidance, patient monitoring, safer human-robot interaction, and motion capture for film.Existing film workflows may require substantial artist labor or expensive multi-camera and green-screen setups.
- Motivation: Current monocular motion capture systems optimize 3D human-model parameters against video measurements and are vulnerable to background clutter, local minima, and initialization errors.These limitations have motivated green-screen-like backdrops and careful initialization.
- Approach: Training combines strong synthetic-data supervision with self-supervision from differentiable keypoint, segmentation, and dense-motion reprojection losses in real videos.The losses match projected 3D quantities to detected 2D keypoints, segmentation, and optical flow.
- Approach: The model maps monocular image sequences to corresponding 3D meshes using SMPL pose and surface parameters rather than directly optimizing mesh parameters.SMPL uses θ for joint angles and β for surface shape, with differentiable mesh generation.
- Results: Experiments on SURREAL and H3.6M show that self-supervised adaptation achieves lower 3D reconstruction error than a pretrained non-adapted model, while optimization baselines remain sensitive to initialization.An ablation also finds the keypoint, segmentation, and dense-motion losses are complementary.
2 Related Work
Prior monocular motion-capture work relies on pose and motion priors, staged prediction, or differentiable geometric learning, while the proposed approach couples skeleton and mesh estimation end to end through test-time adaptation.
- 3D Motion capture: Earlier monocular motion-capture methods used linear, Gaussian-process, or twin-Gaussian-process motion priors to address ambiguity and occlusion.Nonlinear priors were reported to outperform linear counterparts.
- 3D Motion capture: Bogo et al. predicted a 3D skeleton from a static image and then optimized a parametric 3D shape while keeping the skeleton fixed.The proposed work instead couples skeleton and mesh estimation in an end-to-end differentiable framework.
- 3D Human pose estimation: Deep 3D pose methods increasingly regress pose directly from RGB using neural networks and supervised training sets, often using 2D pose as an intermediate or auxiliary representation.The abundance of labeled 2D pose examples supports feature learning.
- Deep geometry learning: Differentiable warping and camera projection have enabled self-supervised learning of 3D camera or object motion through photometric losses.These methods connect deep learning with geometric inference.
3 Learning Motion Capture
The model predicts SMPL mesh parameters from monocular image crops and adapts them through differentiable rendering-based self-supervision. It matches projected keypoints, segmentation, and mesh motion to their 2D counterparts while estimating camera parameters.
- Parametrized body model: SMPL represents the human body with pose parameters θ, surface parameters β, and 6,890 differentiably generated mesh vertices.The network predicts these parameters to reconstruct dense 3D human shape and pose.
- Network supervision: The network predicts β and θ from an image crop centered on a detected person, using synthetic supervision and differentiable-rendering losses.The losses compare rendered 3D keypoints, segmentation, and vertex motion with corresponding 2D estimates.
- Keypoint reprojection: Projected 3D body joints are matched to detected 2D keypoints, allowing reprojection minimization to update the neural parameters predicting β and θ.The dense mesh is related linearly to 3D body joints before projection.
- Camera parameters: The model additionally predicts focal length, camera rotation, and in-plane translation for 3D-to-2D projection, while focal length accounts for person scaling instead of z translation.Camera rotation is parameterized by Euler angles, and translation handles small perturbations in object centering.
- Motion reprojection: Dense mesh motion is projected between consecutive frames and matched against estimated optical flow, with ray-casting visibility used to mask invisible vertices.The TensorFlow ray-casting implementation accelerates visibility inference tenfold relative to interfacing with OpenGL.
- Segmentation reprojection: Segmentation reprojection uses bidirectional Chamfer-distance terms to penalize both under-coverage and over-coverage between the projected mesh and image segmentation.The first term ensures the projected model is covered by the image segmentation, while the second encourages the model to cover the image foreground.
4 Experiments
Experiments evaluate the model on synthetic Surreal and real H3.6M data, using supervised pretraining followed by self-supervised adaptation through differentiable rendering. The proposed approach outperforms fixed pretrained and direct-optimization baselines, while ablations show that combining keypoint, segmentation, and motion losses improves reconstruction.
- Datasets: The method is evaluated on Surreal and H3.6M, with dense mesh ground truth available for Surreal but not H3.6M.H3.6M provides annotated 2D and 3D joint locations, whereas Surreal provides ground-truth 3D human meshes.
- Training and evaluation: The model is pretrained on supervised Surreal skeleton and surface parameters, then self-supervised on both datasets using differentiable rendering and reprojection losses.Self-supervision uses ground-truth 2D keypoints and segmentations; H3.6M segmentation masks may be inaccurate because they come from background subtraction.
- Quantitative results: The proposed model outperforms pretrained fixed models and direct optimization baselines across the reported evaluation metrics.Direct optimization adjusts mesh parameters, pose, translation, rotation, and focal length, while the pretrained baseline does not adapt at test time.
- Ablation: Combining keypoint, segmentation, and dense-motion reprojection losses outperforms using keypoint loss alone or keypoint plus segmentation loss.The ablation identifies the three losses as complementary during unsupervised finetuning.
- Discussion: Supervised pretraining provides good initialization without manual effort, while differentiable-rendering self-supervision enables tighter fitting to test data.The discussion characterizes this combination as integrating supervised learning with test-time optimization.
- Qualitative results: Qualitative results show robust mesh prediction despite poor H3.6M input masks and improved 2D person segmentation as a byproduct.The qualitative comparison includes Surreal in the top four rows and H3.6M in the bottom four rows.
5 Conclusion
The paper presents a learning-based model for dense human 3D body tracking that combines synthetic supervision with self-supervision through differentiable rendering. It shows that unlabelled video can improve the model, while identifying higher-accuracy reconstruction and broader 3D learning as future directions.
- The model combines synthetic-data supervision with self-supervision from differentiable rendering of mesh motion, keypoints, and segmentation.
- Unlabelled video improves the model, which is valuable because dense 3D ground truth is difficult to annotate for motion capture.
- Future work includes iterative additive feedback on mesh parameters to achieve higher 3D reconstruction accuracy.
- Another direction is learning residual free-form deformation on top of the parametric SMPL model in a self-supervised manner.
- Extending the model beyond human 3D shape could enable neural agents to learn 3D from video motion with experience.