PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-based Human Motion Estimation and Synthesis from Videos

Kevin Xie, Tingwu Wang, Umar Iqbal, Yunrong Guo, Sanja Fidler, Florian Shkurti

arXiv:2109.09913v2cs.CV

TL;DR

Human-motion synthesis methods often depend on costly motion-capture data, while video-based pose estimates can be noisy and motion models may neglect physics. This paper refines monocular-video pose estimates with contact-aware physics optimization and uses the corrected motions to train a generative model. The authors report improved pose estimation, synthesis quality, and physical plausibility, with crouching motions remaining a limitation of their character geometry.

  • Problem

    Motion-synthesis methods commonly rely on costly motion-capture data, and many sequence models do not enforce physics constraints or contact.

    Source: §1 · p. 1

  • Method

    The framework refines noisy monocular-video pose estimates with physics constraints and contact forces, then trains a time-series generative model on the refined motions.

    Source: §1 · p. 1

  • Results

    The authors report that physics-refined visual pose estimates can train synthesis models approaching motion-capture prediction quality, with improved motion-estimation and synthesis results on Human3.6m.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework trains motion-synthesis models from raw video pose estimates without motion-capture data.

    Source: §5 · p. 8

    Crouching motions expose limits of the geometric character approximation: it misses true foot geometry and cannot represent significant foot flexion, although the end effectors remain engaged with the ground.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Human motion synthesis is an important problem with applications in graphics, gaming and simulation environments for robotics. Existing methods require accurate motion capture data for training, which is costly to obtain. Instead, we propose a framework for training generative models of physically plausible human motion directly from monocular RGB videos, which are much more widely available. At the core of our method is a novel optimization formulation that corrects imperfect image-based pose estimations by enforcing physics constraints and reasons about contacts in a differentiable way. This optimization yields corrected 3D poses and motions, as well as their corresponding contact forces. Results show that our physically-corrected motions significantly outperform prior work on pose estimation. We can then use these to train a generative model to synthesize future motion. We demonstrate both qualitatively and quantitatively improved motion estimation, synthesis quality and physical plausibility achieved by our method on the Human3.6m dataset~\cite{h36m_pami} as compared to prior kinematic and physics-based methods. By enabling learning of motion synthesis from video, our method paves the way for large-scale, realistic and diverse motion synthesis. Project page: \url{https://nv-tlabs.github.io/publication/iccv_2021_physics/}

1. Introduction

The paper targets motion synthesis from monocular video as an alternative to costly motion-capture training data, while aiming to enforce physical plausibility. It refines image-based poses with contact-aware physics optimization, then uses the results to train a generative model.

1. Introduction

  • Motion-capture training data are costly and logistically challenging, while many synthesis models ignore physics constraints and can produce implausible motion.The paper also notes that recorded motions are often limited to indoor settings and predefined actions.Source: §1 · p. 1
  • The framework trains physically plausible human-motion synthesis directly from monocular RGB video by refining noisy image-based pose estimates with physics constraints.The optimization reasons about contact and computes contact forces.Source: §1 · pp. 1–2
  • A smooth contact loss enables physics-based pose refinement without separately trained contact detectors or nonlinear programming solvers.Source: §1 · p. 2
  • Without motion-capture data, visual pose estimation combined with physics-based optimization can train synthesis models that approach the quality of motion-capture prediction models.Source: §1 · p. 2
  • On Human3.6m, the authors report improved synthesis quality and physical plausibility versus prior learning-based motion-prediction methods.The cited comparisons include PhysCap [44], HMR [16], HMMR [58], and VIBE [18].Source: §1 · p. 2

2. Related Works

The related work spans kinematic and physics-based motion synthesis, alongside kinematic and physics-based pose estimation. The paper distinguishes its approach by allowing contact events to form softly during optimization without separate contact labels.

2. Related Works

  • The related-work review organizes motion synthesis along kinematic versus physics-based methods and imitation learning versus model-based control and reinforcement learning.Source: §2 · p. 2

2.1. Kinematic Motion Synthesis

  • Kinematic synthesis includes nonparametric clip blending and parametric time-series models, which generate predictions without necessarily satisfying physics constraints.Examples include motion matching, motion graphs, motion fields, and learned sequence models [7] [33] [54] [8].Source: §2.1 · p. 2

2.2. Physics-Based Motion Synthesis

  • Physics-based synthesis enforces body dynamics and often contact; related approaches include soft contact-invariant optimization, hard-constraint reinforcement learning, and simplified dynamics models.The table compares features of related works and notes that RFC uses a physics simulator without proper contact dynamics.Source: §2.2 · p. 2, Table 1 · p. 2

2.3. Kinematic and Physics-Based Pose Estimation

  • Kinematic pose estimators do not incorporate physics constraints, while prior video-based physics methods include contact modeling or hard contact constraints.The paper contrasts its approach with PhysCap [44] and methods such as [43].Source: §2.3 · p. 3
  • The proposed soft contact penalty lets contact events form dynamically during optimization, without separate labels or alternating discrete relabeling steps.Optimization runs in two contiguous passes with an unconstrained LBFGS optimizer.Source: §2.3 · p. 3

3. Method

The method turns monocular-video pose estimates into refined motions using a differentiable physics optimization, then trains a generative model on those motions. Its pipeline combines pose fitting, smoothness, contact-aware physics losses, and spline-based motion representations.

3. Method

  • The pipeline estimates 2D and 3D joints from video, converts 3D joints to SMPL rotations with inverse kinematics, refines motion with physics optimization, and trains a synthesis model on the resulting motions.Figure 2 depicts the pose-estimation, initialization, optimization, and training stages.Source: §3 · p. 3, Figure 2 · p. 3

3.1. 3D Pose Estimation

  • Framewise pose estimates have scale ambiguity and spatial and scale jitter, motivating refinement beyond the initial mean-bone-length scale estimate.Source: §3.1 · p. 3

3.2. Motion Representation and Initialization

  • The method represents motion with SMPL local rotations and global root motion, then fits cubic Hermite splines to smooth motion and reduce optimization variables.Spline knots subsample the sequence by a factor of eight; optimizing their timing had negligible impact on final results.Source: §3.2 · pp. 3–4

3.3. Motion Optimization

  • Motion optimization jointly fits body shape, poses, and spline-parameterized contact forces to pose detections using physics, pose-estimation, and smoothness losses.One body-shape parameter set is optimized for the entire sequence.Source: §3.3 · p. 4

3.3.1 Physics Loss

  • The physics loss combines dynamics, contact, and interpenetration terms for motion frames and their global contact forces.Inverse dynamics maps motion to required generalized forces; the implementation uses Recursive Newton Euler and finite-difference time derivatives.Source: §3.3.1 · p. 4
  • Contact penalties scale end-effector distance and its time derivative by a force-related contact variable, while a separate term penalizes interpenetration.The contact variable ranges from zero to one; slipping and rolling contacts are not considered.Source: §3.3.1 · p. 5

3.3.2 Pose Estimation Loss

  • The pose-fitting loss combines local 3D keypoint error, camera-projected 2D keypoint error, pose-prior likelihood, and deviation from mean body shape, with an additional kinematic acceleration penalty for smoothness.Source: §3.3.2 · p. 5

3.3.3 Implementation Details

  • Optimization uses two stages totaling 750 iterations: 250 kinematic iterations followed by 500 physics-enabled iterations.The LBFGS memory is cleared between stages.Source: §3.3.3 · p. 5

4. Experimental Results

Across Human3.6M, physics-corrected pose estimates improve pose and contact metrics and support motion synthesis that remains close to a mocap-trained oracle. The evaluation also identifies crouching-related geometry as a failure boundary.

4.1. Dataset and Experimental Setting

  • Evaluations use Human3.6M, with subjects 9 and 11 as the standard validation set and the same motions as PhysCap, excluding chair interactions and lying or sitting motions.The dataset includes four camera views and motion-capture annotations.Source: §4.1 · pp. 5–6

4.2. Physics-corrected Pose Estimation

  • Our method greatly outperforms PhysCap on root-aligned mean joint position error without Procrustes alignment and approaches video pose estimators trained with AMASS motion capture.The kinematic baseline is competitive with HMR and HMMR on its own.Source: §4.2 · p. 6, Table 3 · p. 6
  • Jointly optimizing motion, body shape, and contact-aware physics lets temporal 2D detections refine bone lengths, unlike PhysCap’s separate iterative stages.The method uses one differentiable objective optimized jointly over the variables, including shape parameters.Source: §4.2 · p. 7
  • Physics loss reduces foot tangential velocity error by more than 40% and foot global height error by 80%, while improving MPJPE, global root position, and esmooth.The table compares contact-sensitive metrics with and without physics loss.Source: §4.2 · p. 7, Table 4 · p. 7
  • Crouching is a major error case because the box-shaped character approximation misses true foot geometry and cannot represent significant foot flexion.Despite this limitation, the character pose remains stable and its end effectors stay engaged with the ground plane.Source: §4.2 · p. 7

4.3. Motion Synthesis from Video

  • Table 5: DLow trained on physics-corrected poses is worse than the mocap oracle by 16.9% in average joint distance, 11.0% in final distance error, and 10.3% in average motion diversity.Adding physics loss consistently improves all evaluated metrics for both DLow and cVAE; training settings were held constant except for input data, with monocular inputs from the first camera.Source: Table 5 · p. 8, §4.3 · p. 8

5. Conclusion

The framework trains motion-synthesis models from video pose estimates without motion-capture data, refining poses with physics constraints and contact forces before synthesizing future motion and forces.

5. Conclusion

  • The framework refines noisy video pose estimates with physics constraints and contact forces, then trains a time-series generative model to synthesize future motion and forces.The authors report performance boosts in both pose estimation and motion synthesis.Source: §5 · p. 8

A. Detailed Loss Formulation

The optimization derives motion velocities and accelerations using implicit finite differences, and combines pose and physics losses to refine motion while constraining contacts and forces.

A. Detailed Loss Formulation

  • Velocities and accelerations for the physics losses are obtained with a finite-difference scheme corresponding to implicit integration.Source: A. Detailed Loss Formulation · p. 11
  • Contact variables depend on contact force, with stiffness controlling how closely the soft contact loss approaches a hard complementarity constraint.An offset makes the contact variable zero when contact force is zero.Source: A. Detailed Loss Formulation · p. 11
  • Contact-force penalties enforce the friction cone and limit each contact point to at most whole-body weight.The stated cap allows each foot to generate total force up to four times body weight.Source: A. Detailed Loss Formulation · p. 11
  • The pose loss combines a per-frame SMPL prior, robust 2D reprojection error, and local 3D keypoint error with global root position removed.The 2D term uses the true camera projection matrix; the 3D term compares relative keypoint positions.Source: A. Detailed Loss Formulation · p. 11

B. Rigid Body Human Body Model

The rigid-body model uses geometrically scaled primitives with mass and inertial properties based on constant density, but its shape representation does not capture body-mass differences between similar skeletons.

B. Rigid Body Human Body Model

C. Additional Comparison

On HumanEva, the method outperforms the comparison method in every reported pose-accuracy metric; the evaluation uses short walking sequences and includes foot and whole-body position measures.

C. Additional Comparison

  • The comparison uses 15 two-second HumanEva walking sequences, with a 6 cm ground-plane offset applied to account for the observed z offset.The evaluation adapts a version of [43] using the authors’ pose-estimator outputs for a fairer comparison.Source: C. Additional Comparison · p. 12
  • The Feet metric measures global position error for the two foot joints, while Body measures whole-body global position error.Body-Align 1 aligns the first-frame root position and is sensitive to its error.Source: C. Additional Comparison · p. 12
  • The method greatly outperforms [43] in every pose-accuracy metric.Source: C. Additional Comparison · p. 12
Loading 2109.09913v2…