PaperlayerView exact revision on arXiv

Source-linked AI summary

Contact and Human Dynamics from Monocular Video

Davis Rempe, Leonidas J. Guibas, Aaron Hertzmann, Bryan Russell, Ruben Villegas, Jimei Yang

arXiv:2007.11678v2cs.CV

TL;DR

Monocular pose estimates can be accurate yet physically implausible, limiting uses that depend on contact and acceleration. The paper predicts foot contacts from 2D poses and applies physics-based trajectory optimization; it reports substantially more plausible motions than state-of-the-art methods.

  • Problem

    Monocular video pose methods can produce accurate overall poses but world-frame motions with physically implausible contacts, lean, and jitter.

    Source: §1 · p. 1

  • Method

    The method predicts foot contacts from 2D poses and uses reduced-dimensional physics-based trajectory optimization to refine initial kinematic estimates.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    The method substantially improves realism over state-of-the-art methods and evaluates motion with kinematic and dynamic plausibility metrics.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The method demonstrates physically more plausible motion for character animation and could support vision tasks that benefit from dynamical motion properties.

    Source: §5 · p. 14

    The method assumes unoccluded feet, a single ground plane, no interaction with other objects, and no contact from body parts such as knees or hands.

    Source: §5 · p. 14

Abstract

from arXiv · show

Existing deep models predict 2D and 3D kinematic poses from video that are approximately accurate, but contain visible errors that violate physical constraints, such as feet penetrating the ground and bodies leaning at extreme angles. In this paper, we present a physics-based method for inferring 3D human motion from video sequences that takes initial 2D and 3D pose estimates as input. We first estimate ground contact timings with a novel prediction network which is trained without hand-labeled data. A physics-based trajectory optimization then solves for a physically-plausible motion, based on the inputs. We show this process produces motions that are significantly more realistic than those from purely kinematic methods, substantially improving quantitative measures of both kinematic and dynamic plausibility. We demonstrate our method on character animation and pose estimation tasks on dynamic motions of dancing and sports with complex contact patterns.

1 Introduction

Monocular pose estimates can be kinematically accurate yet physically implausible, limiting downstream uses sensitive to contact and acceleration. The paper combines contact prediction with physics-based trajectory optimization and reports more plausible motion than state-of-the-art methods.

1 Introduction

  • World-frame estimates can show floating or penetrating feet, extreme forward or backward lean, and jitter despite accurate body-frame poses.The paper argues these physical inaccuracies can prevent later uses such as action, intention, and emotion inference or computer animation.Source: §1 · p. 1
  • Because full-body dynamics optimization is sensitive to local minima and contact events grow exponentially over time, the paper separates contact estimation from motion optimization.Contact is discontinuous, making joint optimization of contact and dynamics difficult.Source: §1 · p. 2
  • The method predicts foot contacts from video 2D poses, then uses reduced-dimensional centroidal dynamics and contact constraints to estimate COM motion, foot positions, and contact forces.A contact prediction network is trained on synthetic data, avoiding hand-labeled contact data.Source: §1 · p. 2
  • The method substantially improves motion realism over state-of-the-art methods and evaluates physical plausibility with kinematics and dynamics metrics.Its demonstrated scope includes single-person dance, walking, and sports, with character-animation retargeting.Source: §1 · p. 3

2 Related Work

Prior monocular pose and motion methods emphasize pixel-space or kinematic fit rather than physical plausibility. The paper distinguishes its approach through contact initialization and reduced-dimensional dynamics that include rotational motion.

2 Related Work

  • Monocular video methods use temporal cues to estimate 3D motion but do not account for physical plausibility, while several interaction methods rely only on static kinematic constraints.Zou et al. infer contact constraints to optimize 3D motion from video; this paper argues dynamics improve human-scene interaction inference.Source: §2 · p. 3
  • Brubaker and Fleet use trajectory optimization with a continuous contact approximation, but soft contacts bring inaccurate transitions and stiffness sensitivity, and their model omits rotational motion.The paper instead initializes contacts in preprocessing and models rotational inertia.Source: §2 · p. 3
  • Li et al. also estimate pose and contacts before trajectory optimization, but focus on human-object interactions and use hand-labeled classifier training data.This paper targets dynamic human motion with complex pose and foot-contact variation and does not consider human-object interaction.Source: §2 · p. 3
  • The optimization combines low-dimensional dynamics with 6D center-of-mass motion and contact constraints, while a classifier estimates contacts before optimization.These features capture rotational and footstep quantities without requiring full-body optimization.Source: §2 · p. 4

3 Physics-Based Motion Estimation

The method initializes motion from video, predicts foot contacts, and applies staged optimization to produce physically valid full-body motion. Its trajectory optimization uses reduced-dimensional dynamics, while contact learning and kinematic initialization provide its inputs.

3 Physics-Based Motion Estimation

  • The pipeline assumes a static camera and fully visible body, then combines contact estimates and initial poses in kinematic and physics-based optimization stages.The kinematic stage refines full-body motion and fits the ground before reduced-dimensional trajectory optimization applies dynamics.Source: §3 · p. 4

3.1 Physics-Based Trajectory Optimization

  • Reduced-dimensional optimization fits COM motion, orientation, feet, forces, and contact durations while enforcing momentum and foot-contact constraints.Staged optimization first fits the input without dynamics constraints, then adds dynamics and allows contact-phase durations to refine; the resulting COM and foot positions are upgraded to full-body motion with IK.Source: §3.1 · pp. 5, 7

3.2 Learning to Estimate Contacts

  • A network classifies toe and heel contacts from 2D pose sequences, trained on automatically labeled synthetic mocap rather than hand-labeled video.It uses a temporal window of lower-body joints and OpenPose confidence values; overlapping predictions are majority-voted to smooth labels.Source: §3.2 · pp. 7–8

3.3 Kinematic Initialization

  • Kinematic initialization starts with MTC 3D poses, then optimizes root translation, joint angles, and ground parameters to fit 2D and 3D inputs while enforcing smooth, grounded contacts.It estimates the ground plane from contact positions and derives COM and inertia trajectories from the resulting full-body motion.Source: §3.3 · p. 9

4 Results

Across synthetic and real videos, learned contact estimation outperforms the tested baselines, while physics optimization improves physical plausibility and retains comparable positional accuracy.

4.1 Contact Estimation

  • The all-lower-body-joint MLP is substantially more accurate than the tested baselines on synthetic and real videos; upper-body joints down to the knees also perform surprisingly well.The evaluation uses a synthetic test set of 78 videos and nine real videos with 700 manually labeled frames.Source: §4.1 · p. 9, Table 1 · p. 10
  • Optimization with network-predicted contacts converges on 94.9% of synthetic test videos, compared with 69.2% using the 3D-joint velocity heuristic.The comparison uses MTC input for the velocity heuristic.Source: §4.1 · pp. 9–10

4.2 Qualitative Motion Evaluation

  • The method fixes floating, penetration, and unstable leaning in a synthetic example, and reconstructs dynamic real-video motion with complex contact patterns.The evaluation also retargets captured motion to a virtual character; the synthetic example uses the true ground plane and an alternate-view reference.Source: §4.2 · pp. 10–11, Figure 5 · p. 11

4.3 Quantitative Motion Evaluation

  • Quantitative comparisons use MTC and a kinematic-only initialization on 78 synthetic videos, with the ground-truth floor supplied for fair evaluation.The method does not require a ground-truth floor; it is used here to isolate the primary contributions from floor fitting.Source: §4.3 · p. 12
  • Peak GRFs are 115–184% of body weight for walking and 127–204% for dancing, versus 319–510% and 765–1223% for kinematic-only motion.The authors report the optimized forces match force-plate data; Table 2 covers physical-plausibility measures on the synthetic test set.Source: §4.3 · p. 12, Table 2 · p. 13
  • Across the test set, physics estimates mean GRF within 1% of idle force and peak added-weight equivalent 174 kg versus 830 kg kinematic; kinematic ballistic GRF is 255%.Ballistic GRF measures unexplained force when no feet should be in contact.Source: §4.3 · pp. 12–13, Table 2 · p. 13
  • Foot plausibility metrics count floating beyond 3 cm, penetration beyond 3 cm, and skate exceeding 2 cm while a joint should be in contact.The evaluation uses ground-truth foot-contact labels.Source: §4.3 · p. 13, Table 2 · p. 13
  • Positional errors across methods are comparable, differing by at most 5 cm, so improving physical plausibility does not negatively affect standard pose measures.The positional evaluation includes synthetic data and HumanEva-I walking sequences.Source: §4.3 · pp. 13–14, Table 3 · p. 14

5 Discussion

The method estimates physically valid motion from initial kinematic poses, producing more plausible results and supporting retargeting and further vision tasks that benefit from dynamics. Its scope is constrained by motion-estimation unknowns and simplifying assumptions, and optimization is expensive.

5 Discussion

  • The method estimates physically valid motions from kinematic pose estimates and produces motions that are more visually and physically plausible than state-of-the-art methods.The paper demonstrates character retargeting and notes potential use in vision tasks benefiting from dynamical properties.Source: §5 · p. 14
  • The work focuses on one subproblem; partially occluded people and ground-plane position are among the other motion-estimation unknowns reserved for future work.The authors describe these problems as substantial challenges in their own right.Source: §5 · p. 14
  • The approach assumes unoccluded feet, a single ground plane, no interaction with other objects, and no knee or hand contacts; optimization takes 30 minutes to 1 hour for a 2-second, 60-frame clip.These assumptions are considered permissible for character animation from video mocap, but relevant to general motion estimation; runtime is attributed primarily to an ill-suited adapted solver implementation.Source: §5 · p. 14

Appendices

The appendices provide details of the contact-estimation model and dataset described in Section 3.2.

Appendices

A.1 Synthetic Dataset

The appendices describe a synthetic dataset and contact-estimation model, then introduce the kinematic optimization used to initialize physics-based optimization.

A.1 Synthetic Dataset

  • The synthetic dataset contains 1,690 videos and 101,000 frames from 13 characters performing 65 motion-capture sequences across dance, sports, walking, and idle motions.It includes RGB frames, 2D OpenPose detections, 3D skeleton poses, heel and toe-base contact labels, and camera parameters; camera, floor texture, and lighting vary across videos.Source: §A.1 · p. 18

A.2 Model Details

  • The contact-estimation MLP classifies contacts jointly over five-frame windows, aggregates overlapping test-time predictions, and marks a joint in contact by majority vote.The synthetic examples include RGB frames, OpenPose detections, skeleton poses, and automatically labeled toe-base and heel contacts.Source: §A.2 · pp. 18–19, Figure A1 · p. 19

B Kinematic Optimization Details

  • The appendix details the kinematic optimization procedure used to initialize the physics-based optimization.It refers readers to Section 3.3 for an overview.Source: B Kinematic Optimization Details · p. 19

B.1 Inputs and Initialization

The kinematic stage fits pose estimates to a fixed-length skeleton and contact constraints, then supplies motion targets to a staged physics-based trajectory optimization.

B.1 Inputs and Initialization

  • The kinematic initialization uses 2D OpenPose detections, estimated contacts, and full-body 3D MTC poses, with a 28-joint skeleton and fixed bone lengths.It preprocesses low-confidence detections by interpolation and uses inverse kinematics to reconstruct the MTC input on its skeleton.Source: §B.1 · pp. 19–20

B.2 Optimization Variables

  • The optimization variables include global root translation, joint Euler angles, and separately fitted ground-plane parameters.The variables are optimized in stages rather than all at once.Source: §B.2 · p. 20

B.3 Problem Formulation

  • The objective includes a 2D reprojection error that minimizes joint deviation from corresponding OpenPose detections weighted by confidence, and velocity smoothing that minimizes changes in joint positions and angles over time.The weights are αproj = 0.5, αvel = αang = 0.1, αacc = 0.5, αdata = 0.3, and αcont = αfloor = 10.Source: §B.3 · pp. 20–21

B.4 Optimization Algorithm

  • Optimization first fits pose without contact-position error, then robustly fits the floor and rejects outlier contacts, and finally enables the full contact-position objective.The final optimization uses Trust Region Reflective with analytical derivatives.Source: §B.4 · pp. 21–22

B.5 Extracting Inputs for Physics-Based Optimization

  • The physics-based stage receives center-of-mass targets, body-frame inertia tensors, root orientation, and four foot positions from the kinematic output.Body parts are treated as points with predefined masses, and the character is assumed to have a mass of 73 kg unless otherwise noted.Source: §B.5 · p. 22

C Physics-Based Optimization Details

C.1 Polynomial Parameterization

  • Cubic Hermite polynomials parameterize center-of-mass motion and orientation, foot positions during flight, and stance contact forces.Stance foot positions are constant and flight contact forces are zero; each phase uses at least six foot-position and force polynomials.Source: §C.1 · p. 22

C.2 Constraint Parameters

  • Kinematic and foot constraints are enforced every 0.08 seconds, while floor and dynamics constraints are enforced every 0.1 seconds.Velocity boundary constraints use mean initial and final velocities over five frames to reduce sensitivity to noisy motion.Source: §C.2 · pp. 22–23

C.3 Contact Timing Optimization

Contact-phase durations are deferred to the final optimization stage because optimizing them with dynamics is harder and less stable. The full-body output uses IK to combine optimized contacts and center-of-mass motion with the input upper-body pose.

C.3 Contact Timing Optimization

  • Contact-phase durations are optimized only at the final stage; the authors select whichever of fixed timings or subsequently adjusted durations improves the motion.Jointly optimizing durations with dynamics does not always improve the solution and is inherently harder and less stable.Source: §C.3 · p. 23

C.4 Full-Body Output

  • IK targets the optimized center of mass and preserves input upper-body offsets, while targeting optimized toe and heel positions for the lower body.The resulting lower-body joints, including ankles, knees, and hips, can differ substantially from the input.Source: §C.4 · p. 23

D Retargeting to a New Character

Retargeting scales the source skeleton and uses IK to fit its joint angles to a target skeleton before physics optimization. The supplementary evaluation reports contact-estimation window comparisons, off-target accuracy, and additional qualitative examples.

D Retargeting to a New Character

  • Retargeting uniformly scales the source skeleton, then uses mapped-joint IK before running physics optimization and full-body output on the target skeleton.The method optimizes directly on the target character skeleton rather than applying a separate physics-based retargeting step afterward.Source: D Retargeting to a New Character · p. 23

E.1 Contact Estimation

  • Precision, recall, and F1 supplement contact-estimation accuracy, providing additional insight because contact labels are slightly imbalanced.Table E1 presents these measures for baseline comparisons and feature-joint ablations.Source: §E.1 · p. 24, Table E1 · p. 24
  • The 9-frame input and 5-frame prediction windows achieve the best contact-classification accuracy on real videos among the tested combinations.Accuracy generally rises with input-window size, while prediction-window size shows no clear trend; Table E2 reports the combinations.Source: §E.1 · p. 24, Table E2 · p. 24
  • Contact predictions remain accurate across the five-frame output window, motivating majority voting at inference time.Although frame index 2 is the target, predictions on the off-target frames degrade only slightly; Figure E1 depicts synthetic-test accuracy across the window.Source: §E.1 · p. 24, Figure E1 · p. 25

E.2 Qualitative Motion Evaluation

  • The supplementary video provides extensive qualitative evaluation using licensed or permitted real-world footage and publicly available datasets.The real-data videos come from publicly available datasets, permitted YouTube videos, and licensed stock footage.Source: §E.2 · p. 25

E.3 Quantitative Motion Evaluation

Quantitative motion evaluations use the ground-truth floor plane, although the method can also fit the floor. The discussion notes that global-motion comparisons are limited and that MTC provides smoother initialization than SMPLify-X.

E.3 Quantitative Motion Evaluation

  • Quantitative motion evaluations use the ground-truth floor plane, while fitted-floor results on the synthetic test set show trends similar to the ground-truth-floor results.Tables E3 and E4 report physical-plausibility and pose evaluations, respectively, using the fitted floor.Source: §E.3 · p. 25, Table E3 · p. 26, Table E4 · p. 26
  • Moving toward or away from the camera can leave MTC's input feet joints noisy, causing poor floor fits that complicate optimization and evaluation.The authors note that their method does not require a ground-truth floor and that floor fitting works well in qualitative live-action results.Source: §E.3 · p. 25

E.4 Discussion on Global Pose Estimation

  • The quantitative evaluation compares the kinematics-only and MTC-initialized versions of the authors' method rather than other global-motion methods.The authors cite the scarcity of comparable prior work on temporally consistent global motion; many methods instead estimate local pose or root translation without coherent global motion.Source: §E.4 · pp. 25–26
  • MTC's temporal tracking gives smoother, more coherent global motion than SMPLify-X on the compared clip, motivating its use to initialize the pipeline.Figure E2 shows the fixed-side-view comparison; SMPLify-X estimates camera extrinsics, local pose, and body shape when applied to video.Source: §E.4 · p. 26

E.5 Pose Estimation Evaluation Details

Pose evaluation covers a synthetic test set and selected HumanEva-I walking sequences, while additional results test recovery across characters with varied mass and mass distributions.

E.5 Pose Estimation Evaluation Details

  • Pose evaluation uses a synthetic test set and HumanEva-I walking sequences selected for dynamic motion and interesting foot-contact patterns.HumanEva sequences are divided into contiguous chunks after corrupted motion-capture frames are removed, then split into roughly 120-frame inputs.Source: §E.5 · p. 26
  • In the fixed-side-view comparison, MTC's tracking refinement produces smoother motion than SMPLify-X's noisy, inconsistent global motion.For the synthetic set, the evaluation uses 16 rig joints; HumanEva uses an adapted 15-joint skeleton and ground-truth floor plane.Source: §E.5 · pp. 26–27, Figure E2 · p. 27

E.6 Multi-Character Generalization

  • The method recovers motion across characters with different body masses and mass distributions, including a 36.5 kg character with 40% of its mass in the head and a 146 kg character with 36% in its arms.The illustrated characters are Ybot (73 kg), Ty (36.5 kg), and Skeleton Zombie (146 kg); Figure E3 shows the examples.Source: §E.6 · p. 27, Figure E3 · p. 27

Related papers

Loading 2007.11678v2…