Source-linked AI summary
SimPoE: Simulated Character Control for 3D Human Pose Estimation
Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, Jason Saragih
TL;DR
Monocular 3D human pose estimation must account for both kinematic geometry and physical dynamics, because kinematic estimates can be physically implausible. SimPoE jointly learns image-based pose refinement and physics-based character control with reinforcement learning, and achieves state-of-the-art pose accuracy while preserving physical plausibility. Its physics-based formulation, however, requires 3D scene modeling, limiting direct evaluation on in-the-wild datasets.
Problem
Accurate monocular 3D pose estimation requires both kinematics and dynamics, but existing methods largely focus on kinematics and can produce physically implausible motion.
Method
SimPoE jointly learns 2D-keypoint-based kinematic pose refinement and physics-based character control in a causal reinforcement-learning framework.
Results
SimPoE outperforms previous methods in pose-based and physics-based metrics on Human3.6M and an in-house motion dataset.
Takeaways & Limitations
The framework provides accurate and physically plausible monocular 3D pose estimation while running in real time and avoiding batch trajectory optimization.
Takeaways & Limitations
SimPoE depends on 3D scene modeling for contact constraints, hindering direct evaluation on in-the-wild datasets such as 3DPW.
Abstract
from arXiv · showhide
Accurate estimation of 3D human motion from monocular video requires modeling both kinematics (body motion without physical forces) and dynamics (motion with physical forces). To demonstrate this, we present SimPoE, a Simulation-based approach for 3D human Pose Estimation, which integrates image-based kinematic inference and physics-based dynamics modeling. SimPoE learns a policy that takes as input the current-frame pose estimate and the next image frame to control a physically-simulated character to output the next-frame pose estimate. The policy contains a learnable kinematic pose refinement unit that uses 2D keypoints to iteratively refine its kinematic pose estimate of the next frame. Based on this refined kinematic pose, the policy learns to compute dynamics-based control (e.g., joint torques) of the character to advance the current-frame pose estimate to the pose estimate of the next frame. This design couples the kinematic pose refinement unit with the dynamics-based control generation unit, which are learned jointly with reinforcement learning to achieve accurate and physically-plausible pose estimation. Furthermore, we propose a meta-control mechanism that dynamically adjusts the character's dynamics parameters based on the character state to attain more accurate pose estimates. Experiments on large-scale motion datasets demonstrate that our approach establishes the new state of the art in pose accuracy while ensuring physical plausibility.
1. Introduction
SimPoE addresses the gap between accurate but physically implausible kinematic estimates and computationally intensive physics-based methods by jointly learning kinematic refinement and dynamics-based control. It achieves accurate, physically plausible pose estimation through causal reinforcement learning and meta-PD control.
- Motivation: Kinematic methods estimate poses accurately but often produce jitter, foot sliding, and ground penetration that limit practical applications.These artifacts can mislead medical monitoring and sports training and hinder animation and virtual or augmented reality.
- Motivation: Trajectory-optimization approaches improve physical grounding but require complex, computationally intensive test-time optimization.Their batch nature makes them poorly suited to real-time use.
- Approach: SimPoE jointly integrates image-based kinematic inference and physics-based dynamics modeling in a causal temporal framework with an integrated physics simulator.The policy takes the current pose and next image frame and produces controls for a simulated proxy character to estimate the next-frame pose.
- Approach: A learnable refinement unit uses 2D keypoint reprojection information to refine the kinematic pose before the policy generates dynamics-based character controls.The refinement and control units are learned jointly with reinforcement learning.
- Results: On Human3.6M and an in-house motion dataset, SimPoE outperforms prior methods in both pose-based and physics-based metrics.The comparison includes kinematic and physics-based monocular 3D pose-estimation methods, with significant pose-accuracy improvement over prior physics-based approaches.
- Contributions: SimPoE uses meta-PD control to avoid manual dynamics-parameter tuning while maintaining real-time, physically plausible pose estimation.The paper reports that the approach is causal and runs without batch trajectory optimization.
2. Related Work
Prior work largely separates kinematic pose estimation from physics-based character control. SimPoE instead uses reinforcement-learned control to integrate precise motion tracking with real-time physical simulation.
- Kinematic 3D Human Pose Estimation: Prior kinematic methods estimate 3D joint locations from monocular video using two-stage or end-to-end frameworks.Parametric body models and optimization- or regression-based approaches are common representations and estimation strategies.
- Physics-Based Human Pose Estimation: Most physics-based pose-estimation methods use trajectory optimization to optimize physical forces that induce the motion observed in video.These methods address human dynamics but differ from SimPoE's learned control formulation.
- Physics-Based Human Pose Estimation: Trajectory optimization is a high-latency, computationally expensive batch procedure that cannot use advanced physics simulators with non-differentiable dynamics.It also lacks a learning mechanism aimed at matching optimized motion to ground truth.
- Physics-Based Human Pose Estimation: SimPoE integrates kinematic inference with reinforcement-learned character control to run in real time and match output motion to ground truth.This combination also supports advanced physics simulators with non-differentiable dynamics.
- Reinforcement Learning for Character Control: Deep reinforcement learning has been used for character control with manually designed rewards, adversarial imitation, hierarchical policies, and motion-capture data.These approaches target control, long-term behavior, or animation rather than the precise video-based tracking addressed here.
3. Approach
SimPoE jointly refines image-based kinematic poses and controls a physics-simulated character to produce accurate, physically plausible motion. Its policy uses next-frame visual cues, reinforcement learning, and adaptive PD control to connect pose inference with dynamics.
- Simulated Character Control: SimPoE initializes a proxy character from an estimated pose and uses current pose, velocities, next-frame kinematic pose, 2D keypoints, and confidences as policy state.The policy runs at 30Hz while the physics simulator runs at 450Hz, with each policy step corresponding to 15 simulation steps.
- Kinematics-Aware Policy: The kinematic refinement unit iteratively updates the next-frame pose, using a keypoint-reprojection gradient as a learned feature rather than explicitly minimizing reprojection loss.The feature is confidence-weighted and expressed in the character’s root coordinate, making it invariant to character orientation.
- Kinematics-Aware Policy: The control generation unit converts the refined pose into PD-controller target angles, while also producing root residual forces and torques and adaptive controller parameters.A residual connection sets target angles from the refined kinematic pose plus a learned update.
- Meta-PD Control: Meta-PD control dynamically adjusts PD stiffness and damping from the character state instead of relying on fixed parameters.Fixed parameter ratios can produce unstable, jittery motion or overly smooth motion that lags behind the ground truth.
4. Experiments
Experiments evaluate SimPoE on Human3.6M and an in-house dataset using pose-based and physics-based metrics, qualitative visualizations, and component ablations. SimPoE improves pose accuracy and physical plausibility over prior methods, while its components each contribute to performance.
- Datasets: Experiments use Human3.6M and an in-house dataset, with the latter containing detailed finger motion and more complex skeletons.Human3.6M provides 1.5 million training images; the in-house dataset contains approximately 335k training and 87k test frames.
- Metrics: Evaluation combines MPJPE and PA-MPJPE for pose accuracy with Accel, FS, and GP for jitter, foot sliding, and ground penetration.The metrics cover both geometric accuracy and physical artifacts.
- Implementation Details: The experiments use MuJoCo characters initialized from kinematic pose estimates, with calibrated camera intrinsics assumed for root-position recovery.The policy runs at 38 FPS on a standard Intel Core i9 PC.
- Comparison to state-of-the-art methods: SimPoE outperforms prior methods on both pose-based and physics-based metrics, nearly halving MPJPE relative to prior physics-based approaches on Human3.6M.It also reduces jitter, foot sliding, and ground penetration by large margins.
- Comparison to state-of-the-art methods: Qualitative comparisons show more accurate foot contact estimation without ground penetration in an alternative rendered view.The figure compares camera-view poses with the same poses rendered from another viewpoint.
- Ablation Studies: Ablations show that meta-PD control, kinematic refinement, residual connections, residual forces, and feature extraction each contribute to performance.Removing residual forces often causes the character to fall, producing a large acceleration error.
5. Discussion and Future Work
SimPoE jointly models kinematics and dynamics through reinforcement learning to improve monocular 3D pose accuracy and physical plausibility. Its physics-based formulation remains dependent on 3D scene modeling for contact constraints, limiting direct evaluation on some in-the-wild data.
- Discussion: Jointly modeling kinematics and dynamics improves the accuracy and physical plausibility of monocular 3D human pose estimation.SimPoE integrates image-based inference with physics-based character control in a joint reinforcement-learning framework.
- Discussion: SimPoE runs in real time and is compatible with advanced physics simulators while addressing drawbacks of prior physics-based approaches.The approach avoids batch trajectory optimization.
- Future Work: SimPoE depends on 3D scene modeling to enforce contact constraints during motion estimation.This hinders direct evaluation on in-the-wild datasets such as 3DPW, which includes motions like climbing stairs or trees.
A. In-House Motion Dataset and Kinematic Pose Estimator
The in-house dataset requires a complex subject-specific skeleton and a dedicated monocular kinematic tracker. Its estimator predicts keypoint heatmaps and pose parameters frame by frame using a cascaded network trained with multiple supervised losses.
- In-House Motion Dataset: The in-house dataset uses a skeleton with twice as many joints as SMPL, including fingers, and subject-specific skinning templates.Templates are recovered offline using multiview 3D reconstruction, nonrigid ICP, and linear blend skinning.
- Kinematic Pose Estimator: Because existing estimators cannot directly handle the dataset's complex skeletons and skinning models, the authors design a monocular tracker called KinPose.KinPose has no temporal component and outputs 2D keypoint heatmaps and joint angles frame by frame.
- Kinematic Pose Estimator: KinPose uses a three-stage cascaded ResNet-50 network with 77 keypoint heatmap channels and 100 pose-parameter channels.The pose channels represent 94 local joint angles and 6 global pose dimensions.
- Kinematic Pose Estimator: Training applies L2 losses to heatmaps, 2D keypoints, 3D joint positions, and joint angles at every network stage.These losses supervise both image-space outputs and pose estimation.
B. Additional Implementation Details
The experiments use dataset-specific hyperparameter settings and train neural policy components for kinematic refinement and control generation. Training uses subject-dependent protocols, finite-difference initialization, balance-based termination, and reinforcement-learning updates.
- Policy Network: The policy network uses two ReLU MLPs: Uθ for kinematic refinement and Vθ for control generation.Uθ has hidden-layer sizes (256, 512, 256), while Vθ uses (2048, 1024).
- Policy Training: Human3.6M uses one policy trained across training subjects and transferred to test subjects, whereas the in-house dataset uses subject-specific models.The in-house protocol addresses variation in body proportion and shape; baselines use the same training data.
- Initialization: Initial velocity is set to the kinematic velocity computed using finite differences.
- Policy Training: Episodes terminate at the end frame or when the character’s root height falls 0.5 below the kinematic pose’s root height.The height condition detects when the character has lost balance.
- Policy Training: The policy is trained for 2000 epochs, collecting 50000 time steps per epoch before updating the policy.The reward weights are (30, 0.2, 100, 0.02), and the value function uses Adam with step size 3 × 10^-4.