PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation

Zhengyi Luo, Ryo Hachiuma, Ye Yuan, Kris Kitani

arXiv:2106.05969v3cs.CVcs.AIcs.LGcs.RO

TL;DR

A single front-facing wearable camera often cannot see the wearer’s body, and recovering object interactions adds scene-context challenges. The paper integrates kinematic pose targets with a learned dynamics controller and simulator feedback. Evaluations report better performance than prior methods on pose-based and physics-based metrics, including real-world scenarios.

  • Problem

    Kinematic estimates can violate physical constraints, while dynamics controllers are hard to learn and generalize poorly, complicating recovery of unseen-body motion and object interaction from egocentric video.

    Source: §1 · pp. 1–2

  • Method

    A kinematic policy provides per-step target poses to a learned humanoid controller in simulation, then uses simulation feedback to refine its targets.

    Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 6

  • Results

    The method outperforms state-of-the-art methods on pose-based and physics-based metrics in controlled and real-world evaluations and generalizes to real-world scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Real-world experiments indicate that 3D human poses and human-object interactions can be estimated from an egocentric view captured by consumer hardware.

    Source: §5.2 · p. 10

    The method is limited to predefined interactions represented in training data and assumes known object category, rough size, and potential interaction mode as priors.

    Source: §3.2 · p. 5, §5.1 · p. 10

Abstract

from arXiv · show

We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamics-regulated training. At each timestep, a kinematic model is used to provide a target pose using video evidence and simulation state. Then, a prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model. By factoring in the 6DoF pose of objects (e.g., chairs, boxes) in the scene, we demonstrate for the first time, the ability to estimate physically-plausible 3D human-object interactions using a single wearable camera. We evaluate our egocentric pose estimation method in both controlled laboratory settings and real-world scenarios.

1 Introduction

Egocentric pose estimation must recover unseen body motion and object interaction from a single front-facing camera, while existing kinematic and dynamics approaches each have limitations. The paper combines them through dynamics-regulated training, with experiments reporting gains over prior methods.

1 Introduction

  • The task infers the wearer’s global 3D full-body pose and object interactions from a single head-mounted camera, despite the body often being unseen.Source: §1 · pp. 1–2
  • Kinematic methods can violate physical constraints, whereas dynamics-based controllers produce plausible motion but are difficult to learn and generalize poorly.Source: §1 · p. 2
  • Dynamics-regulated training uses a kinematic policy to guide a learned humanoid controller and adjusts targets using feedback from the resulting simulation state.The policy and controller are jointly optimized through supervised learning and reinforcement learning.Source: §1 · p. 2
  • Experiments on controlled MoCap and real-world datasets report that the model outperforms state-of-the-art methods on pose-based and physics-based metrics and generalizes to real-world scenarios.Source: §1 · p. 2

2 Related Work

Related work spans third-person pose estimation, egocentric pose estimation, and simulated humanoid control. The paper distinguishes its approach by integrating kinematics and dynamics for egocentric human-object interaction.

Third-person human pose estimation

  • Third-person pose methods are largely kinematic and root-relative, while existing dynamics-based approaches do not faithfully model human-object interaction; SimPoE separates its kinematic model from simulation state.Source: §2 · p. 2

Egocentric human pose estimation

  • Egocentric methods often assume visible body parts or address limited interactions; prior dynamics-based work notes absolute-position errors, while this work covers broader motion and object interaction.Source: §2 · p. 3

Humanoid control inside physics simulation

  • Prior motion-imitation controllers are often limited to single clips or high-quality MoCap, whereas this controller is designed for everyday motion and interactions without task-specific fine-tuning.Source: §2 · p. 3

3 Method

The method combines an object-aware kinematic pose estimator with a pretrained physics-based humanoid controller, using simulated feedback and an RL loss to keep predicted poses aligned with observed motion and followable by the controller. At test time, it rolls out the policy on images using the mean action to obtain simulated poses.

3 Method

  • The framework trains a kinematic policy with a pretrained controller in a closed loop: simulated humanoid motion informs the policy’s next-frame pose estimate.At test time, the policy supplies target motion to the controller, and the resulting simulated state feeds back into the next prediction.Source: §3 · p. 4, §3.3 · p. 6

3.1 Dynamics Model - Universal Humanoid Controller (UHC)

  • The UHC learns task-agnostic motion-following skills from pose-only AMASS data, using the current simulated pose and velocity plus a target pose as its state inputs.Its state omits phase or sequence-level information, enabling training on a larger and more diverse motion dataset.Source: §3.1 · p. 4

3.2 Kinematic Model – Object-aware Kinematic Policy

  • The object-aware kinematic policy estimates initial pose and per-frame motion from egocentric inputs, including scene-object states and camera trajectory or image features.It assumes known object category, rough size, and potential interaction mode; the paper reports that using camera trajectory improves performance over noisy visual input.Source: §3.2 · p. 5

3.3 Dynamics-Regulated Training

  • Dynamics-regulated training feeds the simulated controller pose, rather than only the kinematic prediction, into the next policy step and optimizes the policy using reinforcement learning and supervised learning.The simulated state provides physically plausible, diverse training inputs; reinforcement learning combines motion imitation with self-supervision based on how readily the controller can follow the predicted pose.Source: §3.3 · pp. 6–7
  • The RL loss is particularly helpful for adapting to challenging real-world sequences requiring adjustment to domain shifts and unseen motion.The passage points to Sec. 4.2 for this finding.Source: §3.3 · p. 7
  • At test time, the policy is rolled out on image sequence I1:T to obtain simulated pose q1:T.The procedure follows Alg. 2 and Fig. 2.Source: §3.3 · p. 7
  • At test time, the policy uses the mean action from πstep KIN(ct) instead of sampling from its Gaussian policy.Source: §3.3 · p. 7

4 Experiments

Experiments compare the method with prior approaches on controlled MoCap and real-world datasets using pose and physics metrics. The results favor the proposed method overall, while the ablations examine simulation-state training, reinforcement learning, and camera-pose inputs.

4 Experiments

  • The MoCap dataset has 266 sequences across five actions and an 80–20 train-test split; the real-world test set has 183 sequences recorded with a head-mounted iPhone.Evaluation uses pose and physics metrics, including camera tracking error when real-world ground-truth pose is unavailable; the MoCap data itself has 7.182 mm penetration and 2.035 mm foot sliding per frame.Source: §4 · pp. 7–8

4.1 Results

  • On MoCap, the purely kinematic model performs best on pose metrics, while dynamics-regulated training excels on physics metrics.Both proposed kinematic variants outperform the two state-of-the-art methods across all metrics.Source: §4.1 · pp. 8–9
  • Across MoCap and real-world tests, our method outperforms baselines in almost all metrics; MoCap interaction success is 48.4% for EgoPose versus 96.9% for our method.The real-world comparison reports an advantage in almost all metrics, with EgoPose showing less foot-skating but very low interaction success.Source: §4.1 · pp. 8–9, Table 1 · p. 9

4.2 Ablation Study

  • Real-world ablations associate higher interaction success with simulation-state exposure and the RL loss, while extracted VIO camera poses reduce root drift relative to noisy image features.Simulation-state exposure raises success from 73.2% to 80.9%; the study also reports a benefit from the RL loss and from using camera poses as an input modality.Source: §4.2 · pp. 9–10, Table 2 · p. 9

5 Discussions

The method performs well in the MoCap studio and generalizes to real-world settings, but remains limited by interaction coverage, object inputs, and domain shift. The authors frame the work as a physics-grounded framework and identify broader action and object support as future work.

5.1 Failure Cases and Limitations

  • The method is limited to predefined interactions with training data, relies on externally computed object class and pose, and discards object geometry beyond 6DoF pose.The lower real-world success rate indicates susceptibility to covariate shifts, and the controller can fail at extreme motion.Source: §5.1 · p. 10

5.2 Conclusion and Future Work

  • The authors report outperforming prior methods and demonstrating egocentric 3D pose and interaction estimation with consumer hardware, while identifying broader action support and robustness as future goals.They also suggest applying dynamics-regulated training to visual navigation and third-person pose estimation.Source: §5.2 · p. 10

A Qualitative Results (Supplemantry Video)

The supplementary video provides timestamped qualitative evaluations covering real-world results, comparisons with prior methods, and failure cases.

A Qualitative Results (Supplemantry Video)

B.1 Evaluation Metrics Definition

The appendix defines pose, physics, and interaction metrics, describes a fail-safe for dynamics-based evaluation, and points to the kinematic-policy training hyperparameters.

B.1 Evaluation Metrics Definition

  • Root error compares estimated and ground-truth root transformations, while MPJPE is root-relative and computed after zeroing root translation.Root error reflects position and orientation tracking; MPJPE is a 3D pose metric.Source: §B.1 · p. 14
  • Physics metrics include acceleration error, foot sliding, and penetration; small penetration is expected under MuJoCo’s soft-contact model.Foot sliding is computed similarly to Ling et al..Source: §B.1 · pp. 14–15
  • Interaction success is zero if the humanoid falls, and otherwise depends on action-specific contact, object movement, or ending-position criteria.The criteria cover sitting, pushing or stepping on a box, and avoiding an obstacle.Source: §B.1 · p. 15

B.2 Fail-safe during evaluation

  • When a dynamics-based humanoid fails mid-episode, evaluation resets the simulation using the estimated kinematic pose at the failure timestep.Failure is identified by thresholding the difference between reference and simulated joint positions.Source: §B.2 · p. 15

B.3 Implementation Details

  • Table 3 lists the hyperparameters used to train the kinematic policy.Source: Table 3 · p. 15

B.4 Additional Experiments about Stochasticity

Across three random seeds, the dynamics-regulated kinematic policy shows low stochasticity and maintains high performance on both MoCap and real-world data. Stepping is consistently the hardest action, while avoiding is the easiest.

B.4 Additional Experiments about Stochasticity

  • Across three random seeds, the method maintains high performance with very small stochasticity on both the MoCap test split and the real-world dataset.Table 4 reports results with error bars for the two datasets.Source: §B.4 · p. 15, Table 4 · p. 16
  • The stochasticity experiment trains the full pipeline with three random seeds and evaluates it on the MoCap test split and real-world dataset.The results are reported with error bars in Table 4.Source: §B.4 · p. 15, Table 4 · p. 16
  • Stepping is consistently the hardest action and avoiding the easiest across random seeds.Stepping requires coordination between the kinematic policy and UHC to lift the feet and push up; avoiding requires only basic locomotion skills.Source: §B.4 · p. 16

B.5 Additional Analysis into low Per Joint Error

The per-joint analysis finds larger errors at the toes and hands, while torso estimates can be easier from an egocentric view. The authors relate low MoCap MPJPE to task scope and the use of physical and scene context.

B.5 Additional Analysis into low Per Joint Error

  • Toes and hands have much larger positional errors than other joints in the reported per-joint analysis.The analysis reports the four joints with the smallest and largest errors in ascending order.Source: §B.5 · p. 16, Table 5 · p. 16
  • Inferring hand and toe movement from egocentric video is challenging, while the network can extrapolate positions using physical laws and scene context.Torso movement is highly correlated with head motion, making the torso easier to estimate from an egocentric view than in third-person pose estimation.Source: §B.5 · p. 16
  • The authors attribute low MoCap MPJPE to modeling only a subset of actions and interactions, the egocentric viewpoint, and physical laws and scene context reducing possible trajectories.Source: §B.5 · p. 16

C.1 Implementation Details

The implementation pairs a simulated humanoid controller with a kinematic policy, then evaluates controller imitation on AMASS. On the cleaned AMASS set, the controller imitates 96.964% of realistic motions without falling.

C.1 Implementation Details

  • The simulated proxy humanoid is built from SMPL geometry and kinematic structure, uses the mean SMPL body shape, and has its starting height manually adjusted for ground contact.Bone meshes use convex hulls of assigned vertices, with bone mass defined by mesh volume.Source: §C.1 · p. 16
  • The UHC uses an eight-primitive multiplicative compositional policy, with MLPs for the primitives and composer, to combine motion skills.The primitive MLPs have two hidden layers of 512 and 256 units; the composer has layers of 300 and 200 units.Source: §C.1 · p. 16, Figure 4 · p. 17
  • The policy output combines target angles for 23 non-root joints with a six-dimensional residual force, while its input feature vector is 640-dimensional.The controller workflow maps simulation state and the next target pose to actions.Source: §C.1 · p. 17
  • The imitation reward covers joint rotation, position, and velocity errors, and penalizes reliance on external residual force.The reward terms apply across all humanoid joints, including the root, in world coordinates.Source: §C.1 · pp. 17–18
  • The UHC is trained for 10000 epochs, a process taking about five days.Source: §C.1 · p. 18
  • The UHC training set excludes human-object-interaction sequences because AMASS lacks object information needed to recreate those interactions faithfully.Heuristics and visual inspection reduce the dataset from 13944 sequences to 11299 sequences without human-object interaction.Source: §C.1 · p. 18

C.2 Evaluation on AMASS

  • 10956/11299 (96.964%) of realistic AMASS motions are imitated with high fidelity without the humanoid falling.Failure cases include challenging motions such as breakdancing and cartwheeling.Source: §C.2 · p. 18

C.3 Evaluation on H36M

The UHC is tested on H36M motions unseen during its training and reportedly imitates them with high accuracy and success, significantly outperforming the baseline. Failures include balance loss during crouching or sudden running, a limitation linked to the controller's lack of sequence-level information.

C.3 Evaluation on H36M

  • The UHC imitates unseen H36M motions with high accuracy and success rate, significantly outperforming the baseline method.The H36M poses are fitted SMPL sequences, and the fitting process is imperfect.Source: §C.3 · pp. 18–19
  • The H36M evaluation uses 150 unique motion sequences from the train split as target motions; their fitted SMPL poses have about 10 mm mean per-joint position error.Source: §C.3 · pp. 18–19
  • Failures often occur when the humanoid loses balance while crouching or starts running suddenly, as the controller lacks sequence-level information about upcoming speedups.The authors say this indicates the kinematic policy should adjust target poses using the current simulation state and that a better controller needs further investigation.Source: §C.3 · p. 19

D.1 MoCap dataset.

The appendix describes the MoCap and real-world datasets used for evaluation, including their motion diversity, and notes potential uses and risks of the framework.

D.1 MoCap dataset.

  • The MoCap dataset contains 202 training and 64 testing sequences (148k frames) from three subjects, with paired first-person footage and marker-based human and object poses across five action types.Figure 5 recenters trajectories by object position; Table 8 evaluates UHC motion imitation using H36M target motion, and Table 9 reports dataset speed analysis in meters per second.Source: §D.1 · pp. 19–20, Figure 5 · p. 19, Table 8 · p. 19

D.2 Real-world dataset.

  • The real-world dataset contains 183 testing sequences (55k frames) from an additional subject in living-room and hallway settings, and the method estimates its more diverse motion patterns.For object registration, the subject looks at the object during an initial calibration period, which is later removed; the framework is starting-position and orientation-invariant.Source: §D.2 · p. 20, Figure 6 · p. 20

E Broader social impact.

  • The framework may support physically realistic character creation, while raising risks involving deceptive videos and privacy for smart-glasses and bodycam users.The authors encourage research into detecting computer-generated motion as realism improves.Source: E Broader social impact · p. 20
Loading 2106.05969v3…