PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Human Pose Estimation via Intuitive Physics

Shashank Tripathi, Lea Müller, Chun-Hao P. Huang, Omid Taheri, Michael J. Black, Dimitrios Tzionas

arXiv:2303.18246v3cs.CVcs.AIcs.GR

TL;DR

Image-based 3D human estimates often ignore scene support and therefore produce implausible floor interactions, while existing physics engines are difficult to differentiate and integrate. IPMAN introduces differentiable intuitive-physics terms from SMPL pressure, CoP, and CoM to encourage stable configurations. Across standard datasets and MoYo, it produces more accurate and stable bodies, improving static poses without hurting dynamic ones.

  • Problem

    Existing image-based 3D HPS methods can produce bodies that lean, hover, or penetrate the ground because they ignore physical scene support, while physics simulators are difficult to integrate into HPS frameworks.

  • Method

    IPMAN uses differentiable intuitive-physics terms derived from SMPL pressure, Center of Pressure, Center of Mass, and floor contact in optimization and regression-based HPS.

  • Results

    IPMAN produces more accurate and stable 3D bodies than the state of the art across evaluated datasets, improving static poses without hurting dynamic ones.

  • Takeaways & Limitations

    Intuitive-physics terms provide a simple, differentiable way to promote realistic floor support and physical plausibility in image-based 3D human estimation.

  • Takeaways & Limitations

    The current IP terms address body-floor contact and are designed to help static poses; future work must extend them to general scene contact, diverse surfaces, and dynamic activities.

Abstract

from arXiv · show

Estimating 3D humans from images often produces implausible bodies that lean, float, or penetrate the floor. Such methods ignore the fact that bodies are typically supported by the scene. A physics engine can be used to enforce physical plausibility, but these are not differentiable, rely on unrealistic proxy bodies, and are difficult to integrate into existing optimization and learning frameworks. In contrast, we exploit novel intuitive-physics (IP) terms that can be inferred from a 3D SMPL body interacting with the scene. Inspired by biomechanics, we infer the pressure heatmap on the body, the Center of Pressure (CoP) from the heatmap, and the SMPL body's Center of Mass (CoM). With these, we develop IPMAN, to estimate a 3D body from a color image in a "stable" configuration by encouraging plausible floor contact and overlapping CoP and CoM. Our IP terms are intuitive, easy to implement, fast to compute, differentiable, and can be integrated into existing optimization and regression methods. We evaluate IPMAN on standard datasets and MoYo, a new dataset with synchronized multi-view images, ground-truth 3D bodies with complex poses, body-floor contact, CoM and pressure. IPMAN produces more plausible results than the state of the art, improving accuracy for static poses, while not hurting dynamic ones. Code and data are available for research at https://ipman.is.tue.mpg.de.

1. Introduction

Existing 3D HPS methods can fit images while producing bodies that lean, hover, or penetrate the ground because they ignore scene support. IPMAN adds differentiable intuitive-physics terms based on pressure, CoP, CoM, and floor contact, improving plausibility and accuracy without harming dynamic poses.

  • Motivation: Image-aligned 3D HPS estimates can remain physically implausible because they reason about humans in isolation from scene support.Such bodies may lean, hover, or penetrate the ground, limiting applications that require faithful 3D scene interaction.
  • Approach: IPMAN replaces full physics simulation with simple, differentiable intuitive-physics terms compatible with SMPL and existing optimization or regression methods.The terms use an inferred ground pressure heatmap, Center of Pressure, and projected Center of Mass.
  • Approach: The method encourages sufficient ground contact, penalizes interpenetration, and promotes overlap between the CoP and CoM when optimizing or training HPS models.These terms are incorporated both into an objective extending SMPLify-XMC and into the training loss of the IPMAN regressor.
  • Technical contributions: IPMAN estimates CoM using part-weighted sampling of the SMPL surface and estimates CoP from penetration-derived pressure without requiring pressure sensors.The CoM formulation accounts for SMPL shape, pose, and blend shapes, while deeper penetration serves as a higher-pressure proxy.
  • Evaluation: On Human3.6M, RICH, and the new MoYo dataset, IPMAN produces more accurate and stable bodies than the state of the art, improving static poses without hurting dynamic ones.MoYo provides synchronized multi-view video, ground-truth SMPL-X meshes, pressure measurements, and body CoM for challenging yoga poses.

2. Related Work

Prior HPS work largely omits scene context or uses contact and physics constraints that are limited by binary contact assumptions, non-differentiability, or unrealistic proxy bodies. IPMAN instead derives differentiable CoM, CoP, and BoS quantities directly from the SMPL surface for both optimization and regression.

  • HPS methods: Most HPS methods reconstruct humans without accounting for the surrounding scene, despite otherwise successful image alignment.This isolation motivates methods that incorporate body-scene interaction and physical support.
  • Contact-only scene constraints: Contact-based methods commonly detect contact regions and minimize distances, often restricting contact to feet or other end-effectors.These approaches typically use distance thresholds and relative-motion assumptions to identify contact.
  • Contact-only scene constraints: IPMAN uses the full 3D body surface to compute pressure, CoP, and CoM, making its contact reasoning differentiable rather than binary.This differentiability allows the intuitive-physics terms to be used when training HPS regressors.
  • Physics-based scene constraints: Physics-based methods use simulation or physics-inspired optimization to regularize dynamics, discourage interpenetration, or encourage contact, but IPMAN applies its objectives to both optimization and regression.Related work includes balance criteria based on projected CoM within the BoS and differentiable physics objectives.
  • Positioning: Compared with prior physics approaches using rigid shape primitives or pressure measurements, IPMAN computes CoM, CoP, and BoS directly from an SMPL mesh and uses images as input.The cited pressure-based methods address a different task because they rely on pressure measurements rather than images.

3. Method

IPMAN models stability from a differentiable SMPL body by estimating mass, pressure, contact, and support relationships, then applying intuitive-physics losses to optimization and regression-based HPS.

  • Body and Camera Models: SMPL maps pose and shape parameters to a 3D mesh, with IPMAN applying the resulting physical terms to camera- or world-coordinate body estimates.IPMAN-R uses SMPL, whereas IPMAN-O uses SMPL-X to match its baselines.
  • Stability Analysis: The SMPL body’s pCoM is computed from per-part volumes and uniformly sampled surface points using a volume-weighted mean.Differentiable close-translate-fill operations estimate watertight part volumes without mesh splitting.
  • Stability Analysis: The CoP is inferred from mesh-ground penetration, treating deeper penetration as higher pressure and computing a pressure-weighted surface-point average.The formulation uses a differentiable pressure field motivated by soft-tissue deformation and Hooke’s law.
  • Stability Analysis: Stability is analyzed using gravity-projected CoM and CoP, while the BoS is the convex hull of gravity-projected body-ground contacts and is used only for evaluation.The method replaces the sparse-gradient BoS criterion with a CoM–CoP distance loss.
  • Intuitive-Physics Losses: IPMAN extends HPS with differentiable intuitive-physics losses that encourage ground contact, discourage penetration and hovering, and align projected CoM and CoP.The method supports both regression-based IPMAN-R and optimization-based IPMAN-O.

4. Experiments

Experiments evaluate IPMAN on standard benchmarks and the challenging MoYo dataset using accuracy, stability, pressure, CoP, and CoM measures. Both regression and optimization variants improve physical plausibility and accuracy relative to their baselines.

  • Datasets: MoYo contains 200 complex yoga poses recorded with synchronized motion capture, pressure, and eight calibrated cameras, totaling approximately 1.75M 4K RGB frames with SMPL-X, pressure, and CoM ground truth.Its extreme poses, self-occlusion, and body-ground and self-contact make it more challenging than Fit3D and PosePrior.
  • IPMAN-R: IPMAN-R improves over the regression baseline on RICH and Human3.6M, with joint IP terms yielding the largest gains and improving RICH MPJPE by 3.5 mm and PVE by 2.5 mm.It also improves on largely dynamic Human3.6M poses and outperforms cited ground-aware optimization methods by approximately 12% in MPJPE.
  • IPMAN-O: IPMAN-O improves on SMPLify-XMC across all evaluation metrics on MoYo, while producing more stable and physically plausible world-frame results despite similar image projections.Adding the stability loss specifically improves PVE, but not joint metrics or BoSE, because the terms depend on mesh-surface position relative to the ground.
  • Pressure, CoP and CoM Evaluation: Pressure and CoP evaluation on MoYo reports an IoU of 0.32 and a CoP error of 57.3 mm, while estimated pCoM differs from Vicon CoM by 53.3 mm.Figure 6 compares estimated and ground-truth pressure heatmaps and marks their CoPs separately.
  • Physics Simulation: Posthoc Bullet simulation finds IPMAN-O produces 14.8% more stable bodies than the baseline, with stability measured by mesh displacement after simulation.The evaluation uses convex decomposition because existing physics engines are not directly compatible with SMPL-based HPS methods.

5. Conclusion

IPMAN integrates differentiable intuitive-physics terms into 3D human pose estimation, improving physical plausibility and accuracy across regression and optimization settings. The paper identifies broader scene-contact and dynamic-activity reasoning as future directions.

  • 5. Conclusion: IPMAN improves regression and optimization baselines across metrics and recovers more physically plausible meshes on existing datasets and MoYo.Its terms exploit CoM, CoP, and BoS to promote floor support and reduce body-floor penetration.
  • 5. Conclusion: The method introduces differentiable CoM, CoP, and pressure-based intuitive-physics terms compatible with SMPL and both optimization- and regression-based HPS methods.The approach avoids full physics simulation while reasoning about arbitrary full-body ground contact.
  • 5. Conclusion: Future work should extend body-floor contact reasoning to general body-scene contact and diverse supporting surfaces through 3D scene reconstruction.The authors also identify dynamic activities and multi-person scenarios as possible extensions.

A. MoCap Yoga Dataset (MoYo)

MoYo is a motion-capture yoga dataset designed to evaluate complex human poses and physical interaction with the floor. It combines synchronized cameras, ground-truth body models, pressure measurements, and body center-of-mass annotations.

  • A. MoCap Yoga Dataset (MoYo): MoYo captures a trained yoga professional with 54 infrared cameras, 8 synchronized RGB cameras, and a Zebris pressure mat.The RGB cameras record at 4112x3008 resolution, while the pressure mat measures 10–1200 kPa.
  • A. MoCap Yoga Dataset (MoYo): The dataset provides ground-truth SMPL-X parameters recovered from motion-capture data using MoSh++.The capture setup synchronizes Vicon tracking, RGB video, and pressure sensing.
  • A. MoCap Yoga Dataset (MoYo): Reference whole-body CoM annotations are computed with Vicon Plug-in Gait using anthropometric measurements and weighted per-bone centers of mass.This reference uses known body measurements, per-bone masses, and per-bone CoM locations.
  • A. MoCap Yoga Dataset (MoYo): MoYo includes challenging pose distributions that complement existing motion-capture datasets, including unstable long-tail poses represented in AMASS.The supplied distribution figures compare MoYo and existing MoCap datasets after t-SNE reduction and compare their stability-loss distributions.

B. Method

The stability loss replaces a discontinuous binary CoM-within-BoS criterion with a continuous differentiable L2 formulation. This soft loss also indicates the degree of instability.

  • B. Method: The classic stability criterion is discontinuous because it only tests whether gravity-projected CoM lies inside or outside the BoS.Its sparse gradients make direct optimization difficult.
  • B. Method: The proposed L2 stability loss is continuous, fully differentiable, and informative about the degree of instability.Because CoP lies inside BoS, the loss provides a soft approximation to the classic stability definition.

B.2. Elements of Stability Analysis: Alternative formulations

The appendix analyzes alternative CoM and CoP formulations, ground-loss behavior, experimental configurations, and qualitative stability results. It supports part-weighted mass modeling, pressure-aware contact, joint ground losses, and evaluation across optimization and regression methods.

  • B.2. Elements of Stability Analysis: Alternative formulations: Naive vertex-mean CoM is biased because SMPL and SMPL-X distribute vertices unevenly, with disproportionate representation of the face and hands.Approximately half of SMPL-X vertices lie on the head, causing naive averaging to overemphasize those regions.
  • B.2. Elements of Stability Analysis: Alternative formulations: Surface-sampling and area-weighted formulations assume mass is proportional to surface area, which is a poor approximation.This limitation motivates the part-weighted pCoM formulation.
  • B.2. Elements of Stability Analysis: Alternative formulations: pCoM uniformly samples the SMPL surface and weights points by part-specific volume through M = ρV, outperforming alternative CoM formulations against Vicon ground truth.The comparison is summarized in Table S.1 and illustrated qualitatively in Figure S.3.
  • B.2. Elements of Stability Analysis: Alternative formulations: The proposed CoP is differentiable and uses inferred pressure distributions rather than equally weighting binary-contact vertices.Figure S.3 contrasts pressure-aware CoP with naive binary-contact alternatives.
  • B.3. Ablation of ground losses: Soft Lpull gradients decay with height, while Lpush and Lpull complement each other and are more effective jointly as Lground.Figure S.4 reports minimal contribution from sufficiently high vertices, and Table S.2 evaluates the loss ablation.
  • C. Experiments: The intuitive-physics terms are integrated into both optimization and regression methods using SMPL-X for IPMAN-O and SMPL for IPMAN-R.The design aims to benefit both commonly used HPS method types and different body models.
  • C.1.1 IPMAN-R.: IPMAN-R is trained and evaluated with a matched HMR baseline, while RICH training uses sequences with mainly even ground planes.The experimental setup and sequence partitioning are documented in the implementation description and Table S.3.
  • C.2.1 BoS Error (BoSE) calculation.: BoSE tests whether gravity-projected CoM lies in the convex hull of contact vertices using a linear-programming convex-combination check.Supplementary qualitative figures show IPMAN-R and IPMAN-O results, while Bullet simulation shows IPMAN-O meshes remaining stable when baseline meshes topple.

D. Stability Evaluation via Physics Simulation

IPMAN-O is evaluated for physical stability by simulating estimated meshes in Bullet after convex decomposition. Stability is measured from mesh displacement over 100 simulation steps.

  • D. Stability Evaluation via Physics Simulation: 14.8% more IPMAN-O bodies are stable than the baseline under posthoc Bullet simulation.Estimated meshes are decomposed into convex parts before 100 physics simulation steps; smaller displacement denotes greater stability.
  • D. Stability Evaluation via Physics Simulation: The evaluation uses V-HACD convex decomposition to approximate estimated body meshes for physics simulation.This posthoc procedure does not integrate the physics engine into estimation.
  • D. Stability Evaluation via Physics Simulation: Mesh displacement after 100 Bullet simulation steps serves as the stability criterion.Small displacement indicates a stable pose, whereas large displacement indicates instability.

E. Evaluation of Biomechanical Elements

The evaluation tests inferred pressure, CoP, and CoM against MoYo measurements, then examines intuitive-physics terms within an extended optimization objective on RICH.

  • E. Evaluation of Biomechanical Elements: The pressure heatmap represents relative pressure because the method does not assume ground-truth body mass or anthropometric measurements.Per-point pressure on the SMPL mesh is projected to the ground plane and summed into the heatmap.
  • E. Evaluation of Biomechanical Elements: Estimated pressure, CoP, and CoM agree with ground truth in MoYo.Pressure is evaluated by heatmap intersection-over-union, while CoP and CoM use Euclidean distance errors.
  • E. Evaluation of Biomechanical Elements: IPMAN-O* adds stability and ground-contact losses to an SMPLify-X-based objective alongside reprojection and body-prior terms.The objective includes E_stability and E_ground in addition to image, pose, shape, expression, bending, and self-penetration terms.
  • E. Evaluation of Biomechanical Elements: Applying intuitive-physics terms only at the final optimization stage produces more accurate results than applying them at every stage.The authors hypothesize that later application benefits from better body initialization.

G. Evaluation on 3DPW

Evaluation on 3DPW tests whether intuitive-physics terms remain useful despite inconsistent ground planes and predominantly dynamic motion. The reported results show a slight performance improvement.

  • G. Evaluation on 3DPW: Qualitative evaluations compare HMR with IPMAN-R on RICH and SMPLify-XMC with IPMAN-O on MoYo.The figures show estimated meshes from multiple views, with CoM and pressure-related visualizations in the MoYo evaluation.
  • G. Evaluation on 3DPW: 3DPW contains primarily dynamic activities such as walking, stair climbing, and parkour.Its ground plane can be inconsistent, with subjects in one scene supported by different world-coordinate ground planes.
  • G. Evaluation on 3DPW: The authors report 3DPW results to test whether intuitive-physics terms degrade performance on datasets outside IPMAN’s core assumptions.They state that the method remains applicable to everyday motion without special care.
  • G. Evaluation on 3DPW: IPMAN-R shows a slight performance improvement on 3DPW despite the dataset’s inconsistent ground planes.The improvement is reported in Table S.6.
Loading 2303.18246v3…