PaperlayerView exact revision on arXiv

Source-linked AI summary

Resolving 3D Human Pose Ambiguities with 3D Scene Constraints

Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, Michael J. Black

arXiv:1908.06963v1cs.CV

TL;DR

Monocular 3D human pose estimation often ignores the scene, even though human movement and interaction are constrained by the world. PROX incorporates static 3D scene structure into SMPL-X and SMPLify-X through inter-penetration and contact constraints. The authors report significantly more realistic and accurate motion capture, with quantitative evaluations showing reduced 3D joint and vertex errors.

  • Problem

    Most existing 3D human pose estimation methods do not explicitly consider the 3D scene or human-world interactions.

  • Method

    PROX extends SMPLify-X with the SMPL-X body model and scene constraints that penalize body-scene inter-penetration and encourage likely contact.

  • Results

    Introducing scene constraints significantly reduces 3D joint error and vertex error, and produces more realistic and accurate motion capture.

  • Takeaways & Limitations

    Using static 3D scene structure improves monocular 3D body pose estimation for full-body human interactions with complex scenes.

  • Takeaways & Limitations

    The formulation does not model scene occlusion, while current 2D part detectors may be inaccurate when joints are occluded.

Abstract

from arXiv · show

To understand and analyze human behavior, we need to capture humans moving in, and interacting with, the world. Most existing methods perform 3D human pose estimation without explicitly considering the scene. We observe however that the world constrains the body and vice-versa. To motivate this, we show that current 3D human pose estimation methods produce results that are not consistent with the 3D scene. Our key contribution is to exploit static 3D scene structure to better estimate human pose from monocular images. The method enforces Proximal Relationships with Object eXclusion and is called PROX. To test this, we collect a new dataset composed of 12 different 3D scenes and RGB sequences of 20 subjects moving in and interacting with the scenes. We represent human pose using the 3D human body model SMPL-X and extend SMPLify-X to estimate body pose using scene constraints. We make use of the 3D scene information by formulating two main constraints. The inter-penetration constraint penalizes intersection between the body model and the surrounding 3D scene. The contact constraint encourages specific parts of the body to be in contact with scene surfaces if they are close enough in distance and orientation. For quantitative evaluation we capture a separate dataset with 180 RGB frames in which the ground-truth body pose is estimated using a motion capture system. We show quantitatively that introducing scene constraints significantly reduces 3D joint error and vertex error. Our code and data are available for research at https://prox.is.tue.mpg.de.

1. Introduction

PROX treats the 3D world as a first-class source of constraints for monocular human pose estimation. It penalizes body-scene inter-penetration, encourages likely contact, and evaluates the approach using qualitative and motion-capture-based quantitative datasets.

  • Most 3D human pose estimation methods ignore the world and human interactions with it, despite the environment affecting posture, movement, and behavior.
  • PROX estimates 3D human pose from a single RGB image conditioned on the 3D scene, using scene constraints to make estimation easier and more accurate.
  • The inter-penetration constraint penalizes poses in which the body intersects scene objects through a differentiable loss incorporated into SMPLify-X.
  • The contact constraint favors likely contact when selected body surfaces are close to scene surfaces and share their orientation.
  • PROX extends SMPLify-X with physically motivated environmental terms and uses the realistic SMPL-X body model as a proxy for the real human.
  • Evaluation combines qualitative RGB-D datasets with pseudo ground truth and a separate motion-capture dataset providing ground-truth 3D body shape and pose.

2. Related Work

Prior work studies human pose and scene understanding largely separately or uses limited interaction settings. PROX addresses full articulated bodies interacting with diverse, complex 3D scenes and uses scene structure to improve monocular pose estimation.

  • Traditional pose estimation models bodies in isolation, while scene reconstruction methods generally acquire or analyze scenes without humans.
  • Existing work uses scene context for pose, affordances, or interaction modeling, but scene cues are not generally fed back to improve human pose.
  • Several interaction approaches use RGB-D observations or synthetic scenes, but synthetic methods do not actually capture people in scenes.
  • Prior physical-constraint methods include contact or collision reasoning, often reducing pose degrees of freedom or constraining motion using scene structure.
  • Most prior world-constraint methods focus on ground-plane interaction, and one physics-engine approach has not been extended beyond ground contact.
  • PROX targets the full articulated body in diverse, complex, full 3D scenes and reports improved monocular 3D body pose estimation.

3. Technical Approach

PROX extends SMPLify-X with a SMPL-X body model and static 3D scene constraints to fit poses that respect human–scene interactions.

  • Scene Representation: The scene is represented as a static triangular 3D mesh reconstructed with a Structure Sensor and Skanect, using the scene frame as world coordinates.
  • Human Body Model: SMPL-X models body shape, pose, facial expressions, and finger articulation as a differentiable 3D mesh with 10,475 vertices.Its holistic body, face, and hand representation supports reasoning about interactions.
  • Pose Fitting: PROX extends SMPLify-X by adding human–world interaction constraints to RGB-based pose fitting, while PROX-D additionally uses depth input.The optimization framework was chosen because physically motivated constraints are straightforward to incorporate.
  • Pose Fitting: The objective combines RGB reprojection, optional depth alignment, pose and shape priors, contact encouragement, and self- and human–scene penetration penalties.The RGB term compares detected 2D keypoints with projected SMPL-X joints and uses confidence weighting with a robust error.
  • Contact Constraint: The contact term uses 1,121 annotated vertices on hands, thighs, gluteus, back, and feet to encourage proximity to nearby scene surfaces.The annotated counts are 725 hands, 62 thighs, 113 gluteus, 222 back, and 194 feet; distant vertices are downweighted robustly.
  • Penetration Constraint: The penetration term uses signed scene distances to penalize body vertices inside scene objects, with trilinear interpolation mitigating voxel-grid discretization.Signed distances are negative inside the nearest scene object and positive outside it.

4. Datasets

The paper introduces qualitative datasets of scanned 3D scenes and monocular videos, plus a separate motion-capture dataset for quantitative evaluation. PROX contains 12 indoor scenes and 20 interacting subjects, while the quantitative dataset provides 180 synchronized RGB-D frames.

  • Qualitative Datasets: PiGraphs and PROX provide 3D scene scans and monocular videos of people interacting with the scenes, but no ground-truth bodies.Consequently, these datasets support qualitative rather than quantitative evaluation.
  • PiGraphs dataset: The PiGraphs dataset has unsynchronized, uncalibrated RGB and depth, noisy poses, inaccurate scene reconstructions, 5 fps, 5 subjects, and no ground-truth.These limitations motivated collection of the PROX dataset.
  • PROX dataset: PROX reconstructs 12 indoor scenes across bedrooms, living rooms, sitting booths, and offices, and captures 20 subjects interacting with them.The subjects include 4 females and 16 males.
  • Quantitative Dataset: The separate quantitative dataset contains 180 static RGB-D frames synchronized with a 54-camera Vicon system and includes reconstructed living-room scenes.Body and finger markers support realistic mesh pseudo-ground-truth generated with MoSh++.
  • Quantitative Dataset: The quantitative evaluation uses mesh pseudo-ground-truth based on Vicon and MoSh++ for one dataset and SMPLify-D for selected qualitative-dataset sequences.Table 1 reports PJE, p.PJE, V2V, and p.V2V in millimeters.

5. Experiments

Experiments evaluate PROX through ablations and qualitative comparisons. Adding contact and penetration terms together produces the strongest reported improvement, while qualitative results show more scene-consistent bodies without penetrations.

  • Quantitative Evaluation: 167.08 mm PJE is achieved by PROX when both contact and penetration terms are included on the quantitative dataset.Adding either term alone reduces error slightly, but the combined configuration produces a significantly larger drop than the RGB-only baseline.
  • Quantitative Evaluation: PROX achieves reasonably good performance using RGB only, whereas the pseudo-ground-truth generated by SMPLify-D uses RGB and depth.On selected PROX sequences, the comparison supports a significant accuracy contribution from scene constraints.
  • Implementation: The run time for all experimental configurations is reported in the supplementary material.The main text refers readers to the supplementary material for these measurements.
  • Qualitative Evaluation: Qualitative comparisons show that unconstrained estimates can severely penetrate scene objects, while PROX produces realistic contact and avoids penetrations.The comparison is shown on the quantitative dataset and on the PROX and PiGraphs datasets.

6. Conclusion

The work incorporates human-world interaction constraints into monocular 3D human pose estimation and evaluates the resulting method with qualitative and quantitative data. It concludes that scene constraints improve pose realism and accuracy, while the current formulation does not model scene occlusion.

  • Conclusion: PROX uses SMPL-X to model the body and incorporates interaction-based human-world constraints into an optimization framework.The framework jointly represents the body, face, and fingers, which are relevant to interactions.
  • Conclusion: The authors collect a dataset of 3D scenes with RGB-D sequences containing human interactions and occlusions.The dataset supports evaluation of human motion in real static scenes.
  • Conclusion: Extensive quantitative and qualitative evaluations show benefits from incorporating scene constraints into 3D human pose estimation.The conclusion reports improved realism and accuracy without specifying numerical error values in this section.
  • Limitations and Future work: The current formulation does not model scene occlusion, and future work could extend it to dynamic scenes, human-human interaction, and deformation.The authors also propose reasoning about visibility using known scene structure.

**Supplementary Material**

PROX adds scene-aware terms to pose optimization so that fitted bodies respect the scanned 3D environment. The supplementary discussion emphasizes reduced penetration and improved contact, while noting that assuming a rough scene model remains a scope boundary.

  • Method: PROX stands for Proximal Relationships with Object eXclusion and adds environmental terms to pose optimization.The method extends SMPLify-X and uses scene constraints to refine the fitted human body.
  • Method: Without environment constraints, solutions can look correct from the camera while being grossly inaccurate when placed in the 3D scene.The supplementary examples motivate evaluating the fitted body from additional viewpoints.
  • Method: Adding PROX constraints reduces body-scene inter-penetration and encourages appropriate contact.These terms are designed to make the body more consistent with the surrounding environment.
  • Assumption: The approach assumes that a rough 3D model of the scene is available, although scanning static scenes is described as feasible.The authors identify direct monocular scene estimation as a future direction because it remains difficult and ill-posed.

Qualitative Results - Our Dataset

The supplementary figures present additional qualitative results for PROX on the PROX dataset, comparing scene-aware estimates with an RGB-only baseline from multiple viewpoints.

  • Qualitative Results - Our Dataset: Additional examples compare PROX with the RGB-only baseline using renderings from different viewpoints.The figures show the RGB image followed by renderings from alternative views.

Qualitative Results - PiGraphs

Additional PiGraphs examples compare PROX with an RGB-only baseline and highlight the richer representation used by PROX for reasoning about human-world proximity.

  • Qualitative Results - PiGraphs: On PiGraphs, PROX estimates a full 3D mesh with facial expressions and finger articulation, unlike the baseline's major-joint 3D skeleton.The mesh representation supports reasoning about proximity, contact, and penetrations.
  • Qualitative Results - PiGraphs: The supplementary figure compares PROX and the RGB-only baseline through renderings from different viewpoints.The comparison uses light gray for PROX and yellow for the RGB-only baseline.

Computational Complexity

PROX improves V2V relative to RGB-only estimation, but increases runtime. Table A.1 reports runtimes across the approach’s configurations.

  • Computational Complexity: 24% V2V improvement over RGB alone comes with a 41% runtime increase.This comparison is reported for PROX relative to using RGB alone.
  • Computational Complexity: Table A.1 compares the average runtime of all approach configurations on 10 randomly sampled frames.

Choice of Contact Vertices

Contact-vertex selection affects both performance and runtime. Using all body vertices performs worse and adds approximately seven seconds, motivating targeted candidate selection and future learned alternatives.

  • Choice of Contact Vertices: Using all body vertices deteriorates performance compared with the annotated candidate contact vertices.
  • Choice of Contact Vertices: Using all vertices increases runtime by approximately 7 seconds.
  • Choice of Contact Vertices: The authors suggest that affordances and semantics make targeted contact-vertex selection important.
  • Choice of Contact Vertices: Learning likely contact vertices for object classes is left to future work because suitable training data is needed.
Loading 1908.06963v1…