PaperlayerView exact revision on arXiv

Source-linked AI summary

Capturing and Inferring Dense Full-Body Human-Scene Contact

Chun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, Michael J. Black

arXiv:2206.09553v1cs.CV

TL;DR

The paper addresses the challenge of inferring dense 3D human-scene contact from a single image, which existing methods handle only coarsely or without image evidence. It introduces the RICH dataset and BSTRO, a transformer-based detector, and reports stronger contact detection than prior methods while evaluating pose-estimation performance in contact-rich scenes.

  • Problem

    Dense 3D human-scene contact from a single image remains underexplored, with prior methods using limited contact types, simplified representations, or no direct image evidence.

  • Method

    The paper creates RICH with detailed body, scene, and vertex-level contact data, then trains BSTRO to predict dense body-scene contact directly from an image using non-local transformer relationships.

  • Results

    BSTRO achieves the highest F1 score and lowest geodesic error among compared methods, while RICH exposes difficulties for state-of-the-art human pose estimation in contact-rich scenes.

  • Takeaways & Limitations

    RICH and BSTRO provide a basis for studying and estimating detailed body-scene interaction beyond foot contact or scene reconstruction.

  • Takeaways & Limitations

    BSTRO uses SMPL rather than SMPL-X, reducing the output space by excluding many head vertices because they rarely participate in natural body-scene contact.

Abstract

from arXiv · show

Inferring human-scene contact (HSC) is the first step toward understanding how humans interact with their surroundings. While detecting 2D human-object interaction (HOI) and reconstructing 3D human pose and shape (HPS) have enjoyed significant progress, reasoning about 3D human-scene contact from a single image is still challenging. Existing HSC detection methods consider only a few types of predefined contact, often reduce body and scene to a small number of primitives, and even overlook image evidence. To predict human-scene contact from a single image, we address the limitations above from both data and algorithmic perspectives. We capture a new dataset called RICH for "Real scenes, Interaction, Contact and Humans." RICH contains multiview outdoor/indoor video sequences at 4K resolution, ground-truth 3D human bodies captured using markerless motion capture, 3D body scans, and high resolution 3D scene scans. A key feature of RICH is that it also contains accurate vertex-level contact labels on the body. Using RICH, we train a network that predicts dense body-scene contacts from a single RGB image. Our key insight is that regions in contact are always occluded so the network needs the ability to explore the whole image for evidence. We use a transformer to learn such non-local relationships and propose a new Body-Scene contact TRansfOrmer (BSTRO). Very few methods explore 3D contact; those that do focus on the feet only, detect foot contact as a post-processing step, or infer contact from body pose without looking at the scene. To our knowledge, BSTRO is the first method to directly estimate 3D body-scene contact from a single image. We demonstrate that BSTRO significantly outperforms the prior art. The code and dataset are available at https://rich.is.tue.mpg.de.

1. Introduction

The paper targets dense 3D human-scene contact from a single image, addressing prior methods’ limited contact types, simplified representations, and lack of image evidence. It introduces RICH and BSTRO to capture and directly infer vertex-level contact.

  • Existing HPS methods largely ignore scenes, which can produce physically implausible poses and shapes.
  • Prior HSC methods simplify scenes or bodies, often focus on feet, and do not use image evidence for contact prediction.
  • Multiview capture resolves occlusions and supports outdoor environments compared with monocular RGB-D capture.
  • RICH provides multiview 4K indoor and outdoor sequences, scanned scenes, 3D human data, and dense full-body contact labels.
  • BSTRO directly estimates vertex-level body-scene contact from a single image without reconstructing the body or requiring a 3D scene reconstruction.
  • BSTRO uses a transformer to learn non-local image relationships because contact regions are occluded and require evidence from across the image.

2. Related Work

Prior work estimates bodies, interactions, or contact using simplified representations and indirect cues, while datasets generally lack suitable natural images with dense 3D human-scene contact labels. RICH addresses this gap with richer capture and scene-contact supervision.

  • 2.1. Human Pose and Shape Estimation (HPS): Monocular HPS methods estimate 3D bodies from single color images using parametric or non-parametric representations and varied forms of supervision.
  • 2.1. Human Pose and Shape Estimation (HPS): Existing markerless motion-capture methods mainly estimate skeletons or meshes in constrained settings rather than full-body contact in natural outdoor scenes.
  • 2.1. Human Pose and Shape Estimation (HPS): Body reconstruction methods that ignore scene interaction can produce artifacts such as foot skating and ground penetration.
  • 2.2. Human Scene Interaction (HSI): Prior 3D contact approaches use thresholds, known scans, pose correlations, physics, or zero-velocity assumptions, often with coarse body representations.
  • 2.2. Human Scene Interaction (HSI): Existing HSC datasets provide limited image types or lower-quality RGB-D-fitted bodies, restricting interactions and body-fit quality.

3. Methods: RICH Dataset

RICH captures and reconstructs human bodies across synchronized views, fits SMPL-X models, and places them into scanned scenes to derive contact labels. Its fitting pipeline combines multiview objectives, temporal refinement, robust initialization, and automatic distance-based contact annotation.

  • SMPL-X body vertices are controlled by pose, shape, and facial-expression parameters, with body and hand pose represented separately.
  • RICH associates subjects across synchronized cameras and time, then reconstructs SMPL-X bodies using multiview fitting.
  • The fitting objective combines joint data error with pose, bending, shape, mesh-intersection, and other regularization terms.
  • Multiview consensus reduces noisy landmark influence through soft majority voting, independently of the current body estimate.
  • The pipeline refines frame-wise fits over temporal batches using body- and hand-motion smoothness terms.
  • Automatic contact labels are obtained by thresholding each body vertex’s distance to the scanned scene surface, with surface normals also considered.

4. Methods: BSTRO

BSTRO directly predicts dense vertex-level human-scene contact from a single image, avoiding intermediate body reconstruction. Its transformer uses non-local image evidence to infer occluded contacts and outputs per-vertex contact probabilities.

  • Direct contact estimation: BSTRO replaces the usual body-reconstruction-then-contact pipeline with direct image-to-contact prediction, c = g(I), at dense body-vertex resolution.Prior methods compose body recovery and contact estimation as c = g ◦ f(I), while BSTRO predicts contact directly from the input image.
  • Body representation: The model uses SMPL with 6,890 vertices, reducing the output space relative to SMPL-X because head vertices rarely contact scenes.The contact vector is binary over the SMPL mesh vertices.
  • Architecture: A CNN extracts image features, combines them with SMPL template vertex locations, and feeds one query per vertex to a multi-layer transformer.The transformer outputs one logit per vertex, which is converted into a contact probability with a sigmoid.
  • Prediction and training: BSTRO thresholds each vertex probability at 0.5 to obtain dense contact labels and trains with binary cross entropy on RICH image-contact pairs.The task is formulated as multi-label classification, with each body vertex assigned its own contact probability.
  • Occlusion robustness: Masked Vertex Modeling randomly hides some vertex queries while retaining all outputs, encouraging the transformer to use other queries and image context under occlusion.This training procedure is intended to simulate partial body visibility and support contact hallucination.

5. RICH Dataset

RICH is a large multiview dataset for dense human-scene contact in realistic indoor and outdoor environments. It combines high-resolution imagery, 3D body data, contact labels, and scene scans, while also supporting HPS benchmarking.

  • Capture setup: RICH captures 22 subjects interacting in five static 3D scenes using six to eight static cameras, with an additional moving camera in some scenes.The capture includes both indoor and outdoor environments and was reviewed by the University of Tübingen Ethics Committee.
  • Dataset scale: RICH contains 142 multiview videos, 90K posed 3D body meshes, 90K dense full-body contact labels, and 577K 4K images.Labels are provided in both SMPL-X and SMPL mesh topologies.
  • Realistic scenes: Compared with PROX, RICH emphasizes real, mostly outdoor scenes with dynamic backgrounds, varied viewpoints, multiple subjects, and areas of roughly 60m2.These properties support training and evaluating monocular human-scene contact methods.
  • Additional benchmark use: RICH also provides SMPL-X fits that support monocular or multiview HPS benchmarking alongside human-scene contact evaluation.The dataset enables analyses connecting HPS performance with body-scene contact that are not feasible on existing HPS datasets.

6. Experiments

Experiments evaluate dense HSC detection on RICH-test, compare baselines, analyze generalization, and examine how scene contact affects monocular HPS. BSTRO achieves the strongest overall contact balance, while meaningful scene contact increases HPS errors.

  • Experimental Setup: RICH-test contains withheld scenes and subjects, including a most challenging subset where scene, interaction, and subject are all unseen.The split uses 142 videos divided into 62 training, 28 validation, and 52 testing videos.
  • Evaluation Metrics and Baselines: Precision, recall, and F1 evaluate dense HSC, while geodesic error measures prediction error along the body surface.Geodesic distance is zero for true positives and quantifies the body-surface displacement of incorrect predictions.
  • Main Results: HuMoR has the lowest detection scores and highest geodesic errors, partly because it assumes contact with an even ground plane despite varied real-scene interactions.The compared baselines include sparse-joint methods from Zou et al. and HuMoR, plus dense-contact POSA variants.
  • Main Results: BSTRO has higher precision but lower recall than POSA, yet achieves the highest F1 score and lowest geodesic error.POSA generally has higher recall but produces many false positives; BSTRO provides a better precision-recall balance.
  • Main Results: BSTRO estimates scene contact closer to ground truth than POSAPIXIE, though it sometimes misses hand contact and also works on in-the-wild images.Qualitative comparisons show frequent POSAPIXIE false positives and occasional missed contacts on the hands.
  • Generalization: Withholding human-scene interaction causes a larger performance drop than withholding subjects, while all attributes unseen remains comparable to other subsets.Seeing similar interaction during training helps most; same scenes or subjects alone do not guarantee gains.
  • HPS Evaluation on RICH-test: 214.0mm/172.81mm TR-MPJPE/TR-V2V occurs with meaningful contact versus 161.81mm/121.71mm without contact beyond foot-ground contact.For moving cameras, the corresponding errors are 84.15mm/83.16mm PA-MPJPE/PA-V2V with meaningful contact and 63.67mm/64.37mm without.

7. Conclusion

The paper introduces RICH for studying human-scene contact and uses its detailed data to train BSTRO for single-image 3D contact inference. RICH also exposes problems for human pose estimation and is limited to static scenes.

  • RICH combines natural video sequences, high-resolution 3D scene scans, ground-truth body shapes, reference poses, and detailed 3D contact labels.
  • BSTRO uses RICH contact information to infer 3D body contacts from a single image of a person interacting with a scene.
  • Scenes with significant contact cause problems for state-of-the-art human pose estimation.
  • Limitations and future work: RICH considers only contact with static scenes, excluding dynamic-scene interactions such as hand-held objects and human-human interaction.

A. SMPL-X vs. SMPL HSC labels

RICH maps SMPL-X contact labels to SMPL because the models share vertices below the neck, preserving contact information while supporting both body models. However, HPS parameters differ and SMPL-X is therefore provided as pseudo ground truth.

  • SMPL-X and SMPL share a one-to-one correspondence between vertices below the neck, enabling HSC labels to be mapped between models.
  • Mapping SMPL-X HSC labels to SMPL preserves information while retaining realistic hand articulation in SMPL-X and a smaller SMPL output space.
  • RICH is suitable for HSC benchmarking with both body models because the shared vertices of interest do not affect detection scores or errors.
  • RICH provides only SMPL-X pseudo ground truth because converting HPS parameters can require extra processing and lose hand articulation when converting to SMPL.

B. RICH Dataset

RICH provides large-scale 4K multiview data, 3D bodies, and dense scene-contact labels, with broader and more accurate scene information than prior HPS datasets. The section also describes the bone-vector orientation objective used in multiview fitting.

  • RICH Dataset: Compared with AGORA, RICH provides more 3D bodies, more images, more accurate body shapes, and high-quality scene scans with contact labels.
  • Bone-orientation Term EO: The bone-orientation term factors the parent-joint residual out of the child-joint residual.
  • Bone-orientation Term EO: The target and estimated bone vectors are defined from detected landmarks and SMPL-X joints, respectively.
  • Bone-orientation Term EO: Because the target vector and estimated bone length are fixed during optimization, minimizing the residual maximizes the dot product between the two bone vectors.
  • Bone-orientation Term EO: The objective is minimized when the estimated bone vector has the same orientation as the target bone vector.

D. BSTRO Implementation Details

BSTRO training uses sampled image-contact pairs from RICH with varied viewpoints and backgrounds, then selects the best model using validation performance while withholding the test set.

  • Data sampling: Training samples use every other frame and combine dynamic with random static views, or two static views when no moving camera is available.
  • Data sampling: The sampling strategy provides viewpoint and background variation while keeping the number of training pairs tractable.
  • Optimization: BSTRO is trained on 24K image-contact pairs for 100 epochs with Adam at an initial learning rate of 1e-4.
  • Optimization: The HR-Net backbone is initialized from weights pretrained on ImageNet, Human3.6M, or 3DPW.
  • Model selection: The best checkpoint is selected on RICH validation while RICH test remains completely withheld.
Loading 2206.09553v1…