PaperlayerView exact revision on arXiv

Source-linked AI summary

Populating 3D Scenes by Learning Human-Scene Interaction

Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, Michael J. Black

arXiv:2012.11581v2cs.CV

TL;DR

Human body models and scene representations generally do not explicitly capture semantically meaningful contact between people and 3D environments. POSA learns a body-centric extension of SMPL-X that predicts per-vertex contact and scene semantics, and demonstrates improved scene population and scene-consistent monocular pose estimation. Its main scope is constrained by the consistent mesh topology required to learn the feature model.

  • Problem

    Existing body representations do not explicitly capture human-scene contact, while realistic placement and recovery of people in 3D scenes remain challenging.

  • Method

    POSA is a conditional VAE conditioned on SMPL-X vertices that learns body-centric per-vertex contact probabilities and semantic labels from PROX and PROX-E data.

  • Results

    POSA achieves higher perceptual realism than PLACE for scene population and lower pose estimation errors than PROX for monocular 3D pose estimation in scenes.

  • Takeaways & Limitations

    The learned body-scene representation supports realistic placement of posed people and scene-consistent monocular pose estimation across the paper’s two applications.

  • Takeaways & Limitations

    Learning the feature model requires consistent mesh topology, whereas realistic scans can have different mesh topology for every scan.

Abstract

from arXiv · show

Humans live within a 3D space and constantly interact with it to perform tasks. Such interactions involve physical contact between surfaces that is semantically meaningful. Our goal is to learn how humans interact with scenes and leverage this to enable virtual characters to do the same. To that end, we introduce a novel Human-Scene Interaction (HSI) model that encodes proximal relationships, called POSA for "Pose with prOximitieS and contActs". The representation of interaction is body-centric, which enables it to generalize to new scenes. Specifically, POSA augments the SMPL-X parametric human body model such that, for every mesh vertex, it encodes (a) the contact probability with the scene surface and (b) the corresponding semantic scene label. We learn POSA with a VAE conditioned on the SMPL-X vertices, and train on the PROX dataset, which contains SMPL-X meshes of people interacting with 3D scenes, and the corresponding scene semantics from the PROX-E dataset. We demonstrate the value of POSA with two applications. First, we automatically place 3D scans of people in scenes. We use a SMPL-X model fit to the scan as a proxy and then find its most likely placement in 3D. POSA provides an effective representation to search for "affordances" in the scene that match the likely contact relationships for that pose. We perform a perceptual study that shows significant improvement over the state of the art on this task. Second, we show that POSA's learned representation of body-scene interaction supports monocular human pose estimation that is consistent with a 3D scene, improving on the state of the art. Our model and code are available for research purposes at https://posa.is.tue.mpg.de.

1. Introduction

POSA makes human-scene interaction an explicit, body-centric part of a learned human model, encoding contact and semantic relationships for scene-aware applications. Trained with SMPL-X data, it supports realistic scene population and scene-consistent monocular pose estimation.

  • Motivation: Existing human models do not explicitly represent meaningful body-scene contact, while recovering or synthesizing people in scenes remains challenging.The paper motivates scene-aware interaction as important for realistic reconstruction and virtual character placement.
  • POSA model: POSA extends SMPL-X with a probabilistic, body-centric feature map that predicts per-vertex contact probabilities and semantic labels.The representation makes human-scene interaction an explicit and integral part of the body model.
  • Training: POSA is a conditional VAE trained on SMPL-X meshes from 20 subjects interacting with 12 real 3D scenes, using PROX and PROX-E semantics.Given a posed body, the trained model samples likely contacts and semantic labels for its vertices.
  • Applications: For scene population, POSA searches for placements whose observed feature maps match cVAE-generated targets and achieves higher perceptual realism than PLACE.The method uses SMPL-X bodies fitted to commercial scans as proxies for positioning people in scenes.
  • Applications: For monocular 3D pose estimation, replacing PROX’s hand-coded contacts with POSA’s learned feature map produces lower pose estimation errors.The learned feature map functions as a human-scene interaction prior and automates a heuristic process.
  • Contribution: The model and code are released for research, positioning POSA as a platform for extending learned human body models with body-scene interaction.The paper describes this as the first learned human body model incorporating human-scene interaction.

2. Related Work

Prior work models humans, scenes, and their interactions through isolated representations, geometric constraints, data-driven predictors, or object-centric encodings. POSA differs by using a human-centric representation that combines contact and semantics and is intended to transfer more readily to new scenes.

  • Humans and scenes in isolation: Earlier work often represents scenes without humans and human pose or shape outside scene context, while skeleton-only models omit the interacting body surface.This related-work gap motivates representations that capture surface-level human-scene relationships.
  • Geometric HSI models: Geometric HSI methods use contact constraints, motion patches, force equilibrium, ergonomic rules, or interaction meshes to model spatial and physical relationships.These approaches include both static and dynamic interaction formulations.
  • Data-driven HSI models: Data-driven HSI methods learn pose, affordance, scene-place, or motion relationships from RGB-D scenes, annotations, CAD objects, or graph-based representations.The cited methods span scene-label estimation, action-specific affordance detection, and joint human-object motion prediction.
  • Hand-object interaction: Hand-object interaction work predicts contact or grasps using hand-object mesh features, two-stage networks, or SMPL-X and MANO representations.These methods focus specifically on grasping rather than general human-scene interaction.
  • Closest related work: PSI implicitly encodes HSI from depth and semantics, whereas PLACE explicitly models scene shape and proximal relations without semantics.POSA’s key difference is its human-centric formulation, which the paper states is more portable to new scenes.

3. Method

POSA represents human-scene interaction on the SMPL-X body mesh through per-vertex contact and semantic features, then learns their distribution from body pose and shape. The method constructs these features from body-scene proximity and models them with a graph-based conditional VAE.

  • Human Pose and Scene Representation: The training corpus contains pairs of SMPL-X body meshes and 3D scene meshes with per-vertex semantic labels.Human meshes use a fixed topology with 10,475 corresponding vertices.
  • POSA Representation for HSI: POSA encodes human-scene interaction as an egocentric feature map defined on SMPL-X body vertices.The map combines a contact label with the semantic label of the contacted scene surface.
  • Learning: POSA learns a probabilistic mapping from body pose and shape to likely per-vertex contacts and corresponding semantic labels without taking a scene as input.Global translation and rotations around the y and z axes are factored out, while x-axis rotation remains to distinguish orientations such as standing and lying.
  • Learning: A cVAE conditions the feature map on SMPL-X vertex positions and reconstructs contact and semantic labels using KL regularization and reconstruction losses.The total loss combines α ∗ LKL with Lrec, using α = 0.05 and equal contact and semantic loss weights.

4. Experiments

POSA is evaluated for generating human-scene interaction features, placing bodies in real and synthetic scenes, and supporting pose estimation. The experiments also examine perceptual realism, physical plausibility, and practical limitations involving clothed bodies.

  • 4.1. Random Sampling: POSA samples contact and semantic feature maps from novel body poses, predicting likely interacting vertices and scene objects.The representation is generated from the body pose alone, and its probabilistic decoder supports multiple feature maps for one pose.
  • 4.1. Random Sampling: For standing poses, POSA predicts feet-floor and hand-wall contact, while lying poses produce back-bed or back-sofa contacts.These examples illustrate pose-dependent contact and semantic predictions across different scene contexts.
  • 4.2. Affordances: Putting People in Scenes: The placement pipeline samples a POSA feature map from the posed body, then optimizes translation, orientation, and pose using affordance, penetration, and regularization terms.The pose can change slightly from its initial value to better fit the scene, while the regularizer keeps the adjustment limited.
  • 4.2. Affordances: Putting People in Scenes: POSA automatically places SMPL-X bodies in unseen real PROX scenes and synthetic scenes, producing realistic interactions for poses including lying down and reaching.The reported examples include bodies optimized in a real PROX test scene and in an artist-designed synthetic scene.
  • 4.2. Affordances: Putting People in Scenes: Clothed Renderpeople bodies are placed in Replica scenes by fitting SMPL-X proxies while preserving the original clothed meshes after optimization.This addresses the visual mismatch between unclothed SMPL-X bodies and realistic scenes, while retaining the consistent topology needed for feature prediction.
  • 4.2.1 Evaluation: Perceptual studies compare generated interactions with PROX ground truth and PLACE, while additional experiments evaluate physical plausibility and monocular pose estimation.For pose estimation, the learned feature map replaces PROX’s hand-coded contact points and yields lower pose estimation errors than PROX.

5. Conclusions

POSA extends 3D human body modeling beyond isolated pose and shape by explicitly representing human-scene interactions. The paper shows that this representation supports realistic scene placement and improves scene-consistent pose estimation, while requiring an accurate scene SDF and focusing on a single body mesh.

  • POSA explicitly represents contact and semantic relationships between a body and its surrounding scene.
  • POSA places scanned people in likely scene locations with higher realism than the state of the art.
  • POSA improves monocular RGB-image human pose estimation by incorporating body-centered human-scene interaction.
  • Limitations: POSA requires an accurate scene SDF because noisy scene meshes can cause body-scene penetration.
  • Limitations: POSA focuses on a single body mesh rather than interactions involving multiple bodies.

A. Training Details

Training uses canonicalized body orientations, a graph-convolutional encoder-decoder, and scene signed-distance fields to represent human-scene interactions. The implementation downsamples meshes for efficiency and trains with Adam.

  • The model canonicalizes global y- and z-axis rotations while retaining x-axis rotation to distinguish standing from lying poses.
  • The encoder uses three spiral-convolution and pooling blocks followed by a 512-unit fully connected layer.
  • The input mesh is downsampled from 10,475 to 655 vertices for computational efficiency.
  • Training uses PyTorch with Adam, batch size 64, and learning rate 1e−3 without decay.
  • A 512 × 512 × 512 signed distance field stores signed distances from voxel centroids to the nearest scene surface.

C. Random Samples

Random feature-map samples produce varied but plausible human-scene interactions for the same pose. They consistently capture salient contacts while sometimes assigning semantically inaccurate scene labels, and the pipeline uses these maps to place clothed bodies.

  • POSA generates multiple valid feature maps for the same pose, with feet consistently predicted in contact with the floor.
  • Samples vary between sitting on a chair and sitting on a sofa for the same pose.
  • The predicted semantic map can incorrectly associate lower legs with a bed while predicting that the person sits on a sofa.
  • Affordance detection: For clothed-body placement, the pipeline fits SMPL-X, samples a feature map, minimizes the affordance energy, and replaces the proxy mesh with the original clothed mesh.
  • Additional qualitative examples cover SMPL-X and clothed bodies in real and synthetic scenes, including comparisons with PLACE.

E. Failure Cases

Failure cases arise from residual body-scene penetrations and incorrect feature maps. These failures are linked to thin surfaces missed by the SDF, local optimization minima, and limited training data for unseen poses.

  • Residual body-scene penetrations can remain despite the penetration penalty.
  • Thin surfaces missing from the SDF or optimization becoming stuck in a local minimum can cause penetration failures.
  • Feature maps can be incorrect when the model generalizes poorly to test poses because of limited training data.

F. Effect of Shape

POSA predicts plausible interaction feature maps across a wide range of human body shapes.

  • POSA predicts plausible feature maps for a wide range of human body shapes.

G. Scene population.

Scene population searches candidate body placements using POSA features, retains the 10 lowest-energy candidates, and iteratively optimizes them. The pipeline supports realistic clothed scans and produces qualitative placements in real and synthetic scenes.

  • Scene population evaluates a regular grid of candidate positions, keeps the 10 lowest-energy candidates, and iteratively optimizes them.The final result highlights the best position among the optimized candidates.
  • Random cVAE samples for the same pose visualize contact probabilities and semantic scene categories through separate feature maps.
  • The clothed-scan pipeline uses SMPL-X as a proxy, samples POSA features for the pose, optimizes placement against the scene mesh, and replaces the proxy with the scan.
  • Qualitative examples show SMPL-X meshes automatically placed in real and synthetic scenes, including body shapes and poses not used during training.
Loading 2012.11581v2…