PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating 3D People in Scenes without People

Yan Zhang, Mohamed Hassan, Heiko Neumann, Michael J. Black, Siyu Tang

arXiv:1912.02923v3cs.CV

TL;DR

The paper tackles automatic generation of realistic people in 3D scenes, where bodies must be semantically appropriate and physically feasible. It uses a scene-conditioned CVAE with SMPL-X and geometry-aware refinement, producing realistic, expressive bodies that interact plausibly with environments and outperform a baseline. The learned interaction prior also improves 3D pose estimation from RGB images.

  • Problem

    People-free 3D scenes lack automated tools for generating realistic, diverse, naturally posed humans that interact semantically and physically plausibly with scene geometry.

  • Method

    A CVAE conditioned on scene context generates SMPL-X body meshes, which geometry-aware fitting refines to enforce contact and prevent interpenetration.

  • Results

    The method generates realistic and expressive bodies, outperforms the baseline, and improves 3D pose estimation from RGB images using its human-scene interaction prior.

  • Takeaways & Limitations

    Scene-conditioned human-scene interaction priors support realistic 3D body synthesis and can improve downstream pose estimation.

  • Takeaways & Limitations

    The baseline data and pre-trained model are based on SUNCG and are not publicly available.

Abstract

from arXiv · show

We present a fully automatic system that takes a 3D scene and generates plausible 3D human bodies that are posed naturally in that 3D scene. Given a 3D scene without people, humans can easily imagine how people could interact with the scene and the objects in it. However, this is a challenging task for a computer as solving it requires that (1) the generated human bodies to be semantically plausible within the 3D environment (e.g. people sitting on the sofa or cooking near the stove), and (2) the generated human-scene interaction to be physically feasible such that the human body and scene do not interpenetrate while, at the same time, body-scene contact supports physical interactions. To that end, we make use of the surface-based 3D human model SMPL-X. We first train a conditional variational autoencoder to predict semantically plausible 3D human poses conditioned on latent scene representations, then we further refine the generated 3D bodies using scene constraints to enforce feasible physical interaction. We show that our approach is able to synthesize realistic and expressive 3D human bodies that naturally interact with 3D environment. We perform extensive experiments demonstrating that our generative framework compares favorably with existing methods, both qualitatively and quantitatively. We believe that our scene-conditioned 3D human generation pipeline will be useful for numerous applications; e.g. to generate training data for human pose estimation, in video games and in VR/AR. Our project page for data and code can be seen at: \url{https://vlg.inf.ethz.ch/projects/PSI/}.

1. Introduction

The paper addresses the absence of people in 3D scene datasets by automatically generating diverse, naturally posed human bodies that are semantically and physically plausible in their environments. It combines learned interaction priors with geometry-aware refinement to produce realistic scene interactions.

  • The system generates diverse 3D human bodies that are naturally posed and physically plausible within people-free 3D scenes.Generated bodies should neither float nor interpenetrate objects while interacting naturally with the environment.
  • A conditional variational autoencoder learns semantically plausible human-scene interactions, followed by geometry-aware fitting that enforces feasible contact and collision constraints.
  • SMPL-X provides a detailed, differentiable body mesh that supports contact reasoning, collision prevention, and variation in body shape and pose.Its surface representation includes details such as fingers and enables optimization of human-scene contact.
  • The method is trained with PROX-Qualitative extensions and evaluated in Matterport3D rooms using diversity, physical plausibility, and semantic plausibility metrics.
  • Experiments show that the method outperforms a baseline and that its interaction prior improves 3D pose estimation from RGB images.

2. Related work

Prior work models scene affordances and inserts people or objects into scenes, but commonly relies on 2D compositing, limited interaction modeling, paired data, or coarse stick-figure bodies. These limitations motivate detailed, physically grounded 3D human-scene generation.

  • 2D compositing and inpainting methods add objects or people without modeling interactions between synthetic humans and the surrounding world.
  • Affordance methods represent possible object interactions through human poses and predict likely poses or pose locations for objects and scenes.
  • Sitcom- and 3D-scene-based methods require substantial paired data and manual pose cleaning, limiting generalization to less frequently covered scenes.
  • Stick-figure pose representations lack visual detail and are difficult to validate as realistic human bodies.

3. Methods

The method represents scenes with depth and semantics and bodies with SMPL-X, then learns scene-conditioned body distributions using CVAEs before refining outputs with contact and collision-aware geometry optimization. The training objective combines reconstruction, latent regularization, natural-pose, contact, and collision terms.

  • 3D scene representation: Each scene view is represented by stacked depth and semantic information captured from a single embodied-agent camera view.Inputs are resized to 128 × 128, while projection and inverse projection map between normalized image and 3D coordinates.
  • 3D human body representation: SMPL-X maps low-dimensional shape, pose, hand-pose, rotation, and translation features to a 3D mesh with 10,475 vertices.The joint representation is used to transform generated bodies into world coordinates.
  • Scene-conditioned generation: The CVAE models p(x_h|x_s) either jointly in a one-stage network or successively through global and local body features in a two-stage network.Scene encoders are fine-tuned from ResNet18, and encoded scene features are concatenated with human features.
  • Training loss: The training loss combines reconstruction, KL-divergence, VPoser, human-scene contact, and collision terms with weighted coefficients.The VPoser term encourages natural poses, while the interaction terms incorporate scene contact and collision constraints.
  • Scene geometry-aware fitting: Geometry-aware fitting refines generated meshes to encourage contact and avoid interpenetration without substantially deviating from the generated pose.The fitting objective uses scene geometry and the previously defined loss terms.
  • Implementation: Training uses Adam with learning rate 3e−4, and human-scene interaction losses are enabled after 75% of training epochs.

4. Experiments 1

The experiments extend PROX-Qualitative into PROX-E and evaluate scene-aware human generation using diversity, physical, and perceptual metrics. Results show improved diversity, physical plausibility, and pose estimation compared with baselines.

  • Datasets: PROX-E extends PROX-Qualitative with virtual-camera renderings and uses pseudo-ground-truth SMPL-X meshes from captured human-scene interactions.The evaluation includes virtual-camera training data and real-camera validation and test samples.
  • Evaluation: representation power: The evaluation measures representation power with reconstruction error and negative ELBO, while comparing models with and without human-scene interaction loss.The negative ELBO combines reconstruction error and KL divergence.
  • Evaluation: 3D body mesh generation: The generation evaluation combines diversity, physical plausibility, and user-perceived semantic plausibility across PROX-E and MP3D-R.Diversity considers global positions, body shapes, and poses; physical evaluation measures collision and contact.
  • Evaluation: 3D body mesh generation: Our methods consistently outperform the baseline in diversity and physical metrics, and scene-geometry-aware fitting improves physical plausibility.The reported diversity gains are large, while the influence of the LHS loss is less obvious for body generation than for interaction modeling.
  • Evaluation: 3D body mesh generation: The non-collision score is the fraction of SMPL-X vertices in free space, while contact ratio is the fraction of generated bodies contacting the scene.Higher values indicate better physical generation under the stated constraints.
  • Evaluation: 3D body mesh generation: User scores are similar for the baseline and proposed model, but the proposed method generates more diverse bodies while remaining equally semantically plausible.Geometry-aware fitting improves user-study performance for both systems.
  • Scene-aware 3D Body Pose Estimation: Using the learned scene-dependent pose prior improves 3D body pose estimation from RGB images on PROX-Quantitative.This result supports that the model captures ways people interact with 3D scenes.

5. Conclusion

The framework generates 3D human bodies naturally posed in environments through scene-conditioned generation followed by geometry-aware fitting. Experiments report realistic, expressive bodies with semantically and physically plausible scene interactions.

  • The method generates 3D human bodies with natural poses in 3D environments.
  • A scene context-aware generator learns pose and shape distributions from scene depth and semantics.
  • Geometry-aware fitting imposes physical plausibility on human-scene interactions.
  • Experiments show synthesized bodies are realistic, expressive, and interact with environments in semantically and physically plausible ways.

A. Experiment Details

PROX-E extends PROX-Q to support human-scene interaction modeling from embodied viewpoints. The extension completes room geometry, adds semantics, and expands training views with virtual cameras.

  • PROX-Q records real human interactions in 12 indoor scenes, but its scans lack complete rooms and mesh semantics.
  • Virtual walls, floors, and ceilings enclose the original scans so virtual cameras produce completed depth maps.
  • Manual semantic annotation follows Matterport3D’s 40 common indoor-object categories and supplies scene semantics alongside depth.
  • Virtual cameras are placed according to room structure and human-body position to augment the single real-camera view.

A.2. Creating the MP3D-R dataset

MP3D-R contains seven Matterport3D rooms captured from multiple Habitat-agent views with RGB, depth, semantics, and camera parameters. Scene SDFs are computed after watertight reconstruction.

  • MP3D-R comprises seven Matterport3D rooms selected by annotated bounding boxes and matched to PROX-E room types.
  • Matterport3D meshes are expanded by 0.5 meters around annotated boxes, then rotated to match Habitat’s −Y gravity direction.
  • A Habitat agent captures RGB, depth, semantics, and camera extrinsics from sensors 1.8 meters above the ground.
  • Each room’s mesh is made watertight with Poisson surface reconstruction before computing an SDF on a 256 × 256 × 256 voxel grid.

A.3. Details of the baseline method

The baseline comparison adapts prior stick-figure synthesis to body-mesh generation and trains it on PROX-E. The study evaluates naturalness while highlighting differences in data, representation, and scene inputs.

  • Unlike the prior method, PROX-E provides real human-scene interactions rather than synthesized interactions.
  • SMPL-X enables varied body shapes, fine-grained hand poses, and differentiable surface constraints beyond stick-figure representations.
  • The baseline adapts Li et al.’s automatic stick-figure synthesizer to generate body meshes for fair comparison.
  • The user study renders each result from two views and records naturalness judgments on a 5-point Likert scale.
  • Geometry-aware fitting reduces the standard error of user-study scores, indicating more consistent judgments.

A.5. More discussions on model training

The training scheme uses empirically selected loss weights and KL-divergence annealing to preserve latent-variable representation power, pose plausibility, and variability.

  • The KL-divergence weight is set to 0.1 to improve latent-variable representation power.KL annealing helps avoid a collapsed VAE posterior that would output a constant result regardless of the latent variable.
  • The VPoser weight is set to 0.001 to balance plausibility and variability in generated body poses.

B. Generative Model Latent Space Analysis

The learned 32D latent space produces smooth, partly disentangled changes in body configuration and captures scene-dependent interactions, although poses become less plausible far from the latent origin.

  • Generated bodies smoothly transition across the latent space and learn interactions such as standing on floors, touching desks, and sitting on beds.
  • Different latent dimensions partly disentangle body movement, with examples dominated by motion along the world X or Y direction.
  • Pose plausibility decreases when latent variables move far from zero, resembling the behavior of VPoser.

C. More Qualitative Results and Failure Cases

Qualitative results show generated bodies and geometry-aware fitting across PROX-E and MP3D-R, while failures arise from unreliable samples, imperfect physics, and noisy or complex scene geometry.

  • Qualitative Results: Qualitative evaluations present generative-model outputs and scene geometry-aware fitting in PROX-E and MP3D-R.
  • Failure Cases: Generative samples can mismatch test-scene geometry, causing bodies to float or collide with the scene mesh.
  • Latent-Space Analysis: The 32D latent-space analysis visualizes bodies by varying all, first, last, or middle subsets of latent dimensions along lines from -3 to 3.
  • Failure Cases: Geometry-aware fitting resolves floating and collision but cannot simulate all real physics, including gravity and elasticity, and may produce abnormal configurations or contacts.
  • Failure Cases: Generated-body quality depends on test-scene data quality; noisy scans and complex structures can degrade inputs and make fitting fail.

D. Details of Scene-aware 3D Body Pose Estimation

Scene-aware pose estimation modifies an existing optimization objective by replacing its VPoser regularizer with a learned scene-dependent pose prior while leaving other terms unchanged.

  • Optimization Objective: The pose-estimation experiment modifies prior work's objective to incorporate the learned scene-dependent pose prior.
  • Optimization Objective: The full objective combines joint, depth, body-pose, hand-pose, head-pose, angle, shape, and additional regularization terms with learned weights.
  • Optimization Objective: Only the VPoser regularizer is modified, while the other objective terms remain unchanged.
  • Initialization: Optimization initializes the pose feature with the scene-dependent prior instead of the zero vector used in prior work.
  • Initialization: Using 1.5λθb for the modified regularizer weight yielded better performance in the authors' trials.
  • Qualitative Evaluation: Supplementary qualitative figures compare results before and after fitting for baseline and S1 models in PROX-E and MP3D-R, including failure cases.
Loading 1912.02923v3…