Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments

1. Introduction

• The proposed pipeline has two stages: automatic 3D pose synthesis followed by end-to-end prediction of pose locations and poses.The model jointly learns the distributions of locations and poses.Source: Figure 1 · p. 1, §1 · p. 2
• Prior video-based affordance methods can produce semantically plausible poses but lack 3D geometry and may violate scene physics.The introduction attributes this limitation to the use of video frames and 2D poses without 3D geometric information.Source: §1 · pp. 1–2
• The synthesizer maps 2D poses into annotated indoor scenes and adjusts them in voxel space to avoid intersections and ensure support.It combines natural-behavior knowledge from 2D poses with 3D scene geometry to produce diverse poses automatically.Source: §1 · p. 2
• The prediction model generates full-3D human poses from a single scene image by jointly learning distributions of pose locations and pose appearance.The two distributions are learned through the where and what modules, linked by pose-pelvis locations.Source: §1 · p. 2
• The work introduces a large-scale SUNCG benchmark for human-centric affordance prediction using its pose synthesizer and pose generator.The stated contributions pair automatic pose synthesis with generative prediction of plausible, physically feasible poses.Source: §1 · p. 2

2. Related Work

• Scene-understanding methods detect human-object interactions, while object-functionality methods reason about what objects afford.The latter include functionality-based chair detection and tool-function inference from RGB-D video.Source: §2 · p. 2
• Several human-affordance approaches use data-driven predictions, but earlier video-based collection required manual effort and yielded about 20K examples.The passage also notes that limited data and geometric knowledge make it difficult for CNNs to follow scene geometry.Source: §2 · p. 2
• Instance-placement methods predict locations or appearances, but cited examples require a user-provided template or insert instances in 2D without 3D physical feasibility.The closest cited approach jointly models context-aware location and shape distributions, but focuses on 2D images.Source: §2 · pp. 2–3

3. 3D Pose Synthesis

• The synthesizer learns natural pose distributions from 2D examples, maps generated poses into voxel scenes, and adjusts them for physical correctness.The two target properties are semantic plausibility and avoiding collisions or poses floating in the air.Source: §3 · p. 3

3.1. Affordance Prediction in 2D Scene Images

• The 2D stage adapts a Sitcom-trained pose model to SUNCG, predicting feasible locations and likely pose classes despite sparse annotations.Annotations are expanded from points to local patches, and the model also maps Sitcom poses to 3D poses for generation at sampled locations.Source: §3.1 · p. 3

3.2. Mapping Poses into 3D Scenes

• Pose depth is estimated from sampled standing or sitting human heights, then camera parameters map the joints from image coordinates into 3D.The sampled height distributions are N(1.65, 0.1) for standing and N(1.20, 0.1) for sitting.Source: §3.2 · pp. 3–4

3.3. Affordance Constraint in the 3D World

• 3D adjustment checks free space while allowing necessary contacts, then searches upward-facing affordable surfaces for supported placements and discards poses without valid locations.The free-space test uses pose-to-voxel correlation; support placement uses pose-to-surface correlation and a maximum-score location within a local region.Source: §3.3 · p. 4

4. 3D Affordance Generative Model

• The model jointly learns where poses can occur and what gestures fit those locations, using scene-conditioned VAEs and a geometry-aware discriminator.The pose location links the two subtasks and lets them benefit from each other and the discriminator.Source: §4 · pp. 4–5, Figure 4 · p. 5

4.1. The Where Module: Pose Locations Prediction

• The where module predicts pelvis location, depth, and pose class; sampled locations include sitting near affordable objects and standing at suitable heights.It represents pose classes by normalized cluster-center poses, so similar classes have similar representations.Source: §4.1 · pp. 5–6

4.2. The What Module: Pose Gestures Prediction

• The what module takes the predicted location, depth, pose class, and scene image to generate joint coordinates and depths aligned with scene context.It models pose appearance with a conditional VAE and uses joint-coordinate, KL-divergence, and geometry losses.Source: §4.2 · p. 6

4.3. The Geometry-Aware Discriminator

• The geometry-aware discriminator uses pose and scene-depth information to regularize physical feasibility, but the discriminator and feasibility labels are used only during training.A pretrained network converts joint coordinates and depth into a depth heat map for the discriminator.Source: §4.3 · p. 6

5.1. Dataset Synthesis and Evaluation Metrics

• The synthesizer produces 1.5 million poses across 13,774 SUNCG scenes; evaluation uses an 86%-accurate authenticity classifier and scene-geometry constraints.The classifier's positive-pose ratio is the semantic score; the geometry score measures poses satisfying free-space and support constraints.Source: §5.1 · pp. 6–8
• With RGB input, the model scores 19.47% higher on semantic score and 19.02% higher on geometry score than the baseline; depth input further improves geometry scores.The reported RGB-D and depth-map poses align with the sofa where the RGB-only pose floats above it.Source: §5.2 · p. 8, Table 1 · p. 9

5.3. Ablation Studies

• A single VAE scores 76.23 semantic and 52.94 geometry; removing joint training lowers RGB semantic score by 8.06%.Without joint training, some locations have wrong depth values and generate unreasonably small poses.Source: §5.3 · p. 8
• Geometry-aware adversarial training improves the geometry score by helping generated poses obey scene rules.The discriminator conditions on the scene depth map and distinguishes generated poses from ground-truth poses.Source: §5.3 · p. 8
• Without geometry loss, RGB-input geometry score drops by 4.59%.The loss minimizes the Euclidean distance between predicted and ground-truth poses in world coordinates.Source: §5.3 · p. 8

5.4. Comparison with State-of-the-Art

• 6.66% higher geometry score than the baseline at given locations; the 3D voxel views show the baseline pose is physically invalid despite plausible 2D projections.Table 2 compares the what modules; Figure 10 shows baseline poses in the top row and the proposed method in the bottom row.Source: §5.4 · p. 9, Table 2 · p. 9, Figure 10 · p. 9
• Interpolating pelvis depth changes pose scale and action to fit the scene, including a standing-to-sitting change near a sofa.The test fixes pelvis image coordinates and the scene while varying depth from d−0.5 to d+0.5.Source: Figure 9 · p. 8, §5.4 · p. 9

5.5. Failure Cases

• Failures include semantically inappropriate poses, such as sitting on a cabinet, and geometrically invalid poses that collide with objects.The authors attribute these to object-function understanding or 3D geometry hallucination from 2D information.Source: §5.5 · p. 9, Figure 11 · p. 9

• The two-stage pipeline synthesizes millions of 3D poses by fusing semantic knowledge from the Sitcom dataset with geometric knowledge from a 3D scene dataset.It then predicts pose locations and gestures that are semantically plausible and geometrically feasible.Source: §6 · p. 9
• Experimental results demonstrate the method's effectiveness against the state-of-the-art human affordance prediction method.Source: §6 · p. 9

• Figure 12 shows examples of 2D-to-3D pose mappings, with corresponding 3D poses visualized from two views.Source: Figure 12 · p. 11
• Each 2D pose is mapped to the nearest projected 3D pose after Human3.6M poses are uniformly rotated through angles in [−π, π] and projected onto the xy plane.The selected 3D pose minimizes Euclidean distance to the 2D pose among the projections.Source: A. From 2D Pose to 3D Pose (Section 3.1) · p. 11
• Each pose is represented by 17 joint coordinates, and its location is represented by the pelvis joint coordinates.This yields 34-dimensional 2D poses and 51-dimensional 3D poses.Source: A. From 2D Pose to 3D Pose (Section 3.1) · p. 11

• Pose depth is estimated from the generated image pose and an approximated real-world human height.Source: B. Mapping Poses into 3D Scenes… · p. 11
• Real-world height is sampled from N(1.65, 0.1) for standing poses and N(1.20, 0.1) for sitting poses.The image-space highest and lowest joints define pose height and width for the calculation.Source: B. Mapping Poses into 3D Scenes… · p. 11
• The depth calculation uses camera-coordinate and world-coordinate relations for the highest and lowest joints, with camera intrinsic and extrinsic parameters.For SUNCG scenes, the passage notes that r32 = 0 and that the depth equation is simplified.Source: B. Mapping Poses into 3D Scenes… · p. 11

• The pelvis-location model uses an 18-layer ResNet encoder and an 11-layer CNN decoder, followed by Softmax to normalize the generated heat map.Source: Figure 14 · p. 11
• The heat-map model is trained for 5,000 iterations with Adam, using batch size 100 and learning rate 0.001.Training crops randomly sample 384 × 384 patches from 448 × 448 images.Source: C. Location Prediction in 2D Scene… · pp. 11–12
• At given locations, pose generation directly predicts 3D poses obtained via 2D-to-3D pose mapping rather than predicting 2D poses.Source: C. Location Prediction in 2D Scene… · p. 12

D. More Details for 3D Pose Prediction (Section 4)

• The where and what modules are pretrained for 80,000 iterations, then jointly fine-tuned with a geometry-aware discriminator for 50,000 more.Pretraining uses batch size 100 and learning rate 0.0001.Source: D. More Details for 3D Pose Prediction… · p. 12
• The discriminator regularizes generated poses from the unsupervised paths rather than directly interacting with the VAE blocks.The authors report that this architecture significantly improves generated results.Source: D. More Details for 3D Pose Prediction… · p. 12
• A CNN converts joint coordinates and depth into an input-sized depth heat map for the geometry-aware discriminator.The CNN is trained for 5,000 iterations with learning rate 0.0002.Source: D. More Details for 3D Pose Prediction… · p. 12

E. Additional Experimental Results

• Despite rendering noise in SUNCG-PBR, the pose prediction model still predicts plausible poses.The examples show synthesized poses in scene images and voxels.Source: E. Additional Experimental Results · p. 12
• Synthesized poses contain information about scene occlusion.Figure 15 pairs poses shown in 3D scene geometry with rendered images of the corresponding scenes.Source: Figure 15 · p. 13
• Figure 16 shows generated poses from a single depth image, with poses in images and voxel views from two viewpoints.Source: Figure 16 · p. 13
• Figure 17 visualizes sitting locations in red, standing locations in blue, and areas unsuitable for poses in light yellow, with examples from Sitcom and SUNCG.The bottom-row poses are projections of generated 3D poses.Source: Figure 17 · p. 13
• Figure 18 presents the overall pose-prediction model and the detailed structures of its blocks.The overview includes supervised and unsupervised paths.Source: Figure 18 · p. 14