Source-linked AI summary
Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
Yixin Chen, Siyuan Huang, Tao Yuan, Siyuan Qi, Yixin Zhu, Song-Chun Zhu
TL;DR
Single-view methods have generally addressed 3D scene understanding or 3D human pose estimation separately, while holistic++ jointly estimates both using HOI and physical commonsense. Experiments across three datasets report improvements over state-of-the-art methods on both tasks.
Problem
Single-view research has largely treated 3D scene understanding and 3D human pose estimation separately, despite their coupled nature in indoor scenes.
Source: §1 · p. 1Method
Results
The method outperforms state-of-the-art methods for both 3D scene reconstruction and 3D pose estimation on PiGraphs, Watch-n-Patch, and SUN RGB-D.
Source: §1 · p. 2Takeaways & Limitations
HOI and physical commonsense provide general priors that support plausible predictions and improved generalization across indoor scene datasets.
Source: §1 · p. 2, §7 · p. 83D pose initialization assumes a predefined height h0 for the joint position in world coordinates.
Source: §5.2 · p. 5
Abstract
from arXiv · showhide
We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation. The intuition behind is to leverage the coupled nature of these two tasks to improve the granularity and performance of scene understanding. We propose to exploit two critical and essential connections between these two tasks: (i) human-object interaction (HOI) to model the fine-grained relations between agents and objects in the scene, and (ii) physical commonsense to model the physical plausibility of the reconstructed scene. The optimal configuration of the 3D scene, represented by a parse graph, is inferred using Markov chain Monte Carlo (MCMC), which efficiently traverses through the non-differentiable joint solution space. Experimental results demonstrate that the proposed algorithm significantly improves the performance of the two tasks on three datasets, showing an improved generalization ability.
1. Introduction
Holistic++ jointly estimates 3D human poses and scene structure from one image, using human-object interaction and physical commonsense to constrain the reconstruction. The method outperforms state-of-the-art approaches on both tasks across three datasets; related work had not jointly addressed these detailed human-scene relations.
1. Introduction
- Holistic++ jointly infers 3D human poses, objects, room layout, and camera pose from a single view, addressing a task few prior methods attempted.Prior work generally focused on either holistic scene understanding or human pose estimation, despite their close ties to human-designed indoor activities.Source: §1 · pp. 1–2
- Learned HOI priors constrain fine-grained human-scene relations, reducing single-view geometric ambiguity and helping recover interacted objects such as a keyboard.Recognizing sitting can constrain a person’s position and orientation relative to a chair.Source: §1 · p. 2
- Physical commonsense penalizes physical violations, including support-relation violations, to produce plausible, stable scenes and improve human and object localization.The authors describe HOI and physical commonsense as general priors that support generalization across datasets.Source: §1 · p. 2
- On PiGraphs, Watch-n-Patch, and SUN RGB-D, the method outperforms state-of-the-art methods for both 3D scene reconstruction and 3D pose estimation.Ablations show that HOI improves reconstruction and physical commonsense supports physically plausible predictions.Source: §1 · p. 2
1.1. Related Work
- Prior work addressed pose estimation in clutter, scene reconstruction with human context, HOI, and physical properties separately, but left detailed human-scene relations under-modeled.This method uses 3D HOI priors to refine human-scene spatial relations and models physical relations between humans, objects, and room layout.Source: §1.1 · pp. 2–3
2. Representation
The scene is encoded as a parse graph combining a hierarchical parse tree with contextual links among terminal nodes. Its nodes represent room layout, objects, and people, while its links encode physical relations and human-object interactions.
2. Representation
- The parse graph combines a hierarchical parse tree with contextual relations among terminal nodes.The tree has scene, middle, and terminal levels; its middle nodes represent objects, humans, and room layout.Source: §2 · p. 3
- Terminal nodes represent room layout and objects with 3D bounding boxes, and people with 17 3D joints and concurrent-action labels.Human pose attributes include scale, hip-centered pose center, local joint positions, and orientation.Source: §2 · p. 3
- Contextual links encode support and collision relations for physical plausibility, and HOI relations that constrain human-object positions using learned interaction priors.For example, a detected sitting action constrains the relative 3D positions of a person and chair.Source: §2 · p. 3
3. Probabilistic Formulation
The formulation represents the scene as a parse graph and seeks its maximum a posteriori interpretation given the image. Image consistency, physical plausibility, and human-object interactions contribute to evaluating that interpretation.
3. Probabilistic Formulation
- The optimal parse graph is inferred by maximum a posteriori estimation, with image likelihood and physical and HOI priors contributing to its probability.The likelihood measures consistency between the observed image and inferred 3D scene; the prior combines physical and interaction terms.Source: §3 · pp. 3–4
- The physical prior combines support and collision energies to represent physical commonsense among scene entities.Support costs include overlap and vertical separation; collision costs penalize intersections, with exceptions for HOI objects and possible containers.Source: §3 · pp. 3–4
- The HOI prior scores human-object interactions using the likelihood of relative object position conditioned on the person's action label.For example, the likelihood for sitting models the spatial relation between the hip and a chair center.Source: §3 · p. 4
- Image likelihood compares detected 2D object boxes and human poses with projections of the inferred 3D objects and poses.The formulation uses box intersection-over-union and average pixel-wise Euclidean pose distance.Source: §3 · p. 4
4. SHADE Dataset
SHADE is a self-annotated dataset of dynamic 3D human skeletons and objects, collected from Grand Theft Auto V to learn HOI priors.
4. SHADE Dataset
- SHADE contains over 29 million 3D human-pose frames, including 772,229 annotated frames; six final HOIs were selected from 19 fine-grained HOIs by frequency and merging similar HOIs.Annotated frames average 2.03 action labels and 0.89 HOIs. The selected HOIs include reading, sitting, making a phone call, holding, and using a laptop.Source: §4 · p. 4
5. Joint Inference
Joint inference seeks the parse graph with the highest posterior probability from a single RGB image, progressing from scene and pose initialization through action detection to joint 3D optimization.
5. Joint Inference
- Joint inference maximizes the parse graph posterior through scene initialization, human-pose initialization, concurrent action detection, and joint optimization.Source: §5 · p. 4
5.1. 3D Scene Initialization
- Scene initialization cooperatively estimates the camera pose, room layout, and 3D object boxes, assigning each object a supporting object or layout.Source: §5.1 · p. 4
5.2. 3D Human Pose Initialization
- Human-pose initialization maps detected 2D poses into local 3D coordinates, then aligns them to the world using a visible joint and an assumed joint height.Source: §5.2 · p. 5
5.3. Concurrent Action Detection
- Concurrent action detection predicts multiple HOI and general-pose labels from 2D skeletons, using manually mapped HOI relations as reconstruction priors.Source: §5.3 · p. 5
5.4. Inference
- Simulated-annealing MCMC optimizes layout, objects, and poses in phases, adding HOI matching and joint constraints before top-down sampling can recover missed interacting objects.Source: §5.4 · pp. 5–6
6. Experiments
The evaluation compares joint scene reconstruction and pose estimation across PiGraphs, Watch-n-Patch, and SUN RGB-D using real, partially annotated, and synthetic data. The authors report improved reconstruction and cross-dataset pose-estimation performance, with ablations examining HOI and physical-commonsense contributions.
6.2. Dataset
- Evaluation spans fully annotated real PiGraphs, partially annotated real Watch-n-Patch, and SUN RGB-D augmented with synthetic human poses.PiGraphs has 36,551 test images; Watch-n-Patch contributes 1,210 selected frames, and filtered SUN RGB-D contributes 3,476 testing images.Source: §6 · p. 6, §6.2 · p. 6
6.1. Comparative methods
- Comparisons use [15] for single-image scene reconstruction and VNect [27] for world-coordinate pose estimation; a five-layer network provides a multi-person pose baseline.The baseline uses GGN image features, 2D and local 3D pose, and camera intrinsics, and is trained on synthetic SUN RGB-D.Source: §6.1 · p. 6
6.2. Dataset
- Synthetic SUN RGB-D poses are sampled in 3D scenes and projected into images using the ground-truth camera; pose comparisons use ground-truth 2D skeleton inputs.The imagined human pose is parameterized by action, pose template, translation, rotation, and scale; the sampled pose can also augment an interacted-with object’s box.Source: §6.2 · pp. 6–7, Figure 5 · p. 7
6.3. Quantitative and Qualitative Results
- Across PiGraphs, Watch-n-Patch, and SUN RGB-D, the method improves scene reconstruction and reports consistently strong pose-estimation generalization.The reported reconstruction improvement requires no dataset-specific training; pose comparisons include VNect and a baseline trained on synthetic SUN RGB-D.Source: Table 1 · p. 7, §6.3 · p. 7, Table 2 · p. 7
- Qualitative examples across the three datasets illustrate improved initialization with spatial relations and physical plausibility.The PiGraphs comparison contrasts the full model with the version without physical commonsense.Source: Figure 6 · p. 8, Figure 7 · p. 8
- Without HOI reasoning, scene reconstruction and pose estimation are hindered; the authors attribute gains to inferring agent–object spatial relationships.The HOI ablation evaluates 3D object IoU, 3D pose error, and object miss-detection rate on PiGraphs and synthetic SUN RGB-D.Source: §6.3 · pp. 7–8, Table 3 · p. 7
- 23.5% to 24.9% 3D object detection IoU and 0.223m to 0.150m physical violation: full model versus w/o phy. on PiGraphs.The physical-violation measure is the distance between an object’s lower surface and its supporting object’s upper surface; the full model yields physically plausible results where w/o phy. may float or penetrate objects.Source: §6.3 · p. 8, Figure 7 · p. 8
7. Conclusion
The paper formulates holistic++ understanding as joint single-image 3D scene reconstruction and human pose estimation. It combines HOI and physical commonsense in an MCMC-based joint inference algorithm, and reports efficacy across three datasets.
7. Conclusion
- The method jointly estimates 3D scene structure and human pose from one RGB image, using HOI and physical commonsense within MCMC inference over scene configurations.Experiments on PiGraphs, Watch-n-Patch, and Synthetic SUN RGB-D demonstrate the efficacy of the approach and its general priors.Source: §7 · p. 8