Human-centric Indoor Scene Synthesis Using Stochastic Grammar

1. Introduction

• The method automatically generates large-scale 3D indoor scenes and renders 2D images with pixel-wise ground truth for surface normals, depth, and segmentation.This targets limitations of sensor-derived ground truth, unavailable labels such as 3D object sizes in 2D images, and laborious manual annotation.Source: §1 · p. 1
• The approach addresses variable group sizes, quadratic pairwise relations, irrelevant object links, and looping constraints that make scene layouts difficult to sample and optimize.It integrates human activities and functional grouping and supporting relations to capture context while simplifying scene structure.Source: §1 · p. 1
• Its human-centric probabilistic S-AOG combines scene hierarchy with contextual relations for functional grouping and object support.The model learns object-human affordance distributions and constrains layouts using planned trajectories between furniture pieces.Source: §1 · pp. 1–2
• The paper's stated contributions are joint modeling of objects, affordances, and activity planning, a general learning and sampling framework, and comparative evaluation.It samples scene parse trees from the S-AOG according to a prior probability distribution.Source: §1 · p. 2

1.1. Related Work

• Unlike room-rearrangement methods using fixed object sets [44],,,, the proposed method can add or delete objects without fixing their number.The cited methods optimize layouts using MCMC or RJMCMC; the paper also notes examples of small-object arrangement and volumetric procedural modeling [29].Source: §1.1 · p. 2
• Prior work includes example-driven small-object arrangement, volumetric procedural modeling [29], and interior-design suggestions; Jiang's CRF cannot generate indoor scenes.The passage characterizes CRF as discriminative structured classification rather than a generative model.Source: §1.1 · p. 2
• Unlike prior stochastic grammars used to parse scenes, this work forward-samples a grammar to generate varied indoor scenes.The cited prior grammar applications include indoor and outdoor scenes and images or videos involving humans.Source: §1.1 · p. 2

2. Representation of Indoor Scenes

• The attributed S-AOG combines a scene-to-object PCFG hierarchy with terminal-node MRF links encoding spatial and functional context.The formal model is G = ⟨S, V, R, P, E⟩: scene root, vertices, production rules, probability model, and contextual relations.Source: §2 · p. 3
• Non-terminal And-nodes decompose entities, Or-nodes select alternative decompositions, and Set-nodes group nested And-Or branches that can contain different numbers of objects.Together, these node types let the grammar represent room structure and reconfigure scenes.Source: §2 · p. 3
• Regular terminal nodes encode entities with size, position, orientation, and sampled-human-position attributes; address nodes point to regular nodes to encode context-specific interactions.Address nodes also represent supporting or grouping relations.Source: §2 · p. 3
• Terminal-layer contextual cliques cover furniture, support, functional-pair, and furniture-room relations, using affordances as a bridge to characterize object-human-object relations.These relation types are Ef, Eo, Eg, and Er, respectively.Source: §2 · p. 3
• A parse graph combines a hierarchical parse tree, formed by selecting Or-node children and setting Set-node states, with contextual relations on that tree.Figure 3 illustrates a bedroom parse graph and the four contextual-clique types.Source: §2 · p. 3, Figure 3 · p. 3

3. Probabilistic Formulation of S-AOG

• The scene-configuration prior is a Gibbs distribution over parse graphs, combining parse-tree energy with energy for contextual relations.The parse graph includes scene objects and their associated attributes.Source: §3 · p. 4
• Parse-tree energy covers non-terminal choices and terminal attributes: Or- and Set-node branches are multinomial, And-nodes expand deterministically, and terminal sizes use kernel density estimation.The formulation decomposes parse-tree energy across non-terminal types and internal attributes of regular and address terminal nodes.Source: §3 · p. 4
• Furniture-relation potentials penalize overlapping furniture and use trajectory-map entropy to reward room-space utility, with trajectories planned between furniture pieces by bidirectional RRT.Figure 4 illustrates the resulting human activity probability map.Source: §3 · p. 4, Figure 4 · p. 4
• Supported-object relations use sampled human positions to measure usability for accessing both the furniture and object, while address-node probability contributes an address cost.The potential clique includes the supported object, its connected address node, and the furniture supporting it.Source: §3 · p. 4
• Functional-grouping potentials model relations between core furniture and associated furniture, using a clique formed by the core item, the associated item, and its address node.The grouping potential is defined similarly to the supporting-relation potential.Source: §3 · p. 4
• Room-furniture potentials model distance to the nearest wall with a log-normal distribution and relative orientation to that wall with a von Mises distribution.The room-furniture clique contains a furniture terminal node and the room.Source: §3 · pp. 4–5

Training data

• The model is trained on over 45K SUNCG scenes, using maximum likelihood to learn statistics of room types and sizes, furniture, spatial relations, affordances, groupings, and support.The learned probability-model parameters are denoted Θ.Source: §4 · p. 5

Weights of Loss Functions

• Potential-function weights are learned by contrastive divergence because equilibrium sampling at every gradient-ascent iteration is computationally infeasible; the Markov chain runs for one step from data.The paper sets the chain length to εn = 1.Source: §4 · p. 5

Branching Probabilities

• Or-node, Set-node, and address-terminal branching probabilities are estimated as the frequency of each alternative.Source: §4 · p. 5

Grouping Relations

• Grouping relations are hand-defined, their occurrence probabilities are learned as a multinomial distribution, and supporting relations are extracted automatically from SUNCG.Examples include associating nightstands with beds and chairs with desks or tables.Source: §4 · p. 5

Room Size and Object Sizes

• Room and object sizes use kernel-density-estimated non-parametric distributions; furniture-to-nearest-wall distances use a log-normal distribution, and relative orientations use a mixture of von Mises distributions.Source: §4 · p. 5

Affordances

• Affordance maps are non-parametric heatmaps of possible human positions, accumulated from annotated people and assumed frequent positions at the centers of chairs, sofas, and beds.The examples capture differences among desks, coffee tables, and dining tables, as well as orientation-sensitive and orientation-invariant objects.Source: §4 · p. 5, Figure 5 · p. 6

• Parse-tree structure and internal attributes are sampled directly, while MCMC samples constrained address-node values and external object attributes after the chain converges.Structure choices and object sizes are drawn from closed-form or non-parametric distributions; regular terminal nodes also receive sampled sizes and human positions.Source: §5 · p. 6
• Two proposal dynamics are selected at random: translate an object by a bivariate-normal displacement or rotate it by a normally distributed angular displacement.Each proposal chooses a regular terminal node and perturbs its current position or orientation.Source: §5 · p. 6
• Metropolis–Hastings accepts proposed parse graphs with symmetric proposal rates canceled, and simulated annealing is used to obtain high-probability samples.The new sample is selected after the Markov chain converges.Source: §5 · p. 6
• Figure 7 presents examples from ten scene categories as top views, side views, and affordance heatmaps.Source: Figure 7 · p. 7

Layout Classification

• The classifier distinguished Yu et al. from SUNCG with 87.49% accuracy, while our method achieved 76.18%, indicating greater visual similarity to real scenes than Yu et al..A ResNet-152 classified top-view segmentation maps to evaluate layout without rendering factors; each method and SUNCG contributed 1,500 scenes, split into 800 training, 200 validation, and 500 test scenes.Source: Table 1 · p. 7, §6 · pp. 7–8

Affordance Maps Comparison

• Affordance maps from most sampled room types were similar to SUNCG, though Storage showed larger distances because it lacks clear affordances.The comparison used total variation and Hellinger distances across 500 sampled rooms in 10 categories; distances near zero indicate similar distributions.Source: §6 · p. 8, Table 2 · p. 8

Functionality and naturalness

• Human subjects rated our approach higher in functionality and naturalness than random generation, object-wise modeling without human constraints, and Yu et al..Four subjects rated layouts from 10 room categories on scales of 1–5, without knowing the generation method.Source: §6 · p. 8, Table 3 · p. 8

Complexity of synthesis

• Sampling an interior layout took about 20–40 minutes for 20,000 MCMC iterations, and rendering a 640×480 image took roughly 12–20 minutes on a normal PC.Rendering time depended on illumination, environment, and scene size settings.Source: §6 · p. 8

• The experimental results demonstrate the approach’s effectiveness across a large variety of scenes and different criteria.Source: §7 · p. 8
• The proposed framework synthesizes human-centric indoor scenes by sampling from a spatial And-Or graph.Source: §7 · p. 8
• The authors identify integrating a physics engine as future work for synthesizing physically plausible scenes.Source: §7 · p. 8
• The authors hope the synthesized data will contribute to the broader AI community.Source: §7 · p. 8

• Sampling uses 20,000 iterations and takes around 20 minutes per interior layout.Source: §8 · p. 11
• Geman et al. [?] state that T(t) ≥ T0 ln(1+t) is necessary and sufficient for convergence to the global minimum with probability one.Source: §8 · p. 11

9. Data Effectiveness

• The depth-estimation experiment evaluates models trained under protocols similar to those used for normal prediction.It assesses depth from a single RGB image, a task the paper describes as challenging because reliable depth cues are unavailable.Source: §9 · p. 11
• Depth evaluation uses error metrics, for which lower values are better, and threshold accuracy criteria, for which higher values are better.The listed metrics include absolute relative error, square relative difference, average log10 error, and threshold accuracy.Source: §9 · p. 11
• The synthetic-data-pretrained model fine-tuned on NYU-Depth V2 achieves the best depth-estimation performance in both error and accuracy.The paper presents this as evidence that its dataset can improve scene-understanding performance.Source: §9 · p. 11, Table 4 · p. 12
• For surface-normal estimation, the study compares direct NYU-Depth V2 training with synthetic-data pretraining followed by NYU fine-tuning and testing.Both protocols use the fully convolutional network described by Zhang et al. [46].Source: §9 · p. 11
• Using synthetic data yields better surface-normal estimation performance than training directly on NYU-Depth V2.The reported error mainly occurs where the ground-truth normal map is noisy.Source: §9 · p. 12, Table 5 · p. 12
• The authors argue that noisy normal ground truth, potentially caused by sensor noise or sensing-distance limits, underscores the value of perfect per-pixel ground truth.The proposed pipeline generates such ground truth for training and evaluation.Source: §9 · pp. 11–12