PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-centric Indoor Scene Synthesis Using Stochastic Grammar

Siyuan Qi, Yixin Zhu, Siyuan Huang, Chenfanfu Jiang, Song-Chun Zhu

arXiv:1808.08473v1cs.CV

TL;DR

Indoor-scene synthesis needs scalable data with reliable pixel-level ground truth, while realistic layouts must capture object relations and human context. The paper samples scenes from an attributed spatial And-Or graph combining grammar structure with MRF-based contextual relations, producing varied realistic scenes and improving downstream scene-understanding performance when synthetic data supplements natural images.

  • Problem

    The paper addresses limited large-scale indoor-scene training data with reliable ground truth because sensors are noisy and some labels are impossible or costly to obtain.

  • Method

    It automatically generates 3D indoor scenes and renders 2D images with pixel-wise ground truth using a human-centric attributed S-AOG with MRF-based contextual relations.

  • Results

    The framework robustly samples varied realistic scenes across visual realism, affordance-map accuracy, and human-evaluated functionality and naturalness criteria.

  • Takeaways & Limitations

    Synthetic data from the pipeline improves state-of-the-art depth and surface-normal estimation when combined with natural images.

  • Takeaways & Limitations

    The authors identify integrating a physics engine as future work for synthesizing physically plausible scenes.

Abstract

from arXiv · show

We present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, to obtain large-scale 2D/3D image data with perfect per-pixel ground truth. An attributed spatial And-Or graph (S-AOG) is proposed to represent indoor scenes. The S-AOG is a probabilistic grammar model, in which the terminal nodes are object entities. Human contexts as contextual relations are encoded by Markov Random Fields (MRF) on the terminal nodes. We learn the distributions from an indoor scene dataset and sample new layouts using Monte Carlo Markov Chain. Experiments demonstrate that our method can robustly sample a large variety of realistic room layouts based on three criteria: (i) visual realism comparing to a state-of-the-art room arrangement method, (ii) accuracy of the affordance maps with respect to groundtruth, and (ii) the functionality and naturalness of synthesized rooms evaluated by human subjects. The code is available at https://github.com/SiyuanQi/human-centric-scene-synthesis.

1 UCLA Center for Vision, Cognition, Learning and Autonomy 2 UPenn Computer Graphics Group

The paper addresses limitations in indoor-scene data collection and scene synthesis by generating large-scale scenes with pixel-wise ground truth through a human-centric stochastic grammar. Its framework models objects, affordances, and activities jointly while supporting varied, realistic layouts.

  • Motivation: Noisy sensor ground truth, impossible labels, and costly manual annotation motivate automatically generated large-scale data with perfect per-pixel supervision.The paper specifically targets depth, surface normal, segmentation, and related ground-truth information.
  • Motivation: Indoor-scene synthesis is difficult because functional groups vary, object relations grow quadratically, many relations are irrelevant, and looping constraints hinder sampling.These difficulties make direct constraint-based layout optimization complex.
  • Method: The proposed human-centric approach combines functional grouping, supporting relations, object affordances, and dynamic activity trajectories to constrain scene layouts.Affordance distributions relate objects to humans, while planned furniture-to-furniture trajectories model activity context.
  • Method: An attributed spatial And-Or graph represents hierarchical scene structure and contextual relations, enabling alternative scene configurations and sampling without fixing the number of objects.The representation combines probabilistic grammar structure with contextual relations, while prior work commonly rearranges a fixed object set.
  • Evaluation: The paper evaluates visual similarity, affordance-map accuracy, and human-rated functionality and naturalness, reporting robust sampling across varied realistic scenes.These criteria correspond to comparative realism evaluation, synthesized affordance measurement, and an ablation study of functionality and naturalness.
  • Method: The framework forwards samples from stochastic grammar models rather than using them only for parsing, generating large variations of indoor scenes.Scene sampling selects parse-tree structures according to the grammar's prior distribution.

2. Representation of Indoor Scenes

Indoor scenes are represented with an attributed spatial And-Or graph that combines hierarchical grammar structure with horizontal contextual relations. Terminal nodes encode scene entities and their attributes, while address nodes sparsify context-dependent grouping and support relations.

  • S-AOG structure: The attributed S-AOG combines a probabilistic context-free grammar for hierarchical decomposition with an MRF for spatial and functional relations.The grammar runs from scenes to objects, while contextual relations are represented by horizontal links.
  • S-AOG structure: An S-AOG is formalized as G = ⟨S, V, R, P, E⟩, comprising the scene root, vertices, production rules, probability model, and contextual relations.Contextual relations E connect nodes in the same layer.
  • Node types: Non-terminal And-, Or-, and Set-nodes decompose entities, select alternative scene configurations, and determine collections of child nodes.These node types support hierarchical construction and reconfiguration of scenes.
  • Terminal attributes: Regular terminal nodes represent spatial entities with size, position, orientation, and sampled human-position attributes.Internal attributes encode dimensions, while external attributes encode location and orientation.
  • Address nodes: Address terminal nodes point to regular terminals, encoding context-dependent supporting or grouping relations while allowing null links.A null address means furniture is ungrouped or an object is placed on the floor.
  • Contextual relations: Contextual relations form terminal-layer MRF cliques covering furniture, supported-object, functional-pair, and room-furniture interactions.A parse graph combines a hierarchical parse tree with contextual relations, whose projections produce the terminal-layer cliques.

3. Probabilistic Formulation of S-AOG

The S-AOG assigns a Gibbs prior to scene parse graphs, combining hierarchical grammar structure with contextual potentials over terminal-node relations. These potentials encode furniture collisions, human usability, activity trajectories, grouping, support, and room-wall relationships.

  • Probabilistic formulation: A scene parse graph combines a parse tree with contextual relations, and its prior probability is modeled as a Gibbs distribution.The energy separates parse-tree terms from contextual-relation terms.
  • Parse-tree energy: The S-AOG decomposes parse-tree energy across non-terminal choices, terminal attributes, and multinomial branch distributions.And-nodes expand deterministically, while terminal sizes follow a kernel-density-estimated distribution.
  • Human-centric potentials: Furniture potentials penalize overlapping volume and reward room utility through the entropy of human trajectories planned between furniture.Bi-directional RRT produces the trajectory heatmap used to compute entropy.
  • Human-centric potentials: Supported-object potentials measure human usability by sampling positions from object affordance maps and evaluating access to both object and furniture.These relations connect supported objects, address nodes, and supporting furniture.
  • Scene-context potentials: Additional potentials model functional grouping between furniture and distance and relative orientation between furniture and the nearest wall.Wall distance uses a log-normal model, while relative orientation uses a von Mises distribution.

4. Learning S-AOG

The model learns S-AOG parameters from SUNCG statistics using maximum likelihood, combining closed-form or non-parametric distributions with contrastive-divergence learning for contextual weights. Learned affordance maps represent distributions of possible human positions around furniture and supported objects.

  • Training data: SUNCG provides over 45K realistic scenes from which the method learns room, furniture, relation, affordance, and grouping statistics.Parameters are learned in a supervised way by maximum likelihood estimation.
  • Weight learning: Contrastive divergence avoids waiting for Markov chains to reach equilibrium by using a chain initialized at the data distribution and run for one step.The method uses generated examples from the current Markov chain to learn the weight vector.
  • Distribution learning: Branch probabilities for Or-nodes, Set-nodes, and address terminals are estimated from the frequency of each alternative.Grouping relations are hand-defined, with occurrence probabilities learned multinomially.
  • Distribution learning: Room and object sizes are fitted with non-parametric kernel density estimates, while furniture-wall distances and orientations use log-normal and von Mises mixtures.These distributions capture geometric variation in the training scenes.
  • Affordance learning: Affordance maps are learned as non-parametric distributions by accumulating relative human positions, including annotated humans and assumed frequently visited centers of chairs, sofas, and beds.The resulting maps capture possible human positions around furniture and supported objects.

5. Synthesizing Scene Configurations

Scene synthesis samples parse-tree structure and internal attributes directly, then uses MCMC proposal moves to sample externally constrained positions and orientations. Simulated annealing produces high-probability configurations represented through top views, side views, and affordance heatmaps.

  • Sampling procedure: The synthesis procedure samples a parse graph from the S-AOG prior, directly sampling grammar structure and object sizes from learned distributions.External object attributes remain constrained by multiple potential functions.
  • Sampling procedure: MCMC samples address nodes and external object attributes through proposal moves after the parse-tree structure and internal attributes are initialized.A final sample is selected after the Markov chain converges.
  • Proposal dynamics: The Markov chain proposes object translations with bivariate-normal displacement and rotations with normally distributed orientation changes.The two dynamics are selected randomly with probabilities q_i.
  • Acceptance and annealing: Metropolis-Hastings accepts proposed parse graphs, while simulated annealing is used to obtain high-probability samples.The proposal-rate terms cancel because the proposal moves are symmetric.
  • Visualization: Synthesized examples are presented in ten room categories using top views, side views, and affordance heatmaps.The three views expose geometry and human-centric spatial structure together.

6. Experiments

The experiments evaluate synthesized indoor scenes for visual realism, affordance-map similarity, functionality, and naturalness. The method produces varied layouts that compare favorably with the selected arrangement baseline.

  • Experimental criteria: The evaluation covers visual similarity to manually constructed scenes, affordance-map accuracy, and human-rated functionality and naturalness.These criteria are assessed through classifier-based layout evaluation, affordance-map distances, and human-subject ratings.
  • Layout Classification: The proposed method is evaluated using top-view layout segmentation maps to exclude rendering factors such as object materials.A ResNet-152 classifies synthesized and SUNCG segmentation maps, with Yu et al. and perturbed SUNCG scenes as comparison methods.
  • Layout Classification: 87.49% classification accuracy distinguishes Yu et al. from SUNCG, whereas the proposed method achieves 76.18%.Lower distinguishability from SUNCG indicates higher visual similarity and larger variety under this evaluation.
  • Affordance Maps Comparison: Most sampled scene categories show affordance distributions similar to SUNCG, while Storage exhibits larger distances because its affordances are less clear.The comparison uses total variation and Hellinger distances over 500 rooms from 10 scene categories.
  • Human evaluation: Qualitative comparisons and human-subject ratings assess whether sampled layouts exhibit functionality and naturalness across room categories.Four subjects rated 24 samples per category without knowing the generation method.

7. Conclusion

The paper concludes that sampling from a spatial And-Or graph provides a general framework for human-centric indoor scene synthesis. Future work identifies integrating a physics engine to synthesize physically plausible scenes.

  • Conclusion: The proposed framework samples human-centric indoor scenes from a spatial And-Or graph.The conclusion presents this as the paper’s general framework for indoor scene synthesis.
  • Conclusion: Experimental results demonstrate the approach across a large variety of scenes using different criteria.The conclusion summarizes the experimental evidence without specifying individual metrics.
  • Future work: Integrating a physics engine is identified as future work for synthesizing physically plausible scenes.The conclusion also expresses hope that the synthesized data will contribute to the broader AI community.

Supplementary Material for Human-centric Indoor Scene Synthesis Using Stochastic Grammar

The supplementary material identifies the paper’s authors and their institutional affiliations. The authors are affiliated with UCLA and UPenn computer graphics groups.

  • Authors: The paper lists Siyuan Qi, Yixin Zhu, Siyuan Huang, Chenfanfu Jiang, and Song-Chun Zhu as authors.
  • Affiliations: The affiliations include the UCLA Center for Vision, Cognition, Learning and Autonomy and the UPenn Computer Graphics Group.

8. Simulated Annealing

Scene sampling uses simulated annealing with a 20,000-iteration schedule that takes about 20 minutes per interior layout. The stated temperature condition guarantees global-minimum convergence with probability one.

  • Sampling schedule: The sampling procedure uses simulated annealing because its schedule is important for synthesizing realistic scenes.
  • Sampling schedule: 20,000 sampling iterations require around 20 minutes to sample one interior layout.
  • Convergence condition: The temperature T(t) is scheduled under the condition T(t) ≥ T0 ln(1+t), which ensures convergence to the global minimum with probability one.The passage attributes this necessary and sufficient convergence condition to Geman et al.

9. Data Effectiveness

The synthesized data improves training for single-image depth and surface normal estimation, with evaluations using established error and accuracy metrics. Models pretrained on the synthetic data and fine-tuned on NYU-Depth V2 achieve the strongest reported performance, while noisy normal ground truth remains an error source.

  • Scene understanding tasks: Synthetic data is evaluated for improving single-image depth estimation and surface normal estimation from RGB images.The depth task is challenging because reliable depth cues are unavailable; surface normals support recovery of 3D scene structure.
  • Depth estimation: Depth estimation is compared under training protocols analogous to those used for normal prediction, using established quantitative metrics.The reported depth metrics include absolute relative error, square relative difference, average log10 error, and threshold-based accuracy.
  • Depth estimation: Lower values are better for the error metrics, whereas higher threshold values indicate better depth-estimation accuracy.The threshold criteria measure the fraction of pixels whose predicted and ground-truth depths satisfy the specified threshold.
  • Training protocols: The model pretrained on the synthetic dataset and fine-tuned on NYU-Depth V2 achieves the best performance in both error and accuracy.Table 4 summarizes depth estimation under different training protocols, and the paper reports the same protocol comparison for surface normal estimation in Table 5.
  • Surface normal estimation: Synthetic data improves surface normal estimation, although errors mainly accumulate where the ground-truth normal map is noisy.The authors attribute this noise partly to sensor noise or sensing-distance limits and connect the result to the value of perfect per-pixel ground truth.
Loading 1808.08473v1…