End-to-End Optimization of Scene Layout

Andrew Luo; Zhoutong Zhang; Jiajun Wu; Joshua B. Tenenbaum

Source-linked AI summary

End-to-End Optimization of Scene Layout

Andrew Luo, Zhoutong Zhang, Jiajun Wu, Joshua B. Tenenbaum

arXiv:2007.11744v1Jul 23, 2020cs.CV

TL;DR

Unconditional layout models generate diverse interiors but provide limited fine-grained control over user-specified requirements and modifications. 3D-SLN combines a variational autoencoder with a graph convolutional network to generate diverse, plausible 3D layouts conditioned on scene-graph relationships. Experiments demonstrate more accurate and diverse 3D scene layouts than baselines, with the full model achieving the highest scene-graph accuracy and diversity in the ablation study.

Problem
Unconditional layout models generate diverse interiors but provide limited fine-grained control over user-specified requirements and modifications.
Method
3D-SLN combines a variational autoencoder with a graph convolutional network to generate diverse, plausible 3D layouts conditioned on scene-graph relationships.
Results
Experiments demonstrate more accurate and diverse 3D scene layouts than baselines, with the full model achieving the highest scene-graph accuracy and diversity in the ablation study.
Takeaways & Limitations
Scene graphs enable flexible, controllable conditional synthesis and support exemplar-based layout generation from text or reference images.
Takeaways & Limitations
Scene synthesis performance decreases when scene graphs contain too many objects that might overlap.

Abstract

from arXiv · show

We propose an end-to-end variational generative model for scene layout synthesis conditioned on scene graphs. Unlike unconditional scene layout generation, we use scene graphs as an abstract but general representation to guide the synthesis of diverse scene layouts that satisfy relationships included in the scene graph. This gives rise to more flexible control over the synthesis process, allowing various forms of inputs such as scene layouts extracted from sentences or inferred from a single color image. Using our conditional layout synthesizer, we can generate various layouts that share the same structure of the input example. In addition to this conditional generation design, we also integrate a differentiable rendering module that enables layout refinement using only 2D projections of the scene. Given a depth and a semantics map, the differentiable rendering module enables optimizing over the synthesized layout to fit the given input in an analysis-by-synthesis fashion. Experiments suggest that our model achieves higher accuracy and diversity in conditional scene synthesis and allows exemplar-based scene generation from various input forms.

1. Introduction

The paper introduces 3D-SLN, a stochastic scene-layout synthesizer conditioned on 3D scene graphs, enabling controllable and diverse layouts across input modalities. It further supports differentiable refinement from projected scene information and exemplar-based generation.

1. Introduction: Unlike unconditional layout models that generate diverse scenes but lack user control, conditional synthesis accepts requirements from scene graphs and other inputs such as text or partial layouts.This addresses the need for controllable interior layouts used in simulated navigation, home automation, and interior design.
1. Introduction: Scene graphs provide fine-grained control over synthesis by encoding object identities, attributes, and 3D spatial relationships that users can directly manipulate.The representation also serves as an intermediate form between text descriptions and exemplar images.
1. Introduction: The framework refines generated layouts with a differentiable renderer so they match given depth and semantic information from a single scene example.This enables analysis-by-synthesis optimization using projected scene evidence.
1. Introduction: The same scene graph supports exemplar-based generation, including layouts extracted from text or inferred from reference images and scene-graph-based image synthesis.Multiple layouts can share the input example's structure while varying in arrangement.

3. Methods

3D-SLN is a graph-conditioned variational model that samples diverse 3D layouts from scene graphs, then refines them against target depth and semantic projections through differentiable rendering.

3.1. Scene Layout Generator: 3D-SLN combines a graph convolutional network with a conditional variational autoencoder to generate diverse 3D scene layouts from scene graphs.The scene graph encodes object identities, attributes, and spatial relationships, while sampling from the prior produces multiple layouts.
3.1. Scene Layout Generator: Each generated object is represented by a 3D bounding box and rotation around the vertical Z axis, extending scene-graph layout generation beyond 2D boxes.The graph uses floor-plane X and Y coordinates and an up-direction Z to express spatial relations such as ‘left of’ and ‘on’.
3.1. Scene Layout Generator: During training, an encoder maps a scene graph and exemplar layout to a posterior latent distribution, while a decoder reconstructs layouts using a KL divergence and layout loss.The layout loss combines L1 bounding-box regression with 24-bin negative log-likelihood for object rotations.
3.1. Scene Layout Generator: At test time, sampling Gaussian latent vectors conditioned on a 3D scene graph enables the decoder to generate multiple possible layouts.The same graph can therefore produce varied layouts while preserving the encoded scene structure.
3.2. Gradient-Based Layout Refinement: The refinement module retrieves object meshes, renders depth and semantics, and updates latent vectors and decoder weights so the generated layout matches target projections.It uses differentiable rendering, softmax-based rotation approximation, class-wise depth losses, and a size-change penalty to stabilize optimization.
3.2. Gradient-Based Layout Refinement: Class-conditioned depth losses isolate gradients by semantic class, helping prevent spurious optima during layout refinement.The method retains depths within each class, replaces other values with the class mean, and sums class-wise depth losses.

4. Experiments

Experiments evaluate 3D-SLN for accurate, diverse scene-graph-conditioned layout synthesis and differentiable refinement from 2.5D targets. The model achieves strong scene-graph alignment and diversity, supports latent-space variation and exemplar-based refinement, but performance decreases for heavily overlapping scenes.

Scene Layout Synthesis: The evaluation measures scene graph accuracy, bounding-box L1 loss, and standard deviations of object size, position, and rotation across generated layouts.L1 loss is reported for reference because generating multiple plausible layouts makes it less meaningful; diversity statistics require different aggregation for autoregressive DeepSynth.
Scene Layout Synthesis: 3D-SLN achieves the highest scene graph accuracy and diversity, while Perturbed Layout has lower L1 loss but substantially worse scene graph accuracy.DeepSynth has higher position variability but lower rotation variability and lacks fine-grained synthesis control.
Ablation Study: The full 3D-SLN model outperforms deterministic GCN and noisy GCN+noise variants in scene graph accuracy and diversity.The model combines a graph convolution network with a variational autoencoder to generate multiple plausible layouts from one scene graph.
End-to-End Layout Refinement: For 2.5D refinement, optimization uses 3D IoU, depth-map MSE, and semantic-map cross-entropy to compare proposed layouts with target projections.Experiments optimize over 150 scene graphs, using six attempts per graph to reduce failures from wall or object occlusion.
End-to-End Layout Refinement: After refinement, layouts fit target arrangements: beds, lamps, desks, and bed-sofa distances move toward the target configuration.Figure 4 compares initial, target, and optimized semantic and depth maps for exemplar-based fine-tuning.
End-to-End Layout Refinement: Latent interpolation produces smoothly changing object layouts for the same scene graph, while multiple views and conditional image synthesis demonstrate varied 3D scene outputs.The conditional image comparison reports better results than Johnson et al. [11], whose method does not incorporate 3D information.
End-to-End Layout Refinement: Scene synthesis performance decreases when scene graphs contain too many objects that may overlap.Suggested future remedies include adversarial training, physical-simulation rejection, or collision detection.

5. Applications

The model supports scene graph–based image synthesis, sentence-based layout synthesis, and exemplar-based layout synthesis. Across these applications, it produces sharper images than a baseline, more realistic images than AttnGAN, and varied layouts from example images.

The model enables scene graph–based image synthesis, sentence-based scene layout synthesis, and exemplar-based scene layout synthesis.
5.1. Scene Graph–Based Image Synthesis: The generated images are sharp and photo-realistic with complex lighting, whereas baseline [11] produces blurry images that may obscure objects and 3D structure.
5.2. Sentence-Based Scene Layout Synthesis: 3D-SLN generates more realistic images than AttnGAN [33] for descriptions constrained to at most five sentences.
5.3. Exemplar-Based Scene Layout Synthesis: From a bedroom example image, the model recovers the original layout and generates new layouts with different bed and nightstand locations and rotations.

6. Conclusion

The paper concludes that stochastic scene layout synthesis conditioned on scene graphs provides flexible, controllable generation with accurate and diverse 3D layouts. A differentiable renderer further supports refinement from a single example and downstream layout and image synthesis applications.

Experiments show that the stochastic scene-graph-conditioned model generates more accurate and diverse 3D scene layouts than baselines.
A differentiable renderer refines 3D layouts conditioned on a single example, while the framework supports downstream scene layout and image synthesis tasks.

1. Relational definitions

The 3D scene graph defines relationships between objects using the room floor plane and vertical axis. Table 1 lists these relationships and specifies an angular quantity for object pairs.

The scene graph uses X and Y for the ground plane and Z as the vertical axis when defining object relationships.
Table 1 lists the model’s object relationships and defines θ_i,j as atan2(Y_i − Y_j, X_i − X_j).

2. Attribute definitions

Object attributes in the 3D scene graph are defined in Table 2. The table formalizes object volume and height from their 3D bounding-box coordinates.

Table 2 lists the attributes that can be assigned to each object.
Object volume is V_i = (maxX_i − minX_i) × (maxY_i − minY_i) × (maxZ_i − minZ_i), and height is H_i = maxZ_i − minZ_i.

3. Differentiable fine tuning: perspective and top down views

Differentiable fine tuning compares synthesized layouts before and after optimization against a target scene. Visualizations show changes in object positions, grouping, and scale across perspective and top-down views.

3. Differentiable fine tuning: perspective and top down views: The visualizations compare each scene before fine tuning, against a target, and after fine tuning from top-down views.The rendered examples are intended to show layout refinement with a differentiable renderer.
3. Differentiable fine tuning: perspective and top down views: Fine tuning shifts beds, cabinets, desks, and dressers, while also changing cabinet size to better match the target arrangement.Examples include moving cabinets toward a corner, bringing a bed forward or right, grouping a desk and dresser, and enlarging a cabinet.
3. Differentiable fine tuning: perspective and top down views: The figure presents semantic maps from both an inside-room camera and a top-down viewpoint.The first three columns use the inside-room camera, while the latter three use top-down semantic maps.

4. Object distributions heatmaps for different scene graphs, top down views

The paper visualizes layout diversity and object-centroid distributions for scene graphs with different structures. These heatmaps use top-down views to show how scene complexity affects placement concentration.

4. Object distributions heatmaps for different scene graphs, top down views: The heatmap visualization demonstrates diversity among generated layouts conditioned on an input 3D scene graph.It is presented as a visual analysis of conditional layout generation.
4. Object distributions heatmaps for different scene graphs, top down views: For each scene graph, 20,000 latent vectors sampled from the training-distribution μ and σ produce object-centroid density maps.The densities are visualized from a top-down view.
4. Object distributions heatmaps for different scene graphs, top down views: Scene graphs containing more objects produce more concentrated object placements in the centroid visualizations.The comparison is made across different scene-graph structures.

5. Example scenes from scene graph: perspective and top down views

The paper provides example scenes synthesized from scene graphs using paired perspective and top-down visualizations. Each perspective view is aligned with a top-down scene representation for inspection.

5. Example scenes from scene graph: perspective and top down views: The top-down scenes correspond to the scene-graph-based image-synthesis experiment described in Section 5.1.These views provide the spatial counterparts of that experiment’s synthesized scenes.
5. Example scenes from scene graph: perspective and top down views: The examples show perspective and top-down views of synthesized scenes generated from scene graphs.The perspective image and its corresponding top-down visualization are presented together for each scene block.
5. Example scenes from scene graph: perspective and top down views: Each perspective view is captured from the bottom edge of the top-down layout while looking toward the room’s center.This camera placement links the two displayed viewpoints.