Source-linked AI summary
Deep Generative Modeling for Scene Synthesis via Hybrid Representations
Zaiwei Zhang, Zhenpei Yang, Chongyang Ma, Linjie Luo, Alexander Huth, Etienne Vouga, Qixing Huang
TL;DR
The paper addresses learning generative models for heterogeneous 3D scenes containing varied object categories, shapes, poses, and repetition counts. It proposes a feed-forward generator with hybrid 3D arrangement and projected 2D image representations, and reports qualitative advantages across scene synthesis, interpolation, and completion applications.
Problem
The central challenge is learning effective parametric models for heterogeneous 3D scenes with large variation in object categories, shapes, poses, and repetition counts.
Method
A feed-forward generative model maps latent parameters to indoor scenes using a matrix-parameterized 3D object arrangement and a projected 2D image representation with corresponding discriminators.
Results
The hybrid representation combines coarse object interactions and local image-based interactions, while qualitative evaluations show advantages for 3D scene synthesis.
Takeaways & Limitations
The method supports indoor scene generation with repeated object categories and provides applications in scene interpolation and scene completion.
Takeaways & Limitations
The approach requires training data consisting of semantically segmented 3D scenes, which may be unavailable for scenes reconstructed from point clouds.
Abstract
from arXiv · showhide
We present a deep generative scene modeling technique for indoor environments. Our goal is to train a generative model using a feed-forward neural network that maps a prior distribution (e.g., a normal distribution) to the distribution of primary objects in indoor scenes. We introduce a 3D object arrangement representation that models the locations and orientations of objects, based on their size and shape attributes. Moreover, our scene representation is applicable for 3D objects with different multiplicities (repetition counts), selected from a database. We show a principled way to train this model by combining discriminator losses for both a 3D object arrangement representation and a 2D image-based representation. We demonstrate the effectiveness of our scene representation and the deep learning method on benchmark datasets. We also show the applications of this generative model in scene interpolation and scene completion.
1 INTRODUCTION
The paper models indoor scenes with a feed-forward generator over object arrangements, combining coarse 3D structure with local image-based constraints. The approach supports heterogeneous object categories, shapes, poses, and repetition counts, and is evaluated on benchmark scenes and applications.
- The paper addresses learning parametric 3D scene models from heterogeneous objects with varying categories, shapes, poses, and repetition counts.
- The proposed feed-forward network maps a low-dimensional latent vector to a 3D indoor scene represented as an arrangement of objects.
- The representation selects objects from a superset, encodes their shape, location, size, and orientation, and permits multiple objects from the same category.
- Hybrid training combines arrangement autoencoder and discriminator losses with an image-based discriminator to capture coarse and local scene interactions.
- The method is evaluated on SUNCG living rooms and bedrooms and applied to scene interpolation, extrapolation, and completion.
2 RELATED WORKS
Prior work spans hand-crafted, data-driven, and neural 3D generative models using diverse geometric representations and assembly procedures. This paper distinguishes itself by combining object-arrangement and image-based losses for globally meaningful and locally compatible scenes.
- Hand-crafted parametric models are difficult to create for object classes with substantial geometric or topological variability.
- Existing 3D generation research uses volumetric, point-based, multi-view, part-based, graph, mesh, and spherical representations, with volumetric methods facing limited scene resolution.
- The paper combines object-arrangement and image-based training losses to obtain globally meaningful and locally compatible synthesis results.
- Assembly-based methods construct models from existing parts, while recursive variants insert parts sequentially and may require hierarchical labels.
- Unlike related image-based approaches that predict object attributes, this work learns a parametric 3D generator for scene synthesis.
3 PROBLEM STATEMENT AND APPROACH OVERVIEW
The problem is to generate non-interpenetrating indoor scenes whose semantic object arrangements and local geometric relationships are both captured. The proposed solution combines a vectorized 3D arrangement model with a top-view image discriminator and jointly optimized training variables.
- Scenes are modeled as rigid objects from predefined classes, with shape descriptors, top-view positions and orientations, nonuniform scaling, and ground-related assumptions.
- Scene synthesis must represent semantically meaningful object placement while also capturing geometric details such as potential object intersections.
- The generator combines a variational autoencoder loss, a scene discriminator over vectorized arrangements, and an image discriminator over top-view projections.
- The 3D arrangement is encoded as a matrix whose columns describe object presence, location, size, orientation, and shape.
- Training jointly optimizes the generator, discriminators, scene poses, object orderings, and latent scene configurations after alignment.
4 APPROACH
The approach encodes indoor scenes as fixed-size object arrangements, uses sparse/fully connected networks for arrangement generation, and adds an image-based discriminator to capture local geometric compatibility. Global scene alignment is refined through joint optimization of orientations, translations, and permutations.
- 4.1 Scene Representation: Scenes are represented by selecting category-specific objects and encoding each object's presence, shape, position, orientation, and bounding-box size in a matrix.The representation supports bounded multiplicities for each of nc object categories and uses explicit class labels.
- 4.1 Scene Representation: Permutation and rigid-motion operators factor out object ordering and global scene pose so the matrix encoding can be made consistent across scenes.The method associates latent permutations, rotations, and translations with the scene encoding and enforces consistency during optimization.
- 4.2 3D Object Arrangement Module: Fully connected generators overfit, so the arrangement module interleaves sparsely connected and fully connected layers to reduce model size while retaining expressiveness.Each sparse-layer node connects to h previous-layer nodes; the implementation uses h = 4 with fixed randomized connections.
- 4.4 Joint Scene Alignment: Joint scene alignment synchronizes orientations, translations, and category-wise permutations, substantially improving alignments after noisy pairwise matching.The procedure applies rotation synchronization, translation synchronization, and normalized spectral permutation synchronization sequentially.
5 EXPERIMENTAL EVALUATION
The evaluation shows that the hybrid generator produces diverse, realistic indoor scenes, learns object-location and pairwise relationships, and improves scene completion over sequential baselines. Ablations indicate that image-based discrimination and global scene alignment are important for capturing local compatibility and scene-wide correlations.
- 5.2 Random Scene Synthesis: The generator produces diverse scenes with varied object counts, spatial layouts, and correlated object groups rather than merely memorizing training examples.Generated scenes exhibit noticeable variations from their closest training scenes in spatial layout and object existence.
- 5.3 Perceptual Study: 60%/40% and 56%/44% user-study results were achieved on Bedroom and Living Room, respectively, with the full approach significantly outperforming both baselines.Users favored the synthesized scenes over user-designed scenes more than 30% of the time.
- 5.4 What are Learned: The learned distributions for absolute object locations are fairly close to training data, although Plant generalizes less well because it has fewer training instances.Window distributions are especially difficult to distinguish between synthesized and training scenes.
- 5.4 What are Learned: The generator matches pairwise relative-location distributions, with particularly similar results for Desk/Chair and Bed/Nightstand relationships.Relative front orientations are also evaluated by quantizing angles from 0 to 2π into four bins.
- 5.5 The Importance of Joint Scene Alignment: Removing global scene alignment produces significantly different relative-location distributions from training data, showing that alignment is crucial for learning salient correlations.The study evaluates Chair/Table and Bed/Nightstand locations and orientations on Bedroom scenes.
- 5.7 Applications in Scene Completion: For scene completion, the approach produces more semantically meaningful object groups and local pair compatibility while solving in 1–2 seconds versus 83.1 and 76.4 seconds for the two baselines.The method optimizes the scene layout with respect to all patterns captured by the neural network, whereas the baselines generate scenes sequentially.
6 CONCLUSIONS
The paper concludes that hybrid 3D object-arrangement and projected 2D image representations support feed-forward indoor scene generation, interpolation, and completion. It also identifies missing physical-property encoding and dependence on semantically segmented training scenes as limitations.
- 6 CONCLUSIONS: Combining 3D object-arrangement and projected 2D image representations captures neighborhood structure and local view-dependent patterns in synthesized scenes.The 3D representation supports local and global neighborhood structure, while the image representation preserves local view-dependent patterns and benefits generator training.
- 6 CONCLUSIONS: The feed-forward generator jointly optimizes synthesis factors, unlike recurrent approaches that can struggle to recover from sequential mistakes.The authors report preliminary qualitative evidence favoring the feed-forward architecture over two recurrent approaches, while noting that dominance over recurrent methods remains premature.
- 6 CONCLUSIONS: The model is applied to scene interpolation and completion, with completion results compared against methods by Fisher et al. and Kermani et al.The cited figures show interpolation between source and target scenes and completion outputs alongside prior methods.
- 6 CONCLUSIONS: The approach does not completely encode physical properties needed for applications such as 3D printing.The paper suggests representations such as a shape grammar as a possible way to encode these properties explicitly.
- 6 CONCLUSIONS: Training data must consist of semantically segmented 3D scenes, which may exclude reconstructed scenes that are not segmented into individual objects.The authors propose enforcing consistency among image-based synthesis, object-arrangement synthesis, and a scene-to-arrangement network as a possible remedy.
A ADDITIONAL DETAILS ON PAIR-WISE SCENE ALIGNMENT
The alignment optimization alternates between solving for transformation and assignment variables and updating column weights. The implementation uses four iterations for both alternating minimization and reweighted least squares.
- A ADDITIONAL DETAILS ON PAIR-WISE SCENE ALIGNMENT: The procedure alternates between fixing weights to solve the optimization and updating the weights from the resulting solution.The initial weight vector is set to w(0) = 1.
- A ADDITIONAL DETAILS ON PAIR-WISE SCENE ALIGNMENT: At each inner iteration, alternating minimization updates T and S, whose subproblems have closed-form or linear-assignment solutions.The optimal solution for one subproblem is computed using Horn (1987), while the other is obtained by solving a linear assignment.
- A ADDITIONAL DETAILS ON PAIR-WISE SCENE ALIGNMENT: The implementation applies four iterations of alternating minimization and four iterations of reweighted least squares.The alternating procedure is described as converging fairly fast.
- A ADDITIONAL DETAILS ON PAIR-WISE SCENE ALIGNMENT: Table 1 lists the object classes and instance counts for the Bedroom dataset.
B GRADIENT OF THE IMAGE PROJECTION
The image-projection gradient is derived for signed-distance values associated with an oriented box. The derivation expresses derivatives with respect to the line’s center, orientation, and size parameters.
- B GRADIENT OF THE IMAGE PROJECTION: The projection uses signed-distance-function values with respect to an oriented box, enabling gradients of image pixels through geometric parameters.
- B GRADIENT OF THE IMAGE PROJECTION: For a line parameterized by orientation n and point q = o + sn, o denotes the box center, n the perpendicular axis, and s the size along that axis.
- B GRADIENT OF THE IMAGE PROJECTION: The derivation gives derivatives of d(p,l) with respect to o, n, and s, including a perpendicular vector n⊥.
C STATISTICS ON SUNCG
This section points to dataset statistics tables for the Bedroom and Living Room categories, identifying the classes and instance counts represented in each dataset.
- C STATISTICS ON SUNCG: Table 1 and Table 2 collect statistics for the Bedroom and Living Room datasets.
- C STATISTICS ON SUNCG: Table 2 lists the object classes and number of instances in the Living Room dataset.
- C STATISTICS ON SUNCG: The document records received, revised, final-version, and acceptance dates as February 2007 through July 2009.