Source-linked AI summary
Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models
Daniel Ritchie, Kai Wang, Yu-an Lin
TL;DR
Indoor scene synthesis needs models that can generate plausible object arrangements efficiently and flexibly. The paper presents a top-down, deep-convolutional pipeline that inserts objects sequentially through separate attribute predictors, supports complete and partial scenes, and outperforms prior techniques in reported evaluations.
Problem
Prior image-based scene synthesis struggled with global coordination, object-size modeling, and synthesis speed.
Method
The method uses top-down scene representations and separate modules to iteratively predict object category, location, orientation, and dimensions.
Results
The method outperforms prior techniques across generated-scene statistics, synthetic-scene classification, and forced-choice perceptual evaluation.
Takeaways & Limitations
The pipeline enables rapid synthesis of varied complete scenes and automatic completion of partial scenes.
Takeaways & Limitations
The method remains limited in generating room types with multiple strong modes of variation, such as personal versus conference offices.
Abstract
from arXiv · showhide
We present a new, fast and flexible pipeline for indoor scene synthesis that is based on deep convolutional generative models. Our method operates on a top-down image-based representation, and inserts objects iteratively into the scene by predicting their category, location, orientation and size with separate neural network modules. Our pipeline naturally supports automatic completion of partial scenes, as well as synthesis of complete scenes. Our method is significantly faster than the previous image-based method and generates result that outperforms it and other state-of-the-art deep generative scene models in terms of faithfulness to training data and perceived visual quality.
1. Introduction
Indoor scene synthesis is valuable for virtual environments and synthetic training data, but prior image-based methods struggle with global coordination, object size, and speed. The paper introduces a factorized convolutional pipeline that synthesizes scenes in under 2 seconds and performs as well or better than prior techniques.
- Virtual indoor environments support games, virtual reality, augmented reality, architectural visualization, and interior design tools.
- Generative interior-scene models can provide learning agents with priors over 3D scene structure and composition and generate training corpora for vision and robotics.
- Prior image-based synthesis struggles with global object coordination, omits object size, and requires minutes to synthesize a scene.
- The proposed pipeline factorizes iterative object insertion into decisions about category, spatial extent, location, and orientation, enabling global reasoning and size modeling.
- Under 2 seconds is required on average to synthesize a scene, making the method two orders of magnitude faster than prior image-based work.
- The method generates bedrooms, living rooms, offices, and bathrooms, supports partial-scene completion, and performs as well or better than prior techniques in quantitative and perceptual evaluations.
2. Related Work
Indoor scene synthesis has progressed from rule-based and statistical arrangements toward data-driven models using large scene datasets and deep generative architectures. This work uses deep convolutional models on top-down representations to generate object attributes automatically and sequentially.
- Early approaches used interior-design principles, simple statistical relationships, or fully data-driven synthesis limited by small datasets and older learning methods.
- Later methods used directed graphical models or probabilistic grammars, but relied on layout heuristics, human-activity data, or manual object-group annotations.
- The proposed model uses deep convolutional generative models to generate category, location, orientation, and size fully automatically.
- Deep neural approaches include GANs for attribute matrices and GRAINS for structured scene hierarchies.
- The pipeline uses conditional VAEs and conditional GANs to model potentially multimodal object-attribute distributions.
- Scenes are generated autoregressively, with each object-by-object step conditioned on the scene generated so far.
3. Model
The model represents scenes from above and iteratively inserts objects through four decision modules for category, location, orientation, and dimensions. Its factorization supports global category reasoning, multimodal spatial predictions, constrained orientation and size distributions, and translation- and rotation-invariant processing.
- 3. Model: A sequential generative process initializes an empty or partial scene and inserts one object at a time until completion.
- 3. Model: Four modules determine whether and which category to add, where to place it, which direction it should face, and its physical dimensions.
- 3.1. Next Object Category: The category module uses a <STOP> category and scene information to choose globally appropriate next objects and decide when the scene is sufficiently populated.
- 3.1. Next Object Category: The category module avoids repeated location-driven selections, such as sampling multiple nightstands beside one bed, by reasoning globally about the scene.
- 3.2. Object Location: The location module predicts a category-conditioned pixelwise heatmap, renormalizes the selected category channel, masks invalid room or support-surface regions, and samples the resulting distribution.
- 3.2. Object Location: Predicted location distributions avoid blocking doors for beds and wardrobes and place nightstands in two concentrated modes near the bed’s head.
- 3.3. Object Orientation: The orientation module uses a CVAE to model multimodal directions, while a snapping predictor enables precise cardinal alignment when appropriate.
- 3.4. Object Dimensions: Dimension predictions capture category-specific size variability and lower variance when placement constraints narrow the feasible object sizes.
4. Data & Training
The model is trained on filtered SUNCG room data using object-removal tasks that teach modules to predict scene additions. Category ordering is canonical, while other module orderings are randomized.
- Dataset: Training uses filtered SUNCG scenes from four room types: bedrooms, living rooms, bathrooms, and offices.The dataset contains over forty thousand user-designed scenes before preprocessing.
- Training procedure: Training examples remove objects from a scene and ask modules to predict the next object to add.This procedure supplies training data for all pipeline modules.
- Training procedure: Category prediction uses a stable importance-based ordering, while other modules use randomized object orderings.The canonical category order reduces the number of valid choices at each step and supports coherent multi-object synthesis.
5. Results & Evaluation
The method generates varied complete scenes and multiple completions of partial scenes, while evaluation measures distributional faithfulness, plausibility, and synthesis speed. Across these evaluations, it is reported to outperform or match prior methods, with weaker office-scene preferences.
- Complete scene synthesis: The model captures multiple plausible arrangement patterns within each room type, such as bedrooms with desks or extra seating.Examples also include living rooms organized for conversation or television viewing.
- Scene completion: Partial-scene completion samples varied next-object additions and handles non-rectangular rooms.The same fast generative procedure can produce multiple completions from one starting partial scene.
- Distributional evaluation: Category-distribution fidelity is evaluated with KL divergence between synthesized and training-set object-category distributions.Lower divergence indicates closer agreement with the training distribution.
- Distributional evaluation: Synthetic-scene realism is evaluated by training a classifier to distinguish real from generated scenes, with lower accuracy closer to 50% considered better.The classifier uses the same top-down multi-channel representation as the generative model and is evaluated on held-out scenes.
- Speed comparison: Less than 2 seconds are required to generate a complete scene on an NVIDIA Geforce GTX 1080Ti GPU, two orders of magnitude faster than Deep Priors.The method is slower than end-to-end methods but also supports additional scene-generation capabilities.
- Perceptual study: Our scenes are preferred to Deep Priors for bedrooms and bathrooms, indistinguishable for living rooms, and less preferred for offices.Compared with held-out human-created SUNCG scenes, results are indistinguishable for bedrooms and bathrooms, nearly indistinguishable for living rooms, and less preferred for offices.
6. Conclusion
The paper concludes that its image-based convolutional pipeline rapidly synthesizes varied indoor scenes and completes partial scenes. It identifies multimodal room types, stylistic consistency, and functional environments as open challenges.
- Conclusion: The system uses top-down scene representations and separate modules to predict object category, location, orientation, and size.Together, these modules enable rapid synthesis of plausible complete scenes and automatic partial-scene completion.
- Conclusion: The method is evaluated through scene statistics, synthetic-scene classification, and forced-choice perceptual studies, and is reported to outperform prior techniques in all cases.The conclusion summarizes the evaluation across these complementary measures.
- Future work: Future work includes generating room types with multiple strong variation modes, stylistically consistent interiors, and environments supporting activities of interest.The paper suggests integrating image-based models with higher-level scene structures such as hierarchies, graphs, or programs.
A. Model Architecture Details
The appendix provides detailed neural-network architectures for each system module and reproduces the main paper’s pipeline overview.
- Architecture details: Appendix A documents the neural-network architectures used by each module.The appendix is presented as a source of precise architectural details.
- Architecture details: The appendix reproduces the pipeline overview figure from the main paper as Figure 8.This provides a reference for the system’s overall organization.
- Architecture details: The architecture details complement the paper’s high-level pipeline description.The passage explicitly distinguishes these precise details from the overview in the main paper.
A.1. Next Category
The next-category module selects which object to add from a top-down scene representation, including a stopping decision, using image and category-count information.
- A.1. Next Category: The module predicts the next object category from a top-down scene image and a bag-of-categories representation.It encodes the scene with ResNet18 and category counts with a fully connected network before combining them.
- A.1. Next Category: The broader insertion pipeline separately predicts object category, location, orientation, and dimensions from a top-down representation.
- A.1. Next Category: At test time, the module samples from its predicted category distribution to select the next object.
A.2. Location
The location module predicts a spatial distribution for object placement from a scene image, while the orientation module models multimodal directions and precise cardinal alignment.
- A.2. Location: The location module uses a ResNet34 encoder and up-convolutional blocks to output a (C + 1) × 64 × 64 distribution over object categories and locations.The output includes an additional channel beyond the C room-type categories.
- A.2. Location: Dropout and L2 regularization reduce the location module’s tendency to overfit exact centroid targets when learning smooth location distributions.Dropout is applied around the encoder and before the final 1x1 convolution.
- A.2. Location: The orientation module predicts a normalized front-direction vector through cos θ and the sign probability of sin θ instead of directly predicting the circular angle.
- A.2. Location: A conditional variational autoencoder models multimodal orientations, while a Snap Predictor can align predictions to the nearest cardinal direction.
A.4. Dimensions
The dimensions module predicts object-space bounding-box extents with a conditional variational autoencoder and refines them using an adversarial loss.
- A.4. Dimensions: The dimensions module predicts an object-space bounding box from a scene image transformed into the object category’s local coordinate frame.
- A.4. Dimensions: A conditional variational autoencoder combines a scene encoding with latent code z to generate the bounding-box [x, y] dimensions.
- A.4. Dimensions: An adversarial loss uses a convolutional discriminator on the scene and predicted bounding-box signed distance field to fine-tune size predictions.The module uses separate network weights per object category.
- A.4. Dimensions: The dimensions and orientation components each have separate convolutional architecture diagrams built from ConvBlock modules.
B. Dataset Details
The dataset preparation filters object hierarchies and scaling, augments selected rooms by rotations, and evaluates module performance and scene similarity on held-out data.
- B. Dataset Details: The dataset filtering retains selected frequently occurring objects on visible parent surfaces and removes second-tier objects whose parents were filtered out.
- B. Dataset Details: Rooms with objects scaled by more than 10% along any dimension are discarded, while smaller scaling is removed from transformation matrices.
- B. Dataset Details: Living-room and office scenes receive four rotational augmentations—0°, 90°, 180°, and 270°—to reduce overfitting, particularly for the location module.
- B. Dataset Details: Table 5 reports object-category counts across bedrooms, living rooms, offices, and bathrooms, marking possible second-tier categories in bold.
- B. Dataset Details: Held-out module evaluation reports component-appropriate metrics and improvement relative to randomly initialized networks because no natural baseline is available.Reported metrics include binary accuracy, top-n multiclass accuracy, cross-entropy, and ELBo.
- B. Dataset Details: Scene memorization is assessed by comparing maximal similarity between generated rooms and 5,000 training rooms against corresponding held-out-room distributions.The dataset-to-dataset comparison contains a spike from exact duplicate scenes in the training set.