PaperlayerView exact revision on arXiv

Source-linked AI summary

Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation

Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, Zeyu Ma, Jia Deng

arXiv:2406.11824v1cs.CV

TL;DR

Existing procedural generation was limited to natural scenes, despite the importance of domain-relevant synthetic data for indoor applications. Infinigen Indoors extends fully procedural generation to photorealistic indoor assets and scenes, adds constraint-based composition and simulator export, and improves indoor-scene generalization in evaluated tasks.

  • Problem

    Existing Infinigen is limited to natural scenes, while domain-relevant synthetic data remains important for indoor applications such as robotics and augmented reality.

  • Method

    Infinigen Indoors combines a fully procedural library of indoor assets and scenes with a constraint-language-and-solver arrangement system and export to real-time simulators.

  • Results

    Generated data improves generalization performance on indoor scenes for shadow removal and occlusion boundary detection.

  • Takeaways & Limitations

    The system provides customizable photorealistic indoor data and assets for training embodied agents in real-time simulation environments.

  • Takeaways & Limitations

    The initial room-specific constraint specification averages approximately 15 constraints, or about 15 lines of Python, when customization is required.

Abstract

from arXiv · show

We introduce Infinigen Indoors, a Blender-based procedural generator of photorealistic indoor scenes. It builds upon the existing Infinigen system, which focuses on natural scenes, but expands its coverage to indoor scenes by introducing a diverse library of procedural indoor assets, including furniture, architecture elements, appliances, and other day-to-day objects. It also introduces a constraint-based arrangement system, which consists of a domain-specific language for expressing diverse constraints on scene composition, and a solver that generates scene compositions that maximally satisfy the constraints. We provide an export tool that allows the generated 3D objects and scenes to be directly used for training embodied agents in real-time simulators such as Omniverse and Unreal. Infinigen Indoors is open-sourced under the BSD license. Please visit https://infinigen.org for code and videos.

1. Introduction

Infinigen Indoors extends fully procedural photorealistic scene generation from natural environments to diverse, customizable indoor scenes. It adds procedural indoor assets, constraint-based arrangement, simulator export, and evidence that generated data improves indoor-scene generalization.

  • Motivation: Synthetic data supports large-scale vision and embodied-AI training by providing unlimited renders with automatically generated 3D ground truth.Procedural generation further increases variation through randomized mathematical rules.
  • System scope: Infinigen Indoors addresses Infinigen’s limitation to natural scenes by generating photorealistic indoor objects and complete, physically and semantically plausible multi-room, multi-floor buildings.Its asset library includes furniture, appliances, cookware, dining utensils, architectural elements, and everyday objects.
  • System scope: The generator creates unlimited object- and scene-level variations fully procedurally, without external assets, using mathematical rules from scratch.This preserves the original Infinigen design while extending it to indoor content.
  • Customization: Its constraint-based arrangement system lets users specify symmetry, spatial, quantity, physics, and accessibility requirements through Python APIs.A solver greedily applies simulated annealing to whole-house plans, large furniture layouts, and small objects to maximize constraint satisfaction.
  • Customization: Separating constraint specification from solving improves usability and customizability, allowing objectives to be expressed without modifying implementation-level procedure rules.The system is presented as the first solver integrated with an open-source, fully procedural generator.
  • Deployment: An export tool makes generated objects and scenes directly usable for training embodied agents in real-time simulators such as Omniverse and Unreal.The exported assets can supply simulation environments and enhance domain randomization.
  • Evaluation: Experiments on shadow removal and occlusion boundary detection show that data generated by the system improves generalization performance on indoor scenes.These tasks were selected because they lack abundant existing training data; the system also provides automatic annotations from rendered scenes.

2. Related Work

Prior indoor-scene resources are limited by collection costs, inaccessible or static assets, and less customizable constraint representations. Infinigen Indoors addresses these gaps with procedural indoor assets and a higher-level, compositional constraint language integrated with procedural generation.

  • Real-world datasets: Real-world indoor datasets are labor-intensive to collect, limited in size, and affected by incomplete or unreliable 3D ground truth from depth sensors.Reported sensor issues include limited resolution and range, transparent or reflective surfaces, and object-edge artifacts.
  • Synthetic indoor datasets: Many synthetic indoor datasets rely on inaccessible or static 3D assets, limiting their utility and variation.The related-work discussion contrasts these datasets with systems that provide more open and procedurally varied assets.
  • Scene arrangement: Existing constraint approaches include hardcoded programs, physical relations, learned constraints, and probabilistic graphs, but learned constraints are less customizable, interpretable, and controllable.The paper positions its explicit constraint language against these alternative representations.
  • Infinigen Indoors: Infinigen Indoors integrates a high-level, extensible constraint language directly with procedural object generators.Its constraints target abstract object classes and support compositional relations such as placing glassware on shelves against a dining-room wall.
  • Infinigen Indoors: The system’s procedural library covers architectural elements, appliances, furniture, and bathroom fixtures, including doors, staircases, windows, ovens, sinks, sofas, beds, tables, and shelves.The cited figures show random samples of these generated indoor asset categories.

3. Method

Infinigen Indoors combines procedurally generated indoor assets with a constraint-based system for constructing and exporting photorealistic indoor scenes. Its API expresses geometric and semantic requirements, while a simulated-annealing solver searches hierarchical scene arrangements and supports real-time simulator export.

  • Procedural Asset Generation: Procedural assets include tableware, decorations, materials, architectural fixtures, and large objects for cooking, seating, and storage.Small objects can attach to support surfaces, walls, or ceilings, while soft-body simulation models blankets, clothing, and pillows on supporting surfaces.
  • Procedural Asset Generation: The system provides 79 randomized procedural object generators spanning appliances, architectural elements, furniture, and other indoor assets.Generators expose human-controllable parameters that can be randomized or manually overridden, with meshes created through Blender geometry nodes, modifiers, and mesh manipulation.
  • Constraint Specification API: The Constraint Specification API represents geometric, filtering, and arithmetic operations as compute graphs that can target objects by semantics, scene-graph relations, rooms, or surfaces.Supported geometric properties include distance, symmetry, alignment, free space, accessibility, and object volume or area.
  • Constraint Specification API: Residential constraint programs encode hard and soft requirements for room-specific ergonomics and semantic relations, with an example containing 11 hard constraints and 25 score terms.The example graph contains 1058 nodes, and the initial specification averages approximately 15 room-specific constraints written in about 15 Python lines.
  • Arrangement Solver: The solver uses simulated annealing with Metropolis-Hastings moves, including addition, deletion, resampling, translation, and rotation, while restricting motion to constraint-defined degrees of freedom.Optimization proceeds hierarchically across large, medium, and small objects; move probabilities are scheduled so discrete moves decay while continuous moves increase.
  • Arrangement Solver: The floorplan solver generates room-adjacency graphs from a probabilistic context-free grammar or user input, then optimizes realistic full-house room meshes subject to those constraints.It supports single- and multistory floorplans, with staircases connecting adjacent floors.
  • Data Export: A one-click USD export enables generated scenes to run at interactive frame rates in Omniverse Isaac Sim and Unreal Engine 5 for embodied-agent training.The exporter uses Universal Scene Description or other formats to transfer indoor assets and scenes to real-time simulators.

4. Experiments

Experiments assess solver efficiency, perceptual quality, and synthetic-data utility. The optimized solver improves runtime and score, human subjects prefer the generated scenes on several perceptual criteria, and Infinigen Indoors-trained models generalize better for occlusion-boundary estimation.

  • Solver Performance: The optimized solver provides a ≈3x speedup over the non-optimized version, with most gains from BVH caching and Plane Hashing.When run for the same duration as the non-optimized system, the optimized solver achieves a 28% score increase.
  • Solver Performance: Discrete pose, relation, and resampling moves improve visual quality but reduce quantitative score and increase runtime.The ablation evaluates solver variants over 20 random scenes with 5k solver steps.
  • Perceptual Study: Crowdsourced subjects preferred Infinigen Indoors over several prior systems for realism, layout realism, and lack of errors.The paper notes that perceived realism may also reflect asset and lighting quality.
  • Shadow Removal: The shadow-removal experiment uses 2k image pairs of shadow and shadow-free variants generated by toggling Blender lighting shadows.Shadow masks are produced with Otsu’s thresholding, and ShadowFormer is evaluated with real-only and real-plus-Infinigen-Indoors training variations.
  • Occlusion Boundary Estimation: For occlusion-boundary estimation, the study produces 1464 annotated images and compares U-Net models trained on Infinigen Indoors, Infinigen, and Hypersim data.Evaluation uses a curated test set of photorealistic artist-designed synthetic 3D scenes for architectural visualization and reports ODS, OIS, and mAP.
  • Occlusion Boundary Estimation: The Infinigen Indoors-trained model achieves higher performance across all reported occlusion-boundary metrics and generalizes better.Qualitative results are shown in Figure 14.

5. Contributions & Acknowledgements

The paper credits its contributors with distinct responsibilities spanning coordination, constraint solving, procedural assets, export utilities, and project leadership. It also acknowledges support from the National Science Foundation and Amazon.

  • Contributions: Alexander Raistrick, Lingjie Mei, and Karhan Kayan contributed equally while leading coordination, constraint and object solvers, room solving, procedural assets, and geometric constraints.They are ordered randomly and each may list their name first on a CV.
  • Contributions: David Yan developed the scene exporter and utilities, while other contributors developed procedural assets and utilities.The listed contributors include Yiming Zuo, Beining Han, Hongyu Wen, and Stamatis Alexandropoulos, among others.
  • Acknowledgements: Jia Deng conceptualized and led the project and set its directions.The work was partially supported by the National Science Foundation and Amazon.

A.1. API Description

The API provides composable semantic, relational, geometric, and arithmetic operations for expressing indoor-scene constraints. These operations support placement, stability, accessibility, alignment, and symmetry objectives over selected objects.

  • Filtering and relations: Semantic filtering extracts object subsets by predicates such as dining rooms or shelving, while related-to connects objects through configurable relations.The operations can be composed to express context-dependent constraints over hierarchical object classes.
  • Compositional objectives: The API supports compositional objectives such as maximizing dining chairs related to tables inside rooms adjacent to kitchens.Scene retrieves all current objects, while count, area, and volume provide aggregate numerical operations over object sets.
  • Physical relations: StableAgainst requires parallel surfaces, no overhang, and a specified margin, whereas SupportedBy additionally requires support sufficient to prevent gravitational tipping.A cup can be supported by a table while still overhanging and therefore not being stable against it.
  • Geometric objectives: Geometric operators measure minimum distance, angle alignment, rotational asymmetry, reflection asymmetry, accessibility, area, and volume for selected object sets.Angle alignment can orient chairs toward tables or furniture toward walls, while asymmetry scores encourage regular arrangements.
  • Symmetry objectives: Reflection asymmetry compares objects with reflected counterparts using positional and angular deviations weighted by object bounding-box volume.The process reflects object positions and orientations, performs cost-minimizing bipartite matching, and normalizes total deviation.

B. Extended Random Sample & Constraint Code for Residential Scenes

Residential scenes are generated from a constraint program that combines room-specific and general objectives, while the same arrangement system also specifies warehouse layouts. The constraints cover semantic assignments, proportions, stability, and spatial relationships.

  • Residential constraints: 105 soft and hard constraints define residential layouts, including 19 for dining rooms, 14 for living rooms, 9 for bathrooms, 18 for kitchens, 16 for warehouses, and 30 for all rooms.They govern assignments, object ratios, stability, distances, and related scene properties.
  • Warehouse example: A warehouse constraint program specifies high-level objectives for shelf furniture and smaller objects on wooden pallets.The same system supports further preferences such as object height on shelves and front-of-store placement.

D.1. Floor plan graph generation

Floor plans are generated as constrained room graphs and then converted into geometric spaces whose assignments satisfy connectivity, privacy, exterior access, and shape objectives. Optimization balances twelve floor-plan constraints, including paths, areas, convexity, walls, and collinearity.

  • Floor-plan graphs: Each floor plan contains 1 to 3 floors, with graph nodes representing typed rooms and edges representing room connectivity.Supported room types include kitchens, bedrooms, living rooms, closets, hallways, bathrooms, garages, balconies, dining rooms, utilities, and staircases.
  • Graph generation: A probabilistic context-free grammar grows the graph from a living-room node, after which additional edges and shared hallways are added.Room-count probabilities are conditioned on parent and child room types.
  • Validity constraints: Invalid plans are rejected when they violate bathroom privacy, planarity, designated-room counts, or required living-room and bathroom presence.Exterior-facing rooms must also access the house exterior, and staircase assignments are restricted by staircase-placeholder intersections.
  • Geometric realization: Contours are estimated from typical room areas, diversified by corner profiles, and subdivided through repeated axis-aligned splits before spaces are merged and assigned to graph rooms.Divisions are grid-rounded, poor aspect ratios are rejected, and adjacency supports door placement.
  • Floor-plan optimization: The floor-plan objective combines constraints encouraging direct entrance paths, typical room-area proportions, square room aspect ratios, convexity, concise walls, functional area, and collinear walls.These objectives are evaluated over floor and room assignments to guide optimization.

E.3. Move Implementations

The solver uses discrete and continuous moves to explore constrained scene arrangements, including object addition, deletion, resampling, reassignment, translation, rotation, and pose reinitialization. Procedural asset coverage supplies the objects and materials used by these moves.

  • Discrete moves: Addition searches cardinality bounds and proposes objects whose assignments satisfy relevant semantic, relational, and activated constraints.The relation-assignment search can be exponential, but the relevant constraint depth and branching factor are usually small.
  • Object changes: Resample replaces an object with a new instance of the same class and parameters, potentially changing dimensions or other geometry-dependent scores.Deletion uses the same cardinality-bound logic to remove selected objects.
  • Continuous moves: Translate and Rotate apply Gaussian perturbations within the object’s constrained translational or rotational degrees of freedom.ReinitPose instead samples a new position and orientation on the same constraint subspace.
  • Relation changes: ReassignPlane moves an object to another compatible surface, while ReassignTarget changes its parent object within the scene graph.Examples include moving a sofa to another wall or a plant pot to another shelf or table.
  • Asset coverage: The system provides 79 randomized procedural object generators across appliances, architectural elements, furniture, decorations, and small objects, plus 30 material generators.The implementation contains 40k lines of code, including 25k lines for object and material generators.

G.1. Shadow Removal

The shadow-removal experiments trained two ShadowFormer variants under matched implementation settings, while excluding an unreproducible pretrained checkpoint and an uninformative SSIM metric. The evaluation also compared realism judgments across generated results.

  • Training setup: ShadowFormer was trained for 30k steps per variant, using an L1 loss and a validation set built from four ISTD training scenes.The variants were trained on real data alone or on real data combined with 2k synthetic image pairs.
  • Perceptual evaluation: The perceptual study reported preferences for Infinigen Indoors across realism, layout realism, and absence of obvious errors, with realism potentially affected by asset and lighting quality.
  • Evaluation caveats: The experiments excluded the codebase’s pretrained model after reproduction attempts failed, while applying identical implementation details to both newly trained variants.
  • Evaluation caveats: SSIM was not reported because the two models had equivalent performance after rounding to two decimal places.

G.2. Occlusion Boundaries

The occlusion-boundary experiments trained separate U-Net models on images from three synthetic sources and evaluated them on artist-designed photorealistic scenes. Hypersim boundaries were approximated from depth gradients because photorealistic datasets lack ground-truth occlusion boundaries.

  • Training setup: Three U-Net models were trained from scratch on Infinigen Indoors, Infinigen, and Hypersim images, using augmentation and binary cross-entropy loss.
  • Boundary construction: Hypersim occlusion boundaries were approximated by thresholding gradients of provided depth maps, with the threshold tuned on Hypersim.
  • Evaluation: The models were evaluated on a curated test set of photorealistic artist-designed synthetic 3D scenes, whose ground-truth boundaries were extracted using Infinigen tools.
  • Evaluation: The qualitative results were presented as supporting better generalization for the Infinigen Indoors-trained model.
  • Perceptual evaluation: The perceptual comparison used paired images from Infinigen Indoors, ProcTHOR, ATISS, SceneFormer, and FastSynth to assess realism and layout realism.
  • Asset coverage: The asset coverage includes household appliances, bathroom fixtures, clothing, architectural elements, seating, tables, and tableware.
Loading 2406.11824v1…