PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases

Rio Aguina-Kang, Maxim Gumin, Do Heon Han, Stewart Morris, Seung Jean Yoo, Aditya Ganeshan, R. Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, Daniel Ritchie

arXiv:2403.09675v1cs.CVcs.GR

TL;DR

Indoor scene generators trained on existing 3D datasets use finite room types and object categories, motivating an open-universe approach. This system uses LLM-written relational programs, optimization, and VLM-based mesh retrieval; it reports preference over trained generators on closed-universe tasks and LayoutGPT on open-universe tasks.

  • Problem

    Existing indoor-scene generators trained on 3D datasets are limited to finite room types and curated object categories.

    Source: §1 · p. 2

  • Method

    The system uses LLMs to create declarative object-relation programs, optimizes their layout constraints, and retrieves and orients meshes from an unannotated database.

    Source: §10 · p. 15

  • Results

    In forced-choice perceptual studies, the system is preferred to trained generators for typical rooms and outperforms LayoutGPT for open-universe scenes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The system generates room-scale scenes of requested types with the relevant objects, rather than a fixed vocabulary of rooms and objects.

    Source: §10 · p. 15

    The current system supports only four-walled rooms and four cardinal object orientations.

    Source: §10.2 · p. 15

Abstract

from arXiv · show

We present a system for generating indoor scenes in response to text prompts. The prompts are not limited to a fixed vocabulary of scene descriptions, and the objects in generated scenes are not restricted to a fixed set of object categories -- we call this setting indoor scene generation. Unlike most prior work on indoor scene generation, our system does not require a large training dataset of existing 3D scenes. Instead, it leverages the world knowledge encoded in pre-trained large language models (LLMs) to synthesize programs in a domain-specific layout language that describe objects and spatial relations between them. Executing such a program produces a specification of a constraint satisfaction problem, which the system solves using a gradient-based optimization scheme to produce object positions and orientations. To produce object geometry, the system retrieves 3D meshes from a database. Unlike prior work which uses databases of category-annotated, mutually-aligned meshes, we develop a pipeline using vision-language models (VLMs) to retrieve meshes from massive databases of un-annotated, inconsistently-aligned meshes. Experimental evaluations show that our system outperforms generative models trained on 3D data for traditional, closed-universe scene generation tasks; it also outperforms a recent LLM-based layout generation method on open-universe scene generation.

1 INTRODUCTION

The system generates indoor scenes from open-ended prompts rather than relying on fixed room and object categories. It uses LLM-written relational programs, optimization, and mesh retrieval, and is reported to outperform prior methods on closed- and open-universe tasks.

1 INTRODUCTION

  • Dataset-trained indoor scene generators use finite room types and curated object categories, and large 3D scene datasets are time-consuming and expensive to produce.The paper frames these systems as closed-universe because their possible room and object types are fixed in advance.Source: §1 · p. 2
  • The proposed open-universe model produces structured object layouts from text prompts, addressing scenes beyond a fixed vocabulary of room and object types.The motivating question is whether any type of indoor scene can be generated with whatever objects that scene requires.Source: §1 · p. 2
  • An LLM writes a declarative domain-specific program of scene objects and spatial relations, which is converted into constraints and solved by gradient-based optimization.The authors report that directly asking an LLM to specify metric locations performs poorly, likely because coordinates do not match most of its natural-language training text.Source: §1 · p. 2
  • A VLM-based pipeline retrieves and orients meshes from massive unannotated, inconsistently aligned databases, while the paper also contributes a benchmark of varied input descriptions.The system avoids the category-annotated, mutually aligned mesh databases used by prior work.Source: §1 · p. 2
  • The system is preferred to closed-universe scene-synthesis methods on typical rooms and outperforms LayoutGPT on open-universe scenes in forced-choice studies.Evaluations also cover activity-specific and fantastical rooms, with component ablations used to validate design choices.Source: §1 · p. 2

2 RELATED WORK

Indoor scene synthesis has progressed from hand-crafted rules and constraints to data-driven methods, but most trained models remain closed-universe. Recent LLM and foundation-model work broadens text-driven generation, while this paper distinguishes its structured layouts and uncurated mesh retrieval from those alternatives.

Indoor Scene Synthesis

  • Indoor synthesis has also used hand-crafted rules, design principles, statistical object relationships, and programmatically specified constraints.Early text-to-scene work relied on laboriously hand-crafted rules.Source: §2 · p. 3
  • Deep-learning and earlier data-driven indoor synthesis methods remain closed-universe and require 3D scene training datasets.Prior approaches include Bayesian and mixture models, factor graphs, topic models, grammars, neural networks, transformers, and diffusion models.Source: §2 · p. 3

LLM-based Scene Generation

  • LayoutGPT applies LLMs to layouts, while Holodeck supports general prompts and implicit object relations but uses curated annotated aligned models and grid layouts.Holodeck also uses a simpler relation specification and lacks automatic correction of LLM errors.Source: §2 · p. 3

Open-vocabulary Text-to-3D

  • Text-to-3D methods can create content from arbitrary prompts, but their outputs are lower-quality or inconsistent and are not easily modified as individual objects.Outputs include point clouds, unstructured meshes, and isosurfaces; room-generation extensions also assume an object layout as input.Source: §2 · p. 3

3D Shape Analysis with Foundation Models

  • Foundation-model research analyzes existing 3D shapes through captioning, segmentation, and correspondence, while this system uses VLMs for retrieval, categorization, and orientation.These shape-analysis approaches do not require 3D supervision.Source: §2 · p. 3

Program Synthesis with Large Language Models

  • LLM program synthesis has been explored for competitive programming, geometric reasoning, mathematical discovery, and domain-specific language improvement.Related work also uses DSL-generating programs for visual question answering.Source: §2 · p. 3

3 OVERVIEW

The system takes a scene description, with optional room dimensions and density, and produces a room-scale scene of positioned and oriented database meshes. A program synthesizer, layout optimizer, retrieval module, and orientation module implement this pipeline.

3 OVERVIEW

  • Inputs may specify a detailed or nebulous scene description, optional room dimensions and floor-area occupancy, and assume four walls and cardinal object orientations.The paper notes these assumptions reflect many, but not all, real-world rooms.Source: §3 · p. 4
  • An LLM-based synthesizer writes a declarative Python DSL program, which the optimizer converts into a constraint problem and solves for object locations and orientations.Object declarations are also sent to retrieval from a large unannotated model database.Source: §3 · p. 4
  • The retrieval module selects a matching 3D model for each object, and an orientation module estimates its front direction before insertion into the layout.Source: §3 · p. 4

4 DESCRIBING SCENES WITH PROGRAMS

The scene language represents layouts declaratively through object declarations and spatial relations rather than requiring an LLM to specify precise coordinates. It is embedded in Python and supports reusable constructs for describing arrangements.

4 DESCRIBING SCENES WITH PROGRAMS

  • The DSL uses spatial relations instead of explicit coordinates because LLMs can place objects inaccurately, producing overlaps or floating objects.The authors expect an LLM to reason more readily about relations such as “on” or “adjacent to” than precise numeric values.Source: §4 · p. 4
  • Embedded in Python, the language combines its programming features with domain-specific object constructors, relation functions, and parameter-setting functions.Source: §4 · p. 4
  • The restaurant example uses loops to create tables and chairs, relations to constrain adjacency and facing, and unique_objects for paintings that should use different models.It also describes wall-mounted objects, aligned tables, and a row of cheese wheels on a shelf.Source: §4 · pp. 4–5

5 GENERATING SCENE PROGRAMS

The synthesizer decomposes prompt-to-program generation into staged LLM tasks, moving from a scene description to object declarations and relation code. This decomposition is intended to reduce errors and produce richer programs.

5 GENERATING SCENE PROGRAMS

  • Breaking program synthesis into smaller tasks helps avoid sparse programs and errors; natural-language description is more reliable for the LLM than direct code generation.The object and relation code stages translate a concrete description rather than synthesizing directly from the prompt.Source: §5 · p. 5

5.1 Describing the Scene in Natural Language

  • The first stage describes room properties, objects, their sizes and appearances, and how they relate, while asking the LLM to explain its reasoning.It supplies room size and target object density when the prompt omits them.Source: §5.1 · p. 6

5.2 Declaring Objects

  • The next stage generates Python declarations with object constructors, dimensions, facing information, and category labels used later in retrieval and orientation.If declarations do not meet the target occupied floor area, another LLM generates additional objects until the target is met.Source: §5.2 · p. 6

5.3 Specifying Object Relations

  • The final stage translates the described arrangements into DSL relation calls that reference the declared objects.In-context examples illustrate potentially non-obvious ways to use relations, such as placing an object in a corner with two wall relations.Source: §5.3 · p. 6

6 SCENE LAYOUT OPTIMIZATION

The system executes scene programs to construct geometric constraints, then uses gradient-based optimization to find object positions and directions. It also detects and repairs some program errors instead of discarding the entire program.

6.1 Converting Scene Programs into Constraint Problems

  • Relation functions become geometric constraints, supplemented by bounds and non-overlap constraints that preserve door and window clearance while allowing rugs to overlap other objects.Door and window clearances create space for opening doors and prevent furniture from obstructing windows.Source: §6.1 · p. 7

6.2 Solving the Constraint Problem

  • Gradient descent minimizes constraint losses from random starts; custom NOOVERLAP gradients address zero-gradient overlap cases, while repel forces spread layouts without separating objects meant to be together.Repel forces improve spacing but can prevent convergence to a solution of the original constraints.Source: §6.2 · p. 7, Figure 5 · p. 7
  • Because facing constraints make directions discrete and dependent on final positions, a multi-stage optimizer jointly handles positions and directions; the second descent can rarely leave a mismatch.That final descent usually moves objects only slightly.Source: §6.2 · p. 8

6.3 Error Correction

  • The optimizer catches code exceptions, removes the offending line, and retries execution; it detects contradictory constraint patterns and deletes the last-declared conflicting constraint.Hallucination and misuse errors are caught during execution, while contradiction errors are identified as constraint-graph patterns.Source: §6.3 · p. 8
  • If 10 optimization rounds from random starts fail to reach near-zero loss, the system treats the constraints as unsatisfiable and regenerates the program’s relations.This backtracking returns to the relation-generation stage rather than rejecting the whole scene program.Source: §6.3 · p. 8

7 RETRIEVING 3D OBJECTS

The retrieval pipeline uses VLM embeddings to search unannotated 3D assets, then applies category, single-object, and size checks to refine candidates. It also uses VLM and multimodal-LLM judgments to orient retrieved meshes.

7 RETRIEVING 3D OBJECTS

  • The system retrieves text-matching meshes by comparing a description embedding with per-mesh VLM embeddings averaged from 12 rendered views.Embeddings are precomputed, and approximate-neighbor search takes 0.5 seconds.Source: §7 · p. 8, Figure 6 · p. 9

7.1 Retrieving the Correct Category

  • Category-aware reranking prioritizes matches to the program’s object category, then a multimodal LLM filters candidates until it finds the required number of correct-category objects.Reranking first reduces the number of costly LLM filtering calls.Source: §7.1 · p. 9

7.2 Retrieving Only Single Objects

  • A multimodal LLM filters out meshes that include additional objects on or around the main object, retaining the required number of single-object candidates.Examples include a TV on a stand or chairs around a table.Source: §7.2 · p. 9

7.3 Matching the Specified Object Size

  • Bounding-box aspect ratios filter meshes that poorly match the specified dimensions, with candidates above a distortion threshold discarded unless too few would remain.The system checks a small set of rotations; the threshold is 0.4, and the lowest-loss candidates are kept if filtering would leave fewer than required.Source: §7.3 · pp. 9–10
  • For orientation, the system first aligns bounding-box up directions, then scores four orthogonal views against “the front of a C” and asks a multimodal LLM to choose between the top two.The category C is estimated for each object.Source: Figure 7 · p. 10

8 ORIENTING RETRIEVED OBJECTS

The orientation pipeline reduces object-front estimation to a six-way choice by assuming artist-created meshes generally align their up and front directions with world axes. It combines bounding-box distortion with VLM filtering and multimodal-LLM visual question answering.

8 ORIENTING RETRIEVED OBJECTS

  • The system reduces front estimation from a continuous SO(3) search to choosing which of six axis-aligned bounding-box faces represents the mesh’s front.This relies on most artist-created meshes having their up and front directions aligned with world coordinate axes.Source: §8 · p. 10
  • It uses bounding-box distortion to test upright orientation, rotating a mesh only when the y-up fit improves by a margin of 1.0.If the program gives no facing argument, the system assumes the object has no unique front and skips further orientation.Source: §8 · p. 10
  • For objects with a facing argument, distortion or VLM similarities reject two candidate directions before a multimodal LLM chooses between the remaining views.Bounding-box distortion filters non-square footprints; for squarer footprints, a VLM compares rendered views with text describing the object’s front.Source: §8 · p. 10
  • The multi-step orientation pipeline performs best among the simpler alternatives evaluated.Source: §8 · p. 10

9 RESULTS AND EVALUATION

The evaluation covers qualitative scene generation, closed- and open-universe comparisons, component studies, and timing. Across the perceptual comparisons, participants favored the system over closed-universe generative models and generally over LayoutGPT for open-universe prompts.

9.1 Qualitative Results

  • Generated scenes span typical, purpose-specific, stylistic, non-residential, and fantastical prompts; users can also set room size or fullness and obtain multiple layout variations from one program.Objects along walls can slide and sometimes exchange positions across optimized layouts.Source: §9.1 · p. 11, Figure 9 · p. 11, Figure 10 · p. 11

9.2 Closed-Universe Scene Synthesis Comparison

9.3 Open-Universe Scene Synthesis Comparison

  • Participants generally preferred our open-universe scenes over LayoutGPT except for fantastical prompts; our layouts avoid overlaps, and full retrieval found correct-category meshes where naive retrieval did not for three of every 100 objects.Source: §9.3 · pp. 12–13, Table 2 · p. 13, Figure 12 · p. 13

9.4 Ablation Studies & Other Evaluations

  • Separating natural-language scene synthesis from code translation produced more complex programs and reduced every measured error type except hallucinations.The authors attribute the hallucination increase to later stages inventing relation functions from the free-form description, and judge the trade-off worthwhile.Source: §9.4 · pp. 13–14, Table 3 · p. 13
  • About 90% true-positive and 20% false-positive rates characterized both filters; weighted re-ranking preserved category accuracy with only a small precision hit, unlike category-only ranking.Source: §9.4 · p. 14, Table 4 · p. 14, Figure 13 · p. 14
  • The two retrieval filters keep open-universe mesh retrieval viable despite unannotated assets, retaining enough valid candidates in almost all cases.Source: §9.4 · p. 14
  • Combining a VLM with a multimodal LLM yielded the highest overall orientation accuracy and improved over LLM-only on nearly all object categories.Source: Table 5 · p. 14, §9.4 · p. 15

9.5 Timing

  • About 25 minutes was the median time to generate a scene averaging 17 objects; layout generation took about four minutes, while retrieval and orientation added most of the cost.The authors report this was slower than closed-universe systems that often take seconds, but faster than text-to-3D systems that can take hours.Source: §9.5 · p. 15

10 DISCUSSION & FUTURE WORK

The discussion reports a small qualitative interior-design study, identifies current room, orientation, and program-repair limitations, and frames the system as one starting point within a larger research space.

10 DISCUSSION & FUTURE WORK

  • The system generates room-scale scenes from text using LLM-written object-relation programs, constraint optimization, and VLM-based mesh retrieval.Retrieved meshes come from a massive unannotated dataset, and the system estimates their front-facing orientation.Source: §10 · p. 15

10.1 Viability for Interior Design

  • In a qualitative study with six clients, clients and designers valued object groupings, rug placement, door clearance, and coordinated furniture, but also noted clutter and weak circulation.The authors suggest adding computational design principles to the DSL for LLM selection and application.Source: §10.1 · p. 15

10.2 Other Limitations & Future Work

  • The system supports only four-walled rooms and four cardinal object orientations; continuous orientation changes would require revising the multi-step optimizer.The authors suggest rectangular subdivisions or parametrically specified walls as possible ways to remove the room-shape restriction.Source: §10.2 · p. 15
  • Rare meshes with an up axis misaligned to world axes violate the orientation module’s assumptions and can appear tilted in scenes.Geometric heuristics or multimodal-language-model queries are proposed as possible ways to detect or filter such meshes.Source: §10.2 · p. 15
  • The program synthesizer’s error-fixing mechanisms are not foolproof; the authors propose exploring LLM self-repair as an alternative or addition.The proposed approach collects detected errors and asks the LLM to correct its earlier output.Source: §10.2 · p. 15
  • The paper explores only a small part of the open-ended VLM and LLM design space and presents the work as a springboard and baseline for further research.The authors say future work is needed to map that design space more fully.Source: §10.2 · pp. 15–16

A SCENE DESCRIPTION LANGUAGE

The scene description language embeds object constructors, spatial-relation functions, and scene settings in Python. Its functions express object placement and orientation, while constructors also encode constraints for doors and windows.

A SCENE DESCRIPTION LANGUAGE

  • Object constructors specify descriptions and dimensions, while door and window constructors add wall-placement and clearance constraints.Doors must be adjacent to walls, windows mounted on walls, and both require empty space in front.Source: A SCENE DESCRIPTION LANGUAGE · pp. 17–18
  • The language includes constructors for multiple objects, including unique_objects for retrieving different meshes for objects with the same description.It also provides functions to set scene size and floor and wall textures.Source: A SCENE DESCRIPTION LANGUAGE · p. 18
  • Placement relations express support, wall and ceiling mounting, and adjacency to walls, including optional offsets for positioning objects near walls.The DSL also includes on(top, bottom), mounted_on_wall, and mounted_on_ceiling relations.Source: A SCENE DESCRIPTION LANGUAGE · p. 18
  • The adjacent relation supports touching, directional placement, side alignment, and optional separation distances between objects.Its variants can express arrangements such as a chair north of a desk or a nightstand aligned with a bed’s head.Source: A SCENE DESCRIPTION LANGUAGE · p. 18
  • The aligned relation arranges object centers along an axis, while facing sets an object’s direction relative to a cardinal direction or another object.Objects with no specified facing direction appear in a random orientation.Source: A SCENE DESCRIPTION LANGUAGE · pp. 17–18
  • The surround relation adds objects one by one on sides with the most free space, while respecting existing adjacencies and walls.It is implemented using adjacent and facing relations.Source: A SCENE DESCRIPTION LANGUAGE · p. 18

B LAYOUT OPTIMIZER DETAILS

The layout optimizer represents objects and the scene as cuboids and defines constraint losses over their geometry. For EAST and NORTH cases, the optimizer’s directional constraints generalize to other cardinal directions.

B.1 Constraint Losses

  • The optimizer defines cuboid size, center, lower and upper corners, scene bounds, component-wise ReLU, horizontal-plane projection, and Euclidean distance to formulate constraint losses.The passage introduces the losses as the optimizer’s constraint formulation.Source: §B.1 · p. 18, B LAYOUT OPTIMIZER DETAILS · p. 18
  • For a directional relation, the optimizer selects A and B according to the cuboids’ z-size comparison.The selection is defined as A=a, B=b when a is smaller in z-size, and otherwise A=b, B=a.Source: §B.1 · p. 19
  • The EAST and NORTH constraint definitions generalize to the other cardinal directions.Source: §B.1 · p. 19

B.2 Repel Force Implementation

The repel-force implementation separates objects and walls into connected components using wall and object relations, then repels pairs across different components.

B.2 Repel Force Implementation

  • Pairs across disconnected components receive gradient repulsion proportional to max(1 − d/dmax, 0), with dmax set to one quarter of the scene’s minimum linear size.Small random noise is added to repel forces to help escape local minima.Source: §B.2 · p. 19
Loading 2403.09675v1…