PaperlayerView exact revision on arXiv

Source-linked AI summary

Holodeck: Language Guided Generation of 3D Embodied AI Environments

Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, Chris Callison-Burch, Mark Yatskar, Aniruddha Kembhavi, Christopher Clark

arXiv:2312.09067v2cs.CVcs.AIcs.CLcs.RO

TL;DR

Creating 3D environments for Embodied AI requires substantial human effort, and existing approaches cover limited scene types. Holodeck automatically generates interactive environments from text using language-model planning, asset retrieval, and constraint-based layout optimization. Human studies show it surpasses PROCTHOR [7] in residential scene generation and produces high-quality outputs across diverse scene types.

  • Problem

    Creating semantically consistent 3D environments for Embodied AI requires considerable human effort, and existing approaches cover limited environment types.

    Source: §1 · p. 2

  • Method

    Holodeck combines language-model planning, 3D asset retrieval, and constraint-based layout optimization to automatically generate interactive environments from text.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Human studies show Holodeck surpasses PROCTHOR [7] in residential scene generation while producing high-quality outputs across diverse scene types.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Training on Holodeck-generated scenes supports better object navigation in previously unseen, expert-designed scene types.

    Source: §1 · p. 2

    Holodeck’s asset coverage is limited by objects unavailable in Objaverse, such as a dental X-ray machine.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

3D simulated environments play a critical role in Embodied AI, but their creation requires expertise and extensive manual effort, restricting their diversity and scope. To mitigate this limitation, we present Holodeck, a system that generates 3D environments to match a user-supplied prompt fully automatedly. Holodeck can generate diverse scenes, e.g., arcades, spas, and museums, adjust the designs for styles, and can capture the semantics of complex queries such as "apartment for a researcher with a cat" and "office of a professor who is a fan of Star Wars". Holodeck leverages a large language model (i.e., GPT-4) for common sense knowledge about what the scene might look like and uses a large collection of 3D assets from Objaverse to populate the scene with diverse objects. To address the challenge of positioning objects correctly, we prompt GPT-4 to generate spatial relational constraints between objects and then optimize the layout to satisfy those constraints. Our large-scale human evaluation shows that annotators prefer Holodeck over manually designed procedural baselines in residential scenes and that Holodeck can produce high-quality outputs for diverse scene types. We also demonstrate an exciting application of Holodeck in Embodied AI, training agents to navigate in novel scenes like music rooms and daycares without human-constructed data, which is a significant step forward in developing general-purpose embodied agents.

1. Introduction

HOLODECK automatically generates diverse, customized, interactive 3D environments from text using GPT-4 and constraint-based optimization. Human studies support its scene quality, while navigation experiments demonstrate benefits in previously unseen scene types.

1. Introduction

  • Existing environment-generation approaches require substantial human effort or have limitations in interactivity, geometric quality, overall scene consistency, and dataset dependence.Manual design [6] [13] [26], scanning [8], and hard-coded procedural generation [7] require substantial effort, limiting environment types. Text-to-3D methods [11] [18] [58] introduce artifacts and lack necessary interactivity, while floor-plan [19] and arrangement models lack overall consistency and depend heavily on task-specific datasets.Source: §1 · p. 2
  • HOLODECK builds on AI2-THOR [26] to generate customized, interactive 3D environments from text, using GPT-4 and constraint-based optimization.GPT-4 designs floor plans, assigns materials, and installs doorways and windows; constraint-based optimization arranges assets coherently. The system selects from over 50K diverse, high-quality 3D assets from Objaverse [9].Source: §1 · p. 2
  • HOLODECK uses LLM commonsense and spatial knowledge to interpret specific or abstract prompts, generate diverse scene types, and customize visual styles.Examples include arcades, spas, museums, Star Wars-themed object selection, and Victorian-style textures and designs. Spatial reasoning supports connected-office floor plans and regular object arrangements.Source: §1 · p. 2
  • 680 participants evaluated HOLODECK, which significantly surpassed procedural baseline PROCTHOR [7] on residential scenes and produced high-quality outputs across scene types.Human evaluation also validated accurate asset selection and realistic layout design.Source: §1 · p. 2
  • Agents trained on HOLODECK-generated scenes navigate better in expert-designed, previously unseen scene types, including Daycare and Gym.The experiments target zero-shot object navigation and demonstrate adaptation to new scene types and objects.Source: §1 · p. 2

2. Related Work

Holodeck targets text-driven, interactive environments for embodied-agent training, using an asset database to populate scenes and spatial constraints to optimize layouts. The related work highlights limitations in environment scalability, direct numerical layout generation, and the interactivity of generated 3D representations.

Embodied AI Environments

  • Artist-designed environments [6] [13] [25] [26] are difficult to scale, while scan-based scenes reduce labor but offer less interactivity.PROCTHOR [7] generates large-scale interactive training environments; Phone2Proc [8] uses phone scans to create semantically similar training scenes; concurrent RoboGen [52] generates diversified robotic tasks and scenes. These approaches parallel Holodeck’s aim of training generalizable embodied agents.Source: §2 · p. 2

Large Language Model for Scene Design

  • Learning spatial priors from existing scene databases [4] [50] [59] limits applicability when datasets cover restricted categories, as with 3D-FRONT [12].Another line of scene-design work uses user input to refine scenes iteratively.Source: §2 · pp. 2–3
  • Unlike direct numerical layout generation with LLMs [10] [29], Holodeck samples spatial constraints and uses a solver to produce physically plausible arrangements.Direct numerical predictions can produce overlapping assets. The reported human study favors Holodeck layouts over those generated end-to-end by LLMs.Source: §2 · p. 3

Text-driven 3D Generation

  • Holodeck uses an asset database for interactive environments, whereas text-to-image-and-depth methods produce meshes or NeRFs lacking modular composability and interactive affordances [11] [18] [58].Earlier methods learn shapes or textures from category-specific datasets [16] [35]. CLIP enables zero-shot texture and object generation [14] [20] [28] [32] [33] [39], but these approaches struggle with complex scenes.Source: §2 · p. 3

3. HOLODECK

Holodeck translates natural-language specifications into scenes through four specialized modules for floors and walls, doorways and windows, object selection, and object layout. Its layout module optimizes LLM-generated spatial relations rather than relying on direct numerical object placement.

Overall Prompt Design

  • Each module receives an LLM prompt containing a task description, output format, and one-shot example; post-processing converts responses into low-level scene specifications.The four modules construct floors and walls, add doorways and windows, retrieve objects, and arrange them using spatial constraints.Source: §3 · pp. 3–4

Floor & Wall Module

  • GPT-4 generates rectangular rooms from corner coordinates, proposing dimensions and connectivity to create prompt-appropriate multi-room floor plans.Additional prompt guidance, such as a minimum room area of 9 m2, helps avoid overly small rooms.Source: §3 · p. 4
  • 236 materials, each available in 148 colors, let Holodeck match floor and wall finishes to scene semantics and explicit texture requirements.Examples include concrete finishes for prison cells and requested pink coloring, red brick walls, or checkered floors.Source: §3 · p. 4

Doorway & Window Module

  • 40 door styles and 21 window types support customized openings, with separate LLM queries proposing room connections and windows.Adjustable properties include size, height, and quantity, enabling wider doors for wheelchair accessibility or multiple floor-to-ceiling windows for sunrooms.Source: §3 · p. 4

Object Selection Module

  • LLM-proposed object descriptions and dimensions guide Objaverse retrieval using visual similarity, textual similarity, and size matching.Object customization covers floor-mounted, wall-mounted, tabletop, and ceiling objects.Source: §3 · p. 5

Constraint-based Layout Design Module

  • Holodeck replaces direct bounding-box prediction with LLM-generated spatial relations, treating ten relation types as soft constraints while enforcing collision-free, in-bounds placement.Relations form a room-level scene graph spanning five categories: global location, distance, relative position, alignment, and rotation. Soft constraints may be violated when satisfying them all is infeasible.Source: §3 · p. 5

Constraint Satisfaction

  • 30 seconds of anchor-first, autoregressive depth-first search generates candidate layouts, returning the valid layout satisfying the most total constraints.An LLM selects the anchor object, and every placement must satisfy all hard constraints. Probabilistic LLM outputs allow multiple valid layouts for the same prompt.Source: §3 · pp. 5–6

Leveraging Objaverse Assets

  • 51,464 annotated assets combine curated indoor Objaverse 1.0 models with PROCTHOR assets to support diverse, customized scenes.GPT-4-Vison adds descriptions, scale, and canonical views. AI2-THOR import includes mesh reduction to minimize loading time, plus visibility-point and collider generation.Source: §3 · p. 6

4. Human Evaluation

Human evaluations involving 680 graduate students favor HOLODECK over PROCTHOR for residential scenes and demonstrate its ability to generate diverse scene types. Layout ablations support its spatial relational constraints, although complex layouts and missing assets remain limitations.

4.1. Comparative Analysis on Residential Scenes

  • HOLODECK’s CLIP Score [17] exceeds PROCTHOR’s by large margins and approaches human-designed iTHOR [26] scenes across four residential scene types.The metric measures visual coherence between a scene’s top-down view and a prompt naming its scene type, agreeing with the human evaluation.Source: Figure 10 · p. 6, §4.1 · p. 6
  • 64.4% favor HOLODECK over PROCTHOR for Overall Preference, alongside 59.8% for Asset Selection and 56.9% for Layout Coherence.The residential comparison used 120 scene pairs, with both systems accessing the same Objaverse assets and annotators comparing shuffled top-down images. Across all three studies, 680 graduate students participated.Source: §4.1 · p. 6, §4 · p. 6

4.2. HOLODECK on Diverse Scenes

  • 28 of 52 scene types receive higher human preference scores for HOLODECK than PROCTHOR’s residential scenes.Scene-name-only prompts produced 260 examples across five categories from MIT Scenes Dataset: Stores, Home, Public Spaces, Leisure, and Working Space. Annotators rated top-down images and 360-degree videos from 1 to 5, with residential PROCTHOR and iTHOR scenes providing reference scores.Source: Figure 11 · p. 7, §4.2 · p. 7
  • HOLODECK struggles with complex layouts and unavailable specialized assets; evaluation excludes scene types requiring special structures.Examples include restaurant layouts and missing dental x-ray machines; framework restrictions exclude structures such as swimming pools and subways. The authors propose expanding the asset collection and developing more sophisticated layout algorithms.Source: §4.2 · p. 7

4.3. Ablation Study on Layout Design

  • CONSTRAINT significantly outperforms alternative layouts in bathrooms, bedrooms, and living rooms, but performs similarly to EDGE in kitchens.Humans ranked layouts with identical objects using Mean Reciprocal Rank; EDGE places objects along walls, a common kitchen arrangement. ABSOLUTE, which requests coordinates and orientations directly from an LLM akin to LayoutGPT [10], performs no better than RANDOM because of collision and boundary errors.Source: Table 1 · p. 7, §4.3 · pp. 7–8

5. Object Navigation in Novel Environments

Fine-tuning ObjectNav agents on HOLODECK-generated scenes yields the best average zero-shot performance on NOVELTYTHOR, an artist-designed benchmark of diverse environments. The results suggest that HOLODECK produces more human-like object layouts.

5. Object Navigation in Novel Environments

  • NOVELTYTHOR introduces artist-designed, diverse environments for ObjectNav [1], where robots navigate toward a specified object category.Existing ObjectNav benchmarks [6] [7] cover only household environments and collectively support just 16 object types.Source: §5 · p. 8
  • 100 HOLODECK-generated scenes per scene type fine-tune an ObjectNav model pretrained on PROCTHOR-10K [26] before evaluation on NOVELTYTHOR.HOLODECK receives the novel scene type as its generation prompt.Source: §5 · p. 8
  • HOLODECK achieves the best average zero-shot performance on NOVELTYTHOR, substantially outperforming baselines on Office, Daycare, and Music Room.+OBJAVERSE enhances PROCTHOR with HOLODECK’s scene-type-specific Objaverse asset selection. +HOLODECK and +OBJAVERSE perform similarly on Gym and Arcade.Source: §5 · p. 8
  • 50M training steps are used for each model, using CLIP-based architectures [24] with a CNN visual encoder and a GRU for temporal information.Training uses 100 scenes and takes approximately one day on eight Quadro RTX 8000 GPUs. Each checkpoint is selected using its model’s best performance on its own validation scenes.Source: §5 · p. 8
  • Placement differences between +HOLODECK and +OBJAVERSE suggest that HOLODECK better reproduces human-designed layouts.HOLODECK and NOVELTYTHOR music rooms similarly group pianos, violin cases, and cellos nearby. PROCTHOR often performs near random because its training provides poor object coverage.Source: §5 · p. 8

6. Conclusion and Limitation · A. Details of HOLODECK

HOLODECK uses large language models and text descriptions to generate diverse, interactive Embodied AI environments. The authors assess its quality through large-scale human evaluation and validate its utility through object navigation in novel scenes.

6. Conclusion and Limitation

  • HOLODECK uses large language models to generate diverse, interactive Embodied AI environments.Source: §6 · p. 9
  • Object navigation in novel scenes validates HOLODECK's utility for Embodied AI.Source: §6 · p. 9
  • The authors assess HOLODECK's quality through large-scale human evaluation.Source: §6 · p. 9
  • Text descriptions provide the input for HOLODECK's environment generation.Source: §6 · p. 9
  • The authors plan to expand HOLODECK's collection of 3D assets.Source: §6 · p. 9
  • Future work will explore broader applications of HOLODECK in Embodied AI.Source: §6 · p. 9

A.1. Efficiency and Cost · A.2. Floor & Wall Modules

HOLODECK costs approximately $0.2 per room using gpt-4-1106-preview, and its current implementation generates a room in about 3 minutes on an M1 MacBook. Its floor and wall modules combine LLM-generated room specifications with CLIP -based material matching.

A.1. Efficiency and Cost

  • $0.2 per room is the approximate cost of using OpenAI’s gpt-4-1106-preview model.Source: §A.1 · p. 12
  • 3 minutes is the approximate single-room generation time on an M1 MacBook, including API calls and layout optimization.Source: §A.1 · p. 12
  • 3+3×k API calls are required to create an interactive house containing k rooms.Source: §A.1 · p. 12

A.2. Floor & Wall Modules

  • The Floor Module’s LLM output specifies each room’s type, floor and wall appearance, and four corner-coordinate tuples.Source: §A.2 · p. 12
  • 236 material images, consistent with PROCTHOR [7], are compared against LLM descriptions using CLIP, selecting the highest-similarity material.CLIP also selects the closest descriptive color from 148 Matplotlib colors to refine material selection.Source: §A.2 · p. 12
  • The LLM suggests wall height from the user’s input, potentially recommending high ceilings for commercial spaces such as museums.Source: §A.2 · p. 12

A.3. Doorway & Window Modules

HOLODECK uses doors and windows from PROCTHOR [7], with an LLM specifying their configurations and CLIP matching door images to appearance descriptions.

A.3. Doorway & Window Modules

  • 40 doors and 21 windows from PROCTHOR [7] provide HOLODECK’s collection, with CLIP selecting doors whose images best match LLM-generated appearance descriptions.Source: §A.3 · p. 12
  • The LLM specifies each door’s connected rooms, connection type, size, and appearance.Connection types are doorframe (frame without a door), doorway (frame with a door), and open (no separating wall). Sizes are single (one meter wide) or double (two meters wide).Source: §A.3 · p. 12
  • For windows, the LLM specifies the room, wall direction, type, width and height, quantity per wall, and distance from the floor to the window’s base.Wall directions are south, north, east, or west; window types are fixed, slider, or hung.Source: §A.3 · p. 12

A.4. Object Selection Module

The object selection module retrieves Objaverse assets matching LLM-generated descriptions and dimensions. It selects assets using a weighted combination of visual similarity, textual similarity, and size discrepancy.

A.4. Object Selection Module

  • Retrieval compares each LLM-proposed object's detailed description and bounding-box dimensions with Objaverse assets' descriptions, dimensions, and rendered images.Each asset has images captured at 0°, 45°, and −45°.Source: §A.4 · p. 12
  • Visual similarity V takes the maximum CLIP similarity between the candidate asset's rendered images and the LLM-proposed object's textual description.Taking the maximum selects the strongest match across the available views.Source: §A.4 · p. 12
  • Textual similarity T uses SBERT with the all-mpnet-base-v2 checkpoint to compare asset and requested descriptions, helping retrieve the correct object category.This complements visual matching with description-level category information.Source: §A.4 · p. 12
  • Size discrepancy S compares candidate and requested bounding-box dimensions; similarly sized objects receive smaller S values.Size matching distinguishes otherwise similar assets and supports scene requirements, such as a larger sofa for a large living room.Source: §A.4 · pp. 12–13
  • The module selects the asset with the highest weighted matching score: αV + βT − γS.The weights are α = 100, β = 1, and γ = 10.Source: §A.4 · p. 13

A.5. Layout Design Module … A.9. Prompt

Holodeck combines LLM-generated spatial constraints, GPT-4-V asset annotations, and an automated 3D-asset import pipeline to construct scenes. It supports Unity and Blender rendering and provides prompt templates for its modules and asset annotation.

A.5. Layout Design Module

  • Holodeck positions floor objects using LLM-generated constraints covering global placement, distance, relative position, alignment, and facing direction.Each object receives a combined constraint list, such as placing a coffee table centrally, near and in front of a sofa, and facing it.Source: §A.5 · p. 13
  • The grid-based DFS solver treats spatial relations softly but rejects placements that collide with objects or cross room boundaries.It explores placements for an LLM-selected anchor before placing remaining objects, choosing the solution satisfying the most constraints. Objects use center coordinates, bounding-box dimensions, and rotations of 0°, 90°, 180°, or 270°.Source: §A.5 · p. 13
  • The MILP solver enforces spatial constraints as hard requirements, except near/far distances, which enter the optimization objective.Objects use (x, y, rotate90, rotate180), with Boolean rotation variables and auxiliary variables that linearize OR conditions. GUROBI solves the formulation.Source: §A.5 · p. 13
  • Wall-object placement specifies an underlying floor object and height, while small surface objects use LLM-proposed placements and AI2-THOR RandomSpawn.Wall-object height is measured in centimeters from the floor to the object's base. RandomSpawn provides randomized, efficient placement of small objects on larger surfaces.Source: §A.5 · p. 14

A.6. GPT-4-V for 3D Asset Annotation

  • GPT-4-V annotates assets from four orthogonal views to improve object retrieval and placement.Annotations include category, nearest WordNet synset for navigation, dimensions, volume, mass, front view, description, materials, and Boolean placement attributes. Input views show rotations of 0°, 90°, 180°, and 270°.Source: §A.6 · p. 14

A.7. Importing Objaverse Assets into AI2-THOR

  • Holodeck’s automated import pipeline supports Objaverse assets and arbitrary 3D objects, including text-to-3D outputs.Processing converts models into runtime meshes, generates visibility points, decomposes meshes into convex components for collision detection, and compresses textures and model formats. Caching accelerates repeated loading, while unloading supports memory management for thousands of runtime objects.Source: §A.7 · p. 14

A.8. Rendering Options

  • Holodeck defaults to Unity rendering for efficient embodied-agent training and supports Blender rendering for greater realism.Source: §A.8 · p. 14

A.9. Prompt

  • Complete module prompt templates appear in Figures 20 and 21, while Figure 22 provides the GPT-4-V asset-annotation prompt.Source: §A.9 · p. 14

B. Qualitative Examples · C. NOVELTYTHOR

Additional HOLODECK examples illustrate diverse scene types, layout comparisons, rendering options, and prompting to address GPT-4 cultural bias. NOVELTYTHOR comprises 10 human-designed scenes across five categories, supported by tools for integrating assets into Unity.

B. Qualitative Examples

  • 20 additional HOLODECK scenes showcase scene types chosen from the MIT dataset, distinct from examples in the main paper.Figure 23 presents these additional scenes.Source: B. Qualitative Examples · p. 14
  • HOLODECK uses Unity rendering by default for efficiency in Embodied AI applications, while Blender rendering can improve realism.Figure 18 illustrates these rendering options.Source: Figure 18 · p. 15
  • Prompting can address GPT-4’s cultural bias, as illustrated in Figure 19.Source: Figure 19 · p. 15
  • Five methods receive layout comparisons, while residential-scene comparisons feature iTHOR, PROCTHOR, and HOLODECK.Figure 24 compares layouts; Figure 25 compares residential scenes.Source: B. Qualitative Examples · p. 15

C. NOVELTYTHOR

  • 10 human-designed NOVELTYTHOR scenes span five categories, challenging embodied agents with diverse environments and Objaverse assets.Figures 26 and 27 provide top-down views of the scenes.Source: C. NOVELTYTHOR · p. 15
  • Unity prefab generation packages assets and associated resources, supported by cross-platform conversion tools and an interface for artists and designers.The pipeline runs on macOS and Windows and supports assets beyond Objaverse. The interface enables asynchronous asset integration while optimizing storage efficiency.Source: C. NOVELTYTHOR · p. 15

D. Cultural Bias … (MILP) ABSOLUTE RANDOM EDGE

Holodeck’s cultural biases can arise from its language model and 3D asset retrieval; prompt adjustments may help but are unlikely to eliminate them. Supplementary figures provide qualitative comparisons and views of NOVELTYTHOR.

D. Cultural Bias

  • Cultural biases in Holodeck generation can originate in both the language model and the 3D asset retrieval component.Source: D. Cultural Bias · p. 15
  • More detailed prompts or a “no cultural bias” suffix may reduce cultural bias, but qualitative evidence does not suggest complete removal.Users can apply these adjustments directly to control generation.Source: D. Cultural Bias · p. 15
  • Culturally specific prompts such as “Japanese” may disproportionately produce prototypical objects, including Manga posters.Source: D. Cultural Bias · p. 15

(MILP) ABSOLUTE RANDOM EDGE

  • Figure 24 provides a qualitative comparison of different layout methods, without supplying numerical performance results in the provided caption.Source: Figure 24 · p. 19
  • Figure 25 presents qualitative examples across four residential scene types from iTHOR, PROCTHOR, and Holodeck.Source: Figure 25 · p. 20
  • Two instances per scene type are shown in the NOVELTYTHOR top-down views, continued across Figures 26 and 27.Source: Figure 26 · p. 20, Figure 27 · p. 21

Related papers

Loading 2312.09067v2…