Source-linked AI summary
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, Roozbeh Mottaghi
TL;DR
Embodied AI has lacked the massive, scalable interactive training environments that have supported progress in computer vision and language. ProcTHOR procedurally generates diverse physics-enabled houses and uses them to pre-train simple RGB-only agents, achieving state-of-the-art results across six benchmarks with strong zero-shot transfer.
Problem
Embodied AI agents predominantly train on few simulator scenes, causing overfitting and poor generalization to unseen environments.
Method
ProcTHOR procedurally generates arbitrarily large sets of diverse, interactive, physics-enabled houses from room specifications and varied assets, layouts, lighting, and textures.
Results
State-of-the-art results across 6 embodied navigation and manipulation benchmarks were achieved after pre-training simple RGB-only models on 10,000 generated houses, with strong zero-shot results.
Takeaways & Limitations
ProcTHOR provides a scalable platform for training and evaluating embodied agents across navigation, interaction, and manipulation tasks.
Takeaways & Limitations
ProcTHOR-10K uses only 1-floor houses; multi-floor support and expanded asset databases are planned for ProcTHORv2.0.
Abstract
from arXiv · showhide
Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding. This work presents a platform to enable similar success stories in Embodied AI. We propose ProcTHOR, a framework for procedural generation of Embodied AI environments. ProcTHOR enables us to sample arbitrarily large datasets of diverse, interactive, customizable, and performant virtual environments to train and evaluate embodied agents across navigation, interaction, and manipulation tasks. We demonstrate the power and potential of ProcTHOR via a sample of 10,000 generated houses and a simple neural model. Models trained using only RGB images on ProcTHOR, with no explicit mapping and no human task supervision produce state-of-the-art results across 6 embodied AI benchmarks for navigation, rearrangement, and arm manipulation, including the presently running Habitat 2022, AI2-THOR Rearrangement 2022, and RoboTHOR challenges. We also demonstrate strong 0-shot results on these benchmarks, via pre-training on ProcTHOR with no fine-tuning on the downstream benchmark, often beating previous state-of-the-art systems that access the downstream training data.
1 Introduction
Embodied AI has lagged behind other fields because agents train on few simulator scenes, while existing manual and scanned environments are difficult to scale. ProcTHOR addresses this gap by procedurally generating diverse interactive environments and demonstrates strong benchmark performance from simple RGB-only models.
- Motivation: Embodied AI models often overfit limited training scenes and generalize poorly to unseen environments.Compared with computer vision and natural language models trained on massive task-agnostic datasets, E-AI agents predominantly train in simulators with far fewer scenes.
- Motivation: Manual scene design and real-world 3D scanning require substantial specialized effort and are not practical to scale by orders of magnitude.Manual environments require artists to create assets, arrange spaces, and configure textures and lighting; scanned environments require capturing and stitching real structures.
- ProcTHOR: ProcTHOR procedurally generates fully interactive, physics-enabled environments from room specifications.It creates diverse floorplans, populates them with 108 object types and 1633 interactable instances, and varies lighting, colors, and textures.
- Evaluation: 10,000 generated houses formed PROCTHOR-10K, spanning layouts from small 1-room houses to larger 10-room houses.The sample demonstrates ProcTHOR’s ability to produce diverse environments at scale.
- Evaluation: Simple CNN+RNN agents trained only with RGB images achieved state-of-the-art results across six navigation and interaction benchmarks, including strong zero-shot performance.The agents used no depth sensor, explicit mapping, or human task supervision.
2 Related Work
Prior Embodied AI platforms trade off interaction capabilities, realism, and scalability, while large-scale interactive datasets remain scarce. ProcTHOR combines procedural generation with interactive houses to provide arbitrarily many scenes for embodied research.
- Embodied AI platforms: AI2-THOR variants emphasize agent-object interaction, object-state changes, and physics, whereas Habitat primarily provides scanned but non-interactable house scenes.Habitat 2.0 adds object interactions with a stated trade-off.
- Large-scale datasets: Interactive large-scale datasets remain uncommon in Embodied AI research.The passage contrasts their scarcity with large-scale datasets in computer vision, vision-language, 3D understanding, autonomous driving, and manipulation.
- Large-scale datasets: PROCTHOR provides arbitrarily many procedurally generated interactive houses, unlike HM3D’s 1,000 static 3D-scanned scenes.PROCTHOR supports object movement, state changes, lighting and texture variation, physics, and tasks beyond navigation.
- Scene generation: Prior scene-generation work includes text-to-3D synthesis, floorplan generation, generative indoor modeling, object placement, and procedural grid-world construction.PROCTHOR is positioned specifically as a scene-generation framework for Embodied AI research.
3 PROCTHOR
PROCTHOR procedurally generates diverse, customizable, fully interactive Embodied AI environments from room specifications. Its variation spans layouts, assets, materials, placements, lighting, and scene types while preserving realistic structure and interaction.
- Framework: PROCTHOR extends AI2-THOR to procedurally generate fully interactive environments supporting navigation, object manipulation, and multi-agent interaction.It inherits AI2-THOR’s asset library, robotic agents, and physics simulation.
- Generation pipeline: Given a room specification, PROCTHOR uses multi-stage conditional sampling to generate floor plans, structures, lighting, doors, assets, materials, and placements.The pipeline iteratively samples each scene component from the specification.
- Diversity: Floorplan generation combines iterative boundary cutting, recursive room subdivision, and user-defined connectivity constraints to produce natural room layouts.These constraints can encode typical relationships such as bedrooms adjoining bathrooms.
- Diversity: The framework diversifies scenes through 1,633 household assets across 108 categories, randomized materials, semantically constrained placements, and varied lighting.Semantic Asset Groups model objects that typically co-occur, such as dining tables with chairs.
- Interactivity: PROCTHOR supports object state and location changes, arm-based manipulation, and interaction among agents, capabilities absent from static 3D-scan datasets.Examples include changing object states and physically picking and placing objects.
- Customizability and scale: PROCTHOR can generate customized scene types, including classrooms, libraries, and offices, from simple room, asset, material, and lighting specifications.Its procedural design also supports over 100 billion layouts and roughly 20 million unique asset groups.
4 PROCTHOR-10K
PROCTHOR-10K is a procedurally sampled dataset of 10,000 fully interactive houses spanning varied sizes, layouts, and object populations. It is designed for large-scale embodied-agent training with rendering speeds comparable to existing environments.
- Dataset: PROCTHOR-10K contains 10,000 fully interactive, fully navigable houses, with an additional 1,000 validation and 1,000 testing houses.The houses are generated procedurally and allow agents to traverse rooms without interaction.
- Scene statistics: The dataset uses 16 room specifications and includes houses ranging from 1 to 10 rooms, enabling control over scene size and complexity.The reported distributions cover house area and number of rooms.
- Scene statistics: Rooms contain objects from 95 categories, while Semantic Asset Groups provide 19.3 million possible group instantiations.Examples include household and structural objects, plus combinations such as dining tables with four chairs.
- Rendering speed: PROCTHOR provides framerates comparable to iTHOR and RoboTHOR despite generating larger houses.The rendering benchmark covers navigation, interaction, and environment-data queries on servers with 1 or 8 GPUs.
5 Experiments
Experiments evaluate simple RGB-only CNN+GRU models pretrained on ProcTHOR-10K across navigation and manipulation benchmarks, both zero-shot and after fine-tuning. Performance improves with training scale, while zero-shot transfer generalizes across substantially different environments.
- Tasks: The evaluation covers ObjectNav across five environment benchmarks plus ArmPointNav and one-phase Room Rearrangement.ObjectNav targets object categories; ArmPointNav moves an object between specified 3D locations.
- Models and evaluation: All task models use CNN visual encoders and GRUs, with RGB images only and deliberately simple architectures.The experiments exclude depth and other modalities; no explicit mapping or human task supervision is used in the reported setup.
- Zero-shot transfer: Models trained only on ProcTHOR outperform previous state-of-the-art models on 3 benchmarks in zero-shot evaluation.The transfer setting tests generalization to unseen objects and scenes with different appearance and layout statistics.
- Fine-tuning: Fine-tuning with each downstream benchmark’s training data achieves state-of-the-art results on all benchmarks.The model is ranked first on the Habitat 2022 ObjectNav, AI2-THOR Rearrangement 2022, and RoboTHOR ObjectNav public leaderboards as of June 14, 2022.
- Scale ablation: Performance improves as training expands from 10 to 10,000 houses, demonstrating benefits from larger-scale data.The scale ablation trains models on 10, 100, 1,000, and 10,000 houses without material augmentations.
6 Conclusion
ProcTHOR provides arbitrarily large sets of interactive, physics-enabled houses for Embodied AI research. Simple models pretrained on 10,000 generated houses achieve state-of-the-art results across six navigation and manipulation benchmarks, including strong zero-shot transfer.
- ProcTHOR procedurally generates arbitrarily large sets of interactive, physics-enabled houses for Embodied AI research.
Contributions
PROCTHOR combines procedural house generation with structured room specifications, realistic asset placement, and semantic asset groups to create diverse interactive environments.
- House generation: 16 room specifications support weighted sampling of houses ranging from 1 to 10 rooms, including studios, mansions, and intermediate layouts.Room specifications encode room types, relative sizes, and connectivity constraints.
- House generation: Interior boundaries begin as sampled rectangles and gain polygonal complexity through corner cuts whose distribution increases with house room count.The initial dimensions depend on the number of rooms, while individual specifications may override the default sampling distributions.
- House generation: Recursive subdivision converts each sampled interior boundary into rooms according to the room-specification tree and its connectivity constraints.The process first separates major subtrees, then recursively subdivides each zone into the specified rooms.
- Asset placement: Asset samplers position child objects relative to parent anchors, while semantic asset groups encode realistic co-occurring arrangements such as tables with chairs.Placement uses discrete vertical and horizontal anchors, pivots, and offsets.
- Asset placement: PROCTHOR-10K contains 18 semantic asset groups that can produce over 20 million unique asset combinations.Examples include chairs around tables, pillows on beds, and sofas oriented toward televisions.
B.12 Limitations and Future Work
ProcTHOR-10K is currently limited to single-floor houses, with future work targeting multi-floor environments and larger asset databases.
- Limitations and Future Work: ProcTHOR-10K only uses 1-floor houses.The authors plan multi-floor support in ProcTHORv2.0 to capture a wider range of houses and improve fine-tuning results.
- Limitations and Future Work: Future work plans to expand the asset database using open-source collections including ABO, PartNet, ShapeNet, Google Scanned Objects, and CO3D.
C PROCTHOR Datasheet
PROCTHOR provides procedurally generated, interactive houses for embodied AI and computer vision research, with realistic placement annotations and broad reuse potential.
- Data construction: Each house was procedurally generated rather than collected from real-world structures.The generation procedure is described in Appendix A.
- Preprocessing/Cleaning/Labeling: Realistic asset placement used labeling described in Section B.8, and asset pivots were standardized to face consistent directions.
- Uses: The houses support navigation, interaction, manipulation, vision-and-language, audio-visual navigation, and other embodied AI tasks.They are built into AI2-THOR and can support any task performable in that platform.
- Uses: The dataset also supports computer vision studies involving object detection, NeRFs, segmentation, depth, flow estimation, generative modeling, occlusion reasoning, and pose estimation.
- Uses: The JSON-based house-loading framework supports clutter generation and synthetic environments for factories, offices, grocery stores, and procedurally generated cities.
D.1 Datasheet
ARCHITECTHOR provides artist-designed houses for embodied AI evaluation, especially for comparing agents trained on procedurally generated environments.
- Datasheet: The houses were built towards the beginning of 2022.
- Uses: ARCHITECTHOR supports navigation and interaction tasks through integration with the project’s AI2-THOR build.
- Uses: ARCHITECTHOR is intended to evaluate PROCTHOR tasks and agents trained on different sets of procedurally generated houses.
D.2 Analysis
ARCHITECTHOR provides 10 high-quality interactive houses, while PROCTHOR-10K spans navigable-area scales between existing hand-built datasets and ARCHITECTHOR. Creating ARCHITECTHOR required substantial artist effort, whereas ProcTHOR supports multiple image-based modalities.
- ARCHITECTHOR: 10 interactive 3D houses make up ARCHITECTHOR, including five validation houses shown from above and interior views of kitchens and a bedroom.The houses were designed for embodied-AI evaluation.
- Navigable area: ARCHITECTHOR has substantially larger navigable areas than comparable AI2-iTHOR and RoboTHOR scenes.PROCTHOR-10K spans the navigable-area range between those datasets and ARCHITECTHOR.
- Creation effort: 320 hours of professional 3D-artist work were required to create ARCHITECTHOR’s 10 houses.Custom kitchen assets took the longest, followed by modeling the house structures.
- Modalities: ProcTHOR provides RGB, depth, instance segmentation, semantic segmentation, bounding boxes, and surface normals, with additional modalities supported through the Unity backend.The platform can be extended by modifying its backend.
F.1 ObjectNav experiments
ObjectNav agents receive a target object type and egocentric RGB observations, then navigate using a compact recurrent actor-critic policy. Training uses RGB-only visual encoding, target-aware rewards, and reachability-aware sampling on PROCTHOR-10K.
- Task: ObjectNav asks an agent to navigate to a specified target object type using only forward-facing egocentric RGB images.Experiments use a simulated LoCoBot agent.
- Evaluation: Success requires DONE, target distance within 1 meter of the camera, and target visibility in the final frame.Occlusion or looking away from the target makes the episode unsuccessful.
- Evaluation: SPL evaluates trajectory efficiency using the shortest path to any reachable target instance of the requested type.This minimum is used when multiple reachable objects share the target type.
- Model: The model combines frozen CLIP-ResNet visual features with target and previous-action embeddings, then uses a GRU and actor-critic heads to select actions.The standard input is a 3 × 224 × 224 RGB image; ProcTHOR+Large uses RN50x16, a 1024-dimensional GRU state, and 512×384 images.
- Reward: The reward combines progress toward the nearest reachable target, a success reward of 10, and a step penalty of 0.01.The progress term compares the current distance with the best previously observed distance.
- Training: Agents are pretrained on all 10k PROCTHOR-10K training houses for 423 million steps across 16 target object types.Peak performance was reached by 200 million steps, and training took approximately five days on three AWS machines.
F.2 ArmPointNav experiments
ArmPointNav moves a target object from its starting location to a goal using visual input and relative 3D coordinates. Pretraining on PROCTHOR supports evaluation on AI2-THOR, where fine-tuning did not outperform the zero-shot model.
- Task and model: ArmPointNav moves a target object from a starting location to a goal using its relative location in the agent’s coordinate frame.Visual features and embedded 3D target coordinates are combined before recurrent processing.
- Pretraining: Pretraining uses 7,000 houses, 58 object categories, random pickupable objects, and random target locations for 100M frames.Training used 16 GPUs and 192 CPU cores for three days.
- Evaluation: AI2-iTHOR evaluation covers 20 test rooms, 28 object categories, and 528 tasks.The rooms include five kitchens, living rooms, bedrooms, and bathrooms each.
- Evaluation: Fine-tuning on AI2-iTHOR did not produce a model better than the zero-shot model trained with PROCTHOR pretraining.The evaluation used the provided test rooms and task categories.
- Rearrangement: Rearrangement pretraining uses 2,500 one- and two-room PROCTHOR-10K houses with objects shuffled from target poses.Episodes include altered openness for at most one object or altered locations for up to five objects.
G Performance Benchmark
The performance benchmark measures ProcTHOR’s rendering and environment-operation speed for small and large houses. It covers navigation, interaction, and metadata queries using egocentric RGB rendering.
- Benchmark setup: The benchmark partitions environments into small 1–3-room houses and large 7–10-room houses.This size split is used for FPS analysis.
- Benchmark tasks: FPS is measured separately for navigation, interaction, and querying environment metadata.Navigation uses move and rotate actions, interaction uses pushing, and querying retrieves metadata such as agent dimensions.
- Measurement: Each benchmark step renders one 3 × 224 × 224 egocentric RGB image.The table compares PROCTHOR-S and PROCTHOR-L with AI2-iTHOR and RoboTHOR baselines.