ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

PROCTHOR

• PROCTHOR procedurally generates diverse, customizable, physics-enabled environments with varied layouts, interactable assets, object states, lighting, colors, and materials.Its asset library contains 108 object types and 1,633 fully interactable instances, while dynamic material randomization varies appearances at environment load time.Source: §1 · p. 2

ARCHITECTHOR

• ARCHITECTHOR provides 10 artist-designed, fully interactable, realistic houses for test-only household-environment evaluation, extending beyond single-room or less visually diverse benchmarks.The houses are larger, more diverse, and more realistic than AI2-iTHOR and RoboTHOR environments.Source: §1 · p. 2

Results

• PROCTHOR-10K trains simple RGB-only CNN+RNN agents without depth, explicit mapping, or human task supervision, achieving state-of-the-art results across six embodied-AI benchmarks.The benchmarks cover navigation, interaction, rearrangement, and arm manipulation, with both zero-shot and fine-tuned evaluations.Source: §1 · p. 2
• 8.8 points of SPL improvement over the previous RoboTHOR ObjectNav state of the art is achieved with fine-tuning, while zero-shot performance already surpasses that baseline.The previous state of the art uses RoboTHOR training scenes.Source: §1 · p. 2

Scaling

• Scaling from 10 to 100, 1K, and 10K scenes improves performance, with the ablation indicating further gains from generating even larger environments.The framework is designed to scale orders of magnitude beyond the scene counts supported by present-day simulators.Source: §1 · p. 2

Embodied AI platforms

• ProcTHOR procedurally generates diverse interactive scenes, enabling data augmentation and large-scale Embodied AI training unlike platforms with narrower interaction, physics, or scene-generation goals.AI2-THOR and related platforms focus on interactions and physics, Habitat primarily provides non-interactable scanned scenes, and other systems target specialized simulation or manipulation settings.Source: §2 · p. 3

Large-scale datasets

• ProcTHOR provides arbitrarily many interactive houses, unlike scanned datasets that are static, difficult to scale, navigation-limited, or costly to annotate, while avoiding Megaverse’s game-like appearance.Its physics engine supports moving objects and changing states, lighting and textures can vary, and the framework supports tasks beyond navigation.Source: §2 · p. 3

Scene generation

• ProcTHOR generates physics-consistent interactive scenes with commonsense object placement and augmentation of object layouts, appearances, structures, and lighting.The physics-aware generation prevents objects from clipping through one another while supporting controlled variation across scenes.Source: §2 · p. 3

Framework

• ProcTHOR extends AI2-THOR with fully interactive environments supporting navigation, object manipulation, and multi-agent interaction.It inherits AI2-THOR’s large asset library, robotic agents, and accurate physics simulation.Source: §3 · p. 3

Generation procedure

• Multi-stage conditional sampling generates floor plans, walls, lighting, doors, assets, materials, and realistic object placements from room specifications.The procedure iteratively samples scene components and determines appropriate asset placements.Source: §3 · pp. 4–5

Diversity

• PROCTHOR creates diverse environments through natural floor plans, 1,633 assets across 108 categories, varied materials, realistic placements, and variable lighting.Asset placement uses room and location annotations, Semantic Asset Groups, and dependent sampling while preserving navigable floor space; materials are randomized within categories.Source: §3 · pp. 4–6

Interactivity

• Object states and locations can change, enabling arm-based manipulation and agent interaction that static scanned datasets such as HM3D [94] lack.PROCTHOR supports objects with multiple states and parts with rigid-body motions.Source: §3 · pp. 4, 6

Customizability

• PROCTHOR supports customized environments by specifying rooms, assets, materials, and lighting, including classrooms, libraries, and offices.A few simple specification lines can generate environments tailored to particular interests.Source: §3 · p. 6

Scale and Efficiency

• Sixteen scene specifications can yield over 100 billion layouts and roughly 20 million unique asset groups, enabling arbitrarily large, efficient training environments.Scenes are stored as compact JSON specifications, loaded into AI2-THOR at runtime, and generated automatically with high training framerates.Source: §3 · p. 6

Dataset scale

• PROCTHOR-10K contains 10,000 fully interactive houses, plus 1,000 validation and 1,000 testing houses, all navigable without interaction.The dataset was generated in 1 hour on a local workstation with 4 NVIDIA RTX A5000 GPUs, and scaling beyond 10,000 houses is described as straightforward and inexpensive.Source: §4 · p. 7, Figure 11 · p. 7

Scene statistics

• PROCTHOR-10K spans houses with one to ten rooms generated from 16 room specifications, enabling flexible control over scene size and complexity.The dataset covers a wider spectrum than AI2-iTHOR [63], RoboTHOR [27], Gibson [123], and HM3D [94].Source: §4 · p. 7
• Each house includes objects from 95 categories and 18 Semantic Asset groups, whose combinations yield 19.3 million possible group instantiations.Examples include household and structural objects, as well as groups such as a dining table with four chairs or a bed with two pillows.Source: §4 · pp. 7–8

Rendering speed

• PROCTHOR renders at framerates comparable to iTHOR and RoboTHOR despite larger houses, making it fast enough for models trained for hundreds of millions of steps.Experiments used up to 8 NVIDIA Quadro RTX 8000 GPUs and 120 processes; Table 1 reports navigation, interaction, and environment-query FPS for small and large houses.Source: §4 · p. 8, Table 1 · p. 8

Tasks

• The experiments cover ObjectNav in five environments and two manipulation tasks: ArmPointNav and one-phase Room Rearrangement.ArmPointNav moves an object between specified 3D source and destination locations, while Room Rearrangement targets a desired scene state.Source: §5 · p. 8

Models

• Across tasks, models use CNN visual encoders and GRUs, with CLIP-based architectures for ObjectNav and Rearrangement and a three-layer convolutional encoder for ArmPointNav.The architecture is deliberately simple to isolate the benefits of large-scale training; all models are trained with AllenAct.Source: §5 · p. 8

Experimental settings

• Experiments compare zero-shot transfer with fine-tuning on downstream training scenes, using only RGB images without depth or other modalities.Zero-shot tests generalization to new environments, whereas fine-tuning tests whether PROCTHOR representations provide a useful initialization for rapid adaptation.Source: §5 · p. 8

Zero-shot transfer results

• Zero-shot models trained only on PROCTHOR outperform previous state-of-the-art models on three benchmarks while generalizing across unseen environments, objects, and appearance and layout statistics.Zero-shot evaluation uses no training data from the downstream benchmark.Source: §5 · p. 8, Table 2 · p. 9

Fine-tuning results

• Fine-tuning achieves state-of-the-art results on all benchmarks, with first-place rankings on three public leaderboards despite using a simple RGB-only architecture.The leaderboards were Habitat 2022 ObjectNav, AI2-THOR Rearrangement 2022, and RoboTHOR ObjectNav.Source: §5 · pp. 8–9

Scale ablation

• Performance improves as training scale increases from 10 to 10,000 houses, demonstrating the benefit of large-scale data for embodied AI tasks.The scale ablation uses no material augmentations, and each model is trained to 80% success during training.Source: §5 · p. 9, Table 3 · p. 9

6 Conclusion

• ProcTHOR generates arbitrarily large sets of interactive, physics-enabled houses for Embodied AI research.The framework supports procedural generation of environments at scale.Source: §6 · p. 10
• Models pretrained on 10,000 generated houses achieve state-of-the-art results across 6 embodied navigation and manipulation benchmarks, outperforming prior state of the art on 3 in zero-shot evaluation.The reported results use simple models and demonstrate strong 0-shot performance without downstream fine-tuning.Source: §6 · p. 10

Contributions

• The listed contributors covered house generation, Unity and ARCHITECTHOR development, asset and semantic-group tooling, and ObjectNav, ArmPointNav, and rearrangement experiments.Roles included implementing the JSON-to-house Unity backend, standardizing the asset database, and conducting pre-training and fine-tuning experiments.Source: Contributions · p. 16

B.2 Room Specs

• Room specs define room identities, relative sizes, connectivity constraints, and weighted sampling over 16 specifications spanning 1–10 rooms.They support sampling everything from studio apartments to massive mansions.Source: §B.2 · p. 24

B.3 Sampling Floor Plans

• Floor plans sample integer house dimensions, add room-count-dependent corner cuts, recursively subdivide boundaries using room specs, and scale plans by s ∼U(1.6, 2.2).The cut distribution increases with the number of rooms, while recursive subdivision preserves room-spec traversal constraints.Source: §B.3 · pp. 24–26

B.4 Connecting Rooms

• Adjacent rooms use doorways, door frames, or open connections sampled under room-type and room-spec constraints, with exterior doors prioritized for kitchens and living rooms.Open and door-frame connections are restricted to kitchen–living-room pairs, whose probabilities are 0.375, 0.375, and 0.25 for open, frame, and doorway connections.Source: §B.4 · pp. 26–27

B.5 Structure Materials

• ProcTHOR varies house structure through annotated wall and floor materials, shared-material probabilities, ceiling heights sampled between 2.5 and 7 meters, and multiple time-of-day skyboxes.Lighting places directional and room-centered point lights; the database includes 40 solid wall colors, 122 wall textures, 55 floor materials, 16 midday skyboxes, 5 golden-hour skyboxes, and 1 blue-hour skybox.Source: §B.5 · p. 27, §B.6 · p. 28, §B.7 · p. 28

B.8 Object Placement

• Realistic object placement uses room, location, multiplicity, and support annotations across 1,633 interactive assets and 108 object types, while SAGs encode dependent spatial arrangements.SAG samplers randomize compatible assets and relative positions; with 30 choices for four samplers, one group yields over 800k combinations.Source: §B.8 · p. 28, §B.8.1 · pp. 28–29, §B.8.2 · pp. 29–30

B.8.2 Semantic Asset Groups (SAGs)

• PROCTHOR-10K defines 18 semantic asset groups yielding over 20 million unique asset combinations, with relative placement and rejection sampling preventing mesh collisions.SAGs include arrangements such as chairs around tables, pillows on beds, and televisions on stands.Source: §B.8.2 · pp. 30–31

B.8.3 Floor Object Placement

• Floor objects are iteratively placed in sampled room rectangles using semantic and geometric filters, margins, constrained rotations, and bounding-box subtraction to provide broad room coverage.The process independently places objects room by room and can produce rooms with few objects, while semantic asset groups may add further objects.Source: §B.8.3 · pp. 31–33

B.8.4 Wall Object Placement

• Wall placement adds windows, paintings, and televisions while respecting room types, available wall segments, existing floor objects, mounting constraints, and object-specific sampling rules.Windows avoid interior walls, paintings can share segments or sit above short edge objects, and each room permits at most one wall television.Source: §B.8.4 · pp. 33–35

B.8.5 Surface Object Placement

• Surface placement samples receptacle contents using empirical spawn probabilities plus house, receptacle, and object biases, then accepts noncolliding poses with up to five attempts.At most three objects of a type are placed on one receptacle, while biases are ignored for additional instances.Source: §B.8.5 · pp. 35–36

B.9 Material and Color Randomization

• ProcTHOR randomizes vase, statue, and bottle colors with probability 0.8 and object materials per episode with probability 0.8, while preserving semantic material classes.Wall, ceiling, and floor materials remain unchanged, and materials are exchanged only among semantically similar object classes.Source: §B.9 · p. 36

B.10 Object States

• Object-state randomization varies lamps’ on/off states, beds’ cleanliness, and boxes’ and laptops’ openness to expose agents to more diverse training scenes.These state changes include toggling, cleaning or dirtying, and opening or closing objects.Source: §B.10 · pp. 36, 38

B.11 Validator

• The validator uses teleportation and BFS on a 0.25 × 0.25 meter grid to require at least 5 reachable positions in every room, resampling failed houses with the same room specification.This preserves the distribution of sampled room specifications while ensuring navigable access throughout generated houses.Source: §B.11 · p. 38

Limitations and Future Work

• ProcTHOR-10K uses only 1-floor houses; ProcTHORv2.0 will support multi-floor houses and broaden asset databases using open-source resources including ABO [26], PartNet [81], ShapeNet [15], Google Scanned Objects [30], and CO3D [97].The planned extensions aim to capture a wider range of houses and provide better fine-tuning results.Source: §B.12 · p. 38

C PROCTHOR Datasheet

• Each ProcTHOR house was procedurally generated.The generation process is described in Appendix A.Source: C PROCTHOR Datasheet · p. 39
• Asset labeling supports realistic spawning, while standardized pivots ensure consistent asset orientation.Section B.8 describes the labeling, and every asset was checked for consistent pivot direction.Source: C PROCTHOR Datasheet · p. 39
• ProcTHOR houses support a wide variety of interactive tasks in embodied AI and computer vision.They can be used for navigation, multi-agent interaction, rearrangement, manipulation, Sim2Real transfer, vision-and-language, audio-visual navigation, virtual reality interaction, and numerous computer-vision studies.Source: C PROCTHOR Datasheet · p. 40
• The framework’s JSON-based house loading enables research on scene clutter, realistic homes, and synthetic factories, offices, grocery stores, and cities.These extensions support developing procedurally generated environments for training embodied agents.Source: C PROCTHOR Datasheet · p. 40
• The dataset may be used for both commercial and noncommercial purposes.The supplied datasheet passage states no task exclusion beyond this usage statement.Source: C PROCTHOR Datasheet · p. 40
• Table 4 presents a datasheet for ProcTHOR and ProcTHOR-10K.The table is identified as a datasheet following the referenced datasheet framework [40].Source: Table 4 · p. 41

D ARCHITECTHOR

• Figure 35 shows five custom-built interactive validation houses in ARCHITECTHOR for evaluating agents in larger, more realistic home environments.The figure provides top-down images of these houses.Source: Figure 35 · p. 42

• ARCHITECTHOR houses were built toward the beginning of 2022.Source: §D.1 · p. 44
• The houses support any navigation and interaction task performable in the project’s AI2-THOR build.Source: §D.1 · p. 44
• ARCHITECTHOR is especially useful for evaluating PROCTHOR task sets and agents trained on different sets of procedurally generated houses.Source: §D.1 · p. 44

D.2 Analysis

• ARCHITECTHOR comprises 10 remarkably high-quality, large interactive 3D houses, with validation examples spanning kitchens and bedrooms.The validation set contains five houses, and images are shown from two kitchens and a bedroom.Source: §D.2 · p. 45
• ARCHITECTHOR has substantially more navigable area than comparable AI2-iTHOR and RoboTHOR scenes.The comparison uses validation scenes for ARCHITECTHOR and training scenes for AI2-iTHOR, RoboTHOR, and PROCTHOR-10K.Source: §D.2 · p. 45, Figure 37 · p. 46
• PROCTHOR-10K spans the spectrum of navigable areas between AI2-iTHOR, RoboTHOR, and ARCHITECTHOR.Figure 37 presents box plots of navigable-area distributions across the four datasets.Source: §D.2 · p. 45, Figure 37 · p. 46
• Creating ARCHITECTHOR’s 10 houses required approximately 320 cumulative hours of professional 3D-artist work.Custom kitchen assets took the longest, followed by modeling the houses’ 3D structures.Source: §D.2 · p. 46, Figure 38 · p. 46

E Input Modalities

• ProcTHOR provides RGB, depth, instance-segmentation, semantic-segmentation, bounding-box, and surface-normal image modalities.Additional image modalities can be added by modifying the Unity backend.Source: Figure 39 · p. 47

F Experiment details

• The experiment-details section covers baselines, ProcTHOR pre-training, and environment-specific fine-tuning.These details are discussed for ObjectNav, ArmPointNav, and rearrangement tasks.Source: F Experiment details · p. 48

ObjectNav setup

• ObjectNav agents receive forward-facing egocentric RGB images and must navigate to a specified target object using a simulated LoCoBot agent.The task setup follows common embodied-AI practice [27] [7] [58] [96].Source: §F.1 · p. 48

Evaluation

• Success requires issuing DONE with a target object within 1 meter and visible in the final camera frame, while SPL measures trajectory efficiency.For multiple reachable instances, SPL uses the shortest path to any reachable instance of the target type [6] [7].Source: §F.1 · p. 48

Model

• The shared model encodes 224 × 224 RGB observations with frozen RN50 CLIP-ResNet features, combines them with the target embedding and previous action, and processes them with a 512-dimensional GRU actor-critic.The ProcTHOR+Large variant instead uses RN50x16, a 1024-dimensional GRU state, and 512 × 384 inputs.Source: §F.1 · p. 49

ProcTHOR pre-training

• Agents are pre-trained on all 10k PROCTHOR-10K training houses across 16 target object types for 423 million steps, reaching 90% of peak performance by 200 million steps.Target types are sampled only when a successful episode is possible, using reachable and visible objects; reachability is approximated with 0.25 × 0.25 meter-grid BFS and raycasts.Source: §F.1 · pp. 50–51

Benchmark evaluations

• Evaluation covers zero-shot and fine-tuned RoboTHOR, HM3D-Semantics, and AI2-iTHOR, plus zero-shot ARCHITECTHOR evaluation because it provides no training scenes.HM3D-Semantics additionally compares standard ProcTHOR and a larger-house ProcTHOR+Large pre-training variant.Source: §F.1 · p. 51

F.2 ArmPointNav experiments

• ArmPointNav moves a target object to a goal using relative target coordinates, with visual and coordinate features fused into a GRU policy.Three convolutional layers and a linear layer produce a 512-dimensional visual feature, while three linear layers produce a 512-dimensional coordinate embedding; episodes allow up to 200 steps.Source: §F.2 · p. 51
• ArmPointNav pretraining used 100M frames from 7,000 houses and 58 object categories, with random pickupable objects and target locations.Training used 16 GPUs and 192 CPU cores for three days on four AWS g4dn.12xlarge machines.Source: §F.2 · p. 51
• Zero-shot ProcTHOR pretraining outperformed every attempted AI2-iTHOR fine-tuned model on 528 ArmPointNav tasks.Evaluation covered 20 AI2-iTHOR rooms spanning kitchens, living rooms, bedrooms, and bathrooms, with 28 object categories.Source: §F.2 · p. 52

F.3 Rearrangement experiments

• Rearrangement models use imitation learning for the one-phase task, with final training split between ProcTHOR pretraining and AI2-iTHOR fine-tuning.This follows the approach of [115] [58].Source: §F.3 · p. 52
• Rearrangement pretraining used 2,500 one- and two-room PROCTHOR-10K houses with one to five objects shuffled by openness or location.Training comprised 2 · 105 teacher-forcing steps, 2 million dataset-aggregation steps [99], and about 180 million behavior-cloning steps; checkpoints were selected using 200 episodes from 20 unseen validation houses.Source: §F.3 · p. 52
• AI2-iTHOR rearrangement fine-tuning used 4,000 episodes from 80 single-room scenes and staged rollouts of 64, 96, and 128 steps.A 200-episode subset of the provided validation set was used for model selection, and fine-tuning totaled 15 million steps.Source: §F.3 · p. 52
• Table 11 compares ProcTHOR performance benchmarks with AI2-iTHOR and RoboTHOR baselines across navigation, interaction, and environment-querying FPS.The comparison distinguishes small and large ProcTHOR houses as PROCTHOR-S and PROCTHOR-L; the supplied passage does not provide the benchmark values.Source: Table 11 · p. 53

G Performance Benchmark

• The performance benchmark compares ProcTHOR with AI2-iTHOR and RoboTHOR using small and large houses, defined as 1–3 and 7–10 rooms, respectively.Table 11 reports these comparisons, while the supplied passage does not provide its cell values.Source: G Performance Benchmark · p. 53
• Each benchmark step renders one 3 × 224 × 224 egocentric RGB image while executing navigation, interaction, or metadata-query operations.Navigation uses move and rotate actions, interaction uses pushing, and queries retrieve uncommon metadata such as agent dimensions.Source: G Performance Benchmark · p. 53
• Experiments measure throughput on a server with 8 NVIDIA Quadro RTX 8000 GPUs, using 15 processes for single-GPU tests and 120 for eight-GPU tests.The 120 processes are evenly divided across the GPUs.Source: G Performance Benchmark · p. 53

H Broader Impact

• More capable household robots could assist with cooking, cleaning, social interaction, and domestic support for people with disabilities and older adults.The discussion frames these benefits within the goal of improving robotic-agent generalization across tasks.Source: H Broader Impact · p. 53
• Home-assisted robots could provide personalized education, reduce loneliness, offer crisis counseling, and monitor physical activity, sleep, and diet.These examples span social, emotional, educational, and health-related support.Source: H Broader Impact · p. 53
• Robot adoption could increase dependence and isolation, widen care inequities, compromise privacy and security, and enable malicious surveillance or autonomous warfare.The passage emphasizes that affordability, access to sensitive household information, and malicious use create distinct social risks.Source: H Broader Impact · p. 53
• The authors call for reducing social-robot risks while maximizing benefits for the common good.This recommendation follows the identified concerns about dependence, inequity, privacy, surveillance, and warfare.Source: H Broader Impact · p. 53