InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layouts

1. Introduction

• Existing real scans, designer-created scenes, and procedural scenes respectively face simulation incompatibility, sanitized layouts and collisions, or generation costs and limited diversity; interactive objects are also often missing.These gaps leave existing datasets short of the diversity, realism, and interactivity sought for embodied-agent skills.Source: §1 · p. 2
• InternScenes integrates real-world scans, procedurally generated scenes, and designer-created scenes into a diverse, simulation-ready indoor dataset.Its real-to-sim pipeline replaces scanned objects with Objaverse assets while preserving small objects annotated in EmbodiedScan.Source: §1 · p. 2
• Approximately 40,000 scenes span 48k regions, 15 common types, 1.96M objects, and 288 indoor object classes.The dataset also includes 800k CAD models.Source: §1 · p. 2
• 41.5 objects per region on average; physical simulation addresses object collisions and supports complex, realistic layouts.The paper also describes incorporating interactive objects into the scenes.Source: §1 · p. 3
• Layout-generation methods perform unsatisfactorily on the full version, while complex, cluttered scenes challenge prior navigation policies; more episodes from diverse assets improve policy generalization.The benchmark includes full and simplified layout-generation versions, with the latter removing small objects.Source: §1 · p. 3

Real-world Scans of Indoor 3D Scenes

• Real-world scans preserve scene information but acquisition constraints introduce noise that challenges reconstruction and annotation; richer annotations still leave simulation gaps and make real-to-sim construction labor-intensive.ScanNet++ uses higher-precision equipment, while EmbodiedScan adds small-object annotations, 288 categories, and 9DoF boxes.Source: §2 · pp. 3–4

Simulated Indoor Scenes

• Synthetic-scene research offers multimodal data and rule-based generation, but original assets may be inaccessible, generation can be costly, and generated scenes may lack diversity.Hypersim’s asset-purchase pipeline is described as costing approximately $57K; 3D-FRONT is described as missing small items and often having fewer objects.Source: §2 · p. 4

Real-to-Sim 3D Scene Generation

• Real-to-sim work aligns CAD models to scanned layouts, while image-conditioned methods can produce artifacts or struggle with complex scenes containing many small objects.OpenRooms uses ScanNet point clouds and Scan2CAD; MetaScenes is limited to one data source, and ACDC struggles with scenes populated by numerous small objects.Source: §2 · p. 4

3. Dataset

• The two-stage dataset pipeline integrates and cleans multi-source scenes to extract layouts, then places objects using those layouts and applies optimization and physics simulation.A statistical analysis follows to characterize dataset quality and advantages.Source: §3 · p. 5

3.1. Multi-Source Data Processing

• For Real2Sim, EmbodiedScan’s region and small-object annotations support retrieval from real scans, and similar small regions are merged so each resulting region contains at least 8 objects.Asset labels are mapped and verified, orientation constraints are corrected, and candidate selection with fuzzy label replacement aligns assets to real-world arrangements.Source: §3.1 · p. 5
• InternScenes-Gen extracts layouts from Infinigen Indoors, whose constraint-based solver generates scenes under composition rules for symmetry, spatial relations, quantity, physics, and accessibility.The generated settings include detailed arrangements such as dining tables and cabinet interiors.Source: §3.1 · p. 5
• For designer-created scenes, manual region and instance annotation addresses unclear region boundaries and inconsistent object hierarchies through instance splitting or merging.The processed objects are labeled and verified, then coordinates, bounding-box dimensions, and rotations are extracted as layout information.Source: §3.1 · pp. 6–7

3.2. Physics-Aware Scene Composition

• Bounding-box fine-tuning optimizes large furniture for overlap, floor alignment, and position regularity, then SAPIEN physics simulation adjusts the layout to reduce collisions and floating among smaller objects.Smaller assets are bound to nearby larger objects to preserve relative positions during fine-tuning.Source: §3.2 · pp. 7–8

• Region layouts provide object categories, model names, centers, bounding-box dimensions, and ZXY Euler angles; mesh assets enable 3D reconstruction.Examples are shown for Real2Sim, procedurally generated, and designer-created scenes.Source: §4 · p. 8
• 39,870 scenes and 48,381 regions across 15 categories contain 1.96 million objects from 288 categories, averaging 41.5 objects per region.The three subsets are InternScenes-Real2Sim, InternScenes-Gen, and InternScenes-Synthetic.Source: §4 · p. 8
• Object bounding-box volumes are summarized overall and for chair, bed, couch, bottle, and book, selected to span large furniture to small items.Figure 9 presents the overall and representative-category volume distributions.Source: §4 · p. 9, Figure 9 · p. 10
• The dataset reports distributions across 15 region types and region areas; chairs, toys, books, lights, and bottles are the five most frequent object categories.Figure 11 encodes category frequency within regions using rectangle size and color depth.Source: §4 · pp. 9, 11, Figure 7 · p. 10, Figure 11 · p. 11
• The average object density is 1.296 objects per square meter, and Figure 10 shows density distributions across regions.Source: §4 · p. 11, Figure 10 · p. 11
• Each selected container or support object relates to an average of 3.45 other objects, rising to 5.57 when objects without such relations are excluded.The statistics count smaller objects inside or on top of selected common indoor object categories.Source: §4 · p. 12

5. Experiment

• The benchmarks examine scene generation and point-goal navigation as applications of InternScenes, with realistic, diverse layouts posing challenges in both tasks.Source: §5 · p. 12, §5.1 · p. 12, §5.2 · p. 13

5.1. Interior Scene Generation

• Generation experiments use Resting, Living, and Dining regions in full and small-object-removed versions, evaluating three unconditional baselines with four generation metrics.The baselines are ATISS, DiffuScene, and PhyScene; evaluation uses FID, KID, SCA, and CKL.Source: §5.1 · p. 12
• On full InternScenes scenes, all three generation baselines decline, and PhyScene has a pronounced advantage over DiffuScene; all struggle to model complex small-object distributions.DiffuScene and PhyScene are largely comparable across most metrics on the simplified version.Source: §5.1 · pp. 12–13, Table 2 · p. 13

5.2. Navigation

• Navigation uses an IsaacSim benchmark with a ClearPath Dingo, 20 Real2Sim scenes, and 10 Gen scenes, evaluating Success Rate and SPL.Three baselines include DD-PPO, pretrained NavDP, and fine-tuned NavDP; the fine-tuning data contains 118,784 trajectories.Source: §5.2 · pp. 13–14
• NavDP reaches approximately 50% success, and fine-tuning with InternScenes trajectories slightly improves overall performance; scaling model capacity for larger datasets remains open.The benchmark reports navigation metrics in Table 3.Source: §5.2 · p. 14, Table 3 · p. 14
• The navigation benchmark identifies clutter requiring precise planning and collision recovery, narrow pathways dependent on embodiment information, and small obstacle components that challenge perception.The authors identify these as challenges for navigation methods in their benchmark.Source: §5.2 · p. 14

6. Limitations and Conclusion

• InternScenes comprises 40,000 scenes and over 1.96 million objects from 288 classes, supporting layout-generation and visual-navigation benchmarks.The authors also commit to open-sourcing the dataset and tools.Source: §6 · p. 14
• The current pipeline relies on manual annotation, and the authors identify scene diversity as an area for improvement.Future work aims to reduce human involvement and improve the 3D asset library.Source: §6 · p. 14

A.1. Retrieval Details of InternScenes-Real2Sim

• Ambiguous object labels are refined using the object's position and the semantics of neighboring objects to infer a more specific category.For example, an object on a desk may be refined to a book, plant, or lamp, narrowing the candidate assets.Source: §A.1 · p. 18
• Couch models are grouped as left-L, right-L, or standard, with mirror transformations used to expand the available L-shaped candidates.Source: §A.1 · pp. 18–19
• Candidate assets are selected by normalized bounding-box similarity to reduce scale-stretch distortion, then transformed to match the original object's size, translation, and rotation.Source: §A.1 · p. 19

A.2. Annotation Details of InternScenes-Synthetic

• Annotation combines a BEV map with rendered views from eight directions around each floor sampling point to support scene perception and region labeling.The BEV map is generated from scene point clouds, while rendered views are captured at 45-degree increments.Source: §A.2 · pp. 19, 21, Figure 13 · p. 20
• Instance annotation supports splitting or merging scene elements, with labels for complete objects, assemblies, partial objects, and room structures.Selecting an element highlights it and adjusts the camera to focus on its location.Source: Figure 14 · p. 20, §A.2 · p. 21
• Semantic labels are assigned automatically from six rendered views of each processed instance using InternVL.Random inspection covered 2,550 objects across 50 scenes; automatic captioning accuracy exceeded 85%.Source: §A.2 · pp. 21, 23, Table 5 · p. 22

A.3. Details of Physics-Aware Scene Composition

• Large-furniture optimization penalizes overlaps, aligns furniture bottoms with the ground, and limits movement from annotated positions.The loss combines IoU, ground-alignment, and regularization terms.Source: §A.3 · p. 23
• Small-object physics simulation addresses floating and interpenetration artifacts that bounding-box optimization cannot effectively resolve.Cavity-containing furniture is segmented before convex collision decomposition so internal cavities are represented in the collision geometry.Source: §A.3 · p. 24

B. Experiments

• The supplementary section covers details of the layout-generation and navigation experiments.Source: B. Experiments · p. 24

B.1. Interior Scene Generation

• Experiments use resting, living, and dining regions in full scenes or a simplified version retaining 45 large-furniture categories.Source: §B.1 · p. 24
• Unconditional generation is evaluated with ATISS, DiffuScene, and PhyScene under shared network architectures, training hyperparameters, and experimental setup.The object-retrieval and rendering pipelines are also held identical across methods.Source: §B.1 · pp. 24–25
• Full-version baselines produce floating or interpenetrating small objects, while simplified-version generations show more reasonable large-furniture layouts.The authors suggest limited contextual modeling capacity may make it difficult for current models to capture layouts with many objects.Source: §B.1 · p. 25

B.2. Navigation

• Navigation episodes use randomly sampled start–destination pairs 3–10 m apart in floor regions with ESDF values above 0.5 m.The evaluation uses a ClearPath Dingo in Isaac Sim and controls its wheel speeds rather than teleporting it to predicted poses.Source: §B.2 · p. 25
• The navigation-scene figure identifies example evaluation scenes, while the two layout-generation figures distinguish simplified- and full-dataset examples.Their captions do not report navigation results or comparisons.Source: Figure 17 · p. 26, Figure 18 · p. 27, Figure 19 · p. 28
• Tables 6–8 are captioned as reporting FPS, CPU usage and memory, and GPU memory across parallel-simulation levels.The supplied captions give no numerical values.Source: Table 6 · p. 28, Table 7 · p. 28, Table 8 · p. 28

C. System Performance and Resource Requirements

• The evaluation reports GPU memory, CPU usage and memory, and rendering throughput (FPS) for scenes from all three dataset subsets.Experiments use Isaac Sim in headless mode with one GPU, 16 CPU cores, and 128 GB RAM on a node with 128 vCPUs, 1024 GB DDR5 RAM, and eight RTX 4090 GPUs.Source: C. System Performance and Resource… · p. 29
• Measurements run for 2,000 steps, discard the first 200 as warm-up, and compute FPS and memory footprints over the remaining steps.All scenes use the Stage Light environment.Source: C. System Performance and Resource… · p. 29
• Parallel-simulation tests incrementally load scenes from each data source into one Isaac Sim World and monitor performance as environment count increases.The passage refers to Tables 6–8 for quantitative results but supplies no values.Source: C. System Performance and Resource… · p. 29

D. Discussion on Procedural Generation with Infinigen Indoor

• Infinigen Indoors uses randomized asset generation and constraint-based layout optimization with simulated annealing to mitigate bias in spatial configurations and object co-occurrence.Furniture alone has 17 generators and 216 controllable parameters, reducing reliance on a static model set.Source: D. Discussion on Procedural Generation… · p. 29
• Simulated annealing searches a large layout state space to avoid templated or repetitive scenes, accepting improvements and sometimes accepting worse states.The acceptance probability for worse states changes as temperature cools from 0.25 to 0.001, supporting broader early exploration and later convergence.Source: D. Discussion on Procedural Generation… · pp. 29–30
• The authors state that this optimization yields diverse, high-quality spatial configurations while suppressing generated bias.Source: D. Discussion on Procedural Generation… · p. 30

E. Support for Articulated Objects

• Interactive-scene construction uses URDF assets from PartNet-Mobility, whose quality and annotation accuracy are described as community-validated.Source: E. Support for Articulated Objects · p. 30
• A programmatic Isaac Sim pipeline imports URDF assets as physics-enabled articulations, preserving their kinematic structure in the final USD scene.The object pose is then set in the world.Source: E. Support for Articulated Objects · p. 30