Source-linked AI summary
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
Santhosh K. Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X. Chang, Manolis Savva, Yili Zhao, Dhruv Batra
TL;DR
Existing real-world indoor 3D datasets are limited in size or reconstruction completeness, constraining available environments for embodied AI research. HM3D provides 1,000 diverse building-scale reconstructions and yields the best PointGoal navigation performance across HM3D, Gibson, and MP3D evaluations.
Problem
Real-world indoor 3D datasets remain limited in total size or suffer from incomplete reconstructions, restricting available building-scale environments for embodied AI research.
Method
HM3D constructs 1,000 diverse building-scale textured reconstructions from real-world interiors scanned with Matterport Pro2 sensors.
Results
HM3D-trained PointGoal agents achieve the best performance across Gibson, MP3D, and HM3D test sets.
Takeaways & Limitations
HM3D demonstrates value as a dataset for embodied AI by supporting agents that match or outperform those trained on other datasets across evaluations.
Takeaways & Limitations
HM3D supports only geometric tasks in static environments and lacks semantic annotations.
Abstract
from arXiv · showhide
We present the Habitat-Matterport 3D (HM3D) dataset. HM3D is a large-scale dataset of 1,000 building-scale 3D reconstructions from a diverse set of real-world locations. Each scene in the dataset consists of a textured 3D mesh reconstruction of interiors such as multi-floor residences, stores, and other private indoor spaces. HM3D surpasses existing datasets available for academic research in terms of physical scale, completeness of the reconstruction, and visual fidelity. HM3D contains 112.5k m^2 of navigable space, which is 1.4 - 3.7x larger than other building-scale datasets such as MP3D and Gibson. When compared to existing photorealistic 3D datasets such as Replica, MP3D, Gibson, and ScanNet, images rendered from HM3D have 20 - 85% higher visual fidelity w.r.t. counterpart images captured with real cameras, and HM3D meshes have 34 - 91% fewer artifacts due to incomplete surface reconstruction. The increased scale, fidelity, and diversity of HM3D directly impacts the performance of embodied AI agents trained using it. In fact, we find that HM3D is `pareto optimal' in the following sense -- agents trained to perform PointGoal navigation on HM3D achieve the highest performance regardless of whether they are evaluated on HM3D, Gibson, or MP3D. No similar claim can be made about training on other datasets. HM3D-trained PointNav agents achieve 100% performance on Gibson-test dataset, suggesting that it might be time to retire that episode dataset.
1 Introduction · 2 Related Work
HM3D addresses the limited availability and incompleteness of real-world indoor 3D datasets with 1,000 near-complete, high-fidelity building-scale reconstructions spanning substantial navigable space. The related work contrasts HM3D with synthetic, reconstruction, floorplan, and panorama-based datasets that differ in scale, realism, or environmental detail.
- 1 Introduction: Real-world indoor 3D datasets remain scarce: earlier efforts often reconstruct room regions or individual rooms, while building-scale datasets are limited in size or incomplete.The passage cites SceneNN, ScanNet, BuildingParser, Matterport3D, and Gibson as examples.
- 1 Introduction: HM3D provides 1,000 near-complete high-fidelity reconstructions of entire buildings, covering more than 10,600 rooms across approximately 1,920 building floors.Its total navigable area is 112.5k m2.
- 1 Introduction: 1.4 - 3.7× larger navigable area distinguishes HM3D from prior real-world scanned indoor datasets and supports embodied AI tasks such as navigation.HM3D scenes typically represent complete buildings, including multi-floor private residences.
- 1 Introduction: 34 - 91% reduction in reconstruction artifacts reflects HM3D’s greater completeness relative to prior real-world scanned indoor datasets.The supplied passage identifies reduced incompleteness as a key characteristic, but does not provide the omitted artifact comparison details.
- 1 Introduction: HM3D is evaluated through comparisons of rendered images, reconstruction artifacts, and PointGoal navigation performance against other 3D scan datasets.These analyses examine visual fidelity, surface-reconstruction holes, and embodied-agent performance.
- 2 Related Work: 3D datasets are categorized as synthetic/CAD-based, 3D reconstruction or mesh-based, floorplan-based, and panorama-based.These categories organize the related-work comparison.
- 2 Related Work: Synthetic scenes can support embodied-AI simulation but often simplify room connectivity and fail to reflect real-world architectural variety, object arrangement, and clutter.The passage notes that some scenes are isolated rooms connected via a magic portal, while authored building-scale scenes remain limited in realism.
- 2 Related Work: Floorplan datasets simplify architectural layout and omit textured environments, while reconstruction datasets remain limited in scale and panorama-based datasets form a separate category.The reconstruction discussion notes that Gibson is the largest building-level reconstruction dataset with 571 scenes.
3 Dataset
HM3D is a globally diverse collection of 1,000 large-scale, multi-floor indoor 3D reconstructions curated for quality and broad architectural variation. It exceeds prior datasets in physical scale, navigable space, scene clutter, reconstruction completeness, and rendered-image fidelity.
- Dataset composition: HM3D contains 1,000 Matterport Pro2-scanned 3D reconstructions from 38 countries and 181 geographic regions, including residences, stores, and other private indoor spaces.RGB-D alignment, meshing, and texturing used Matterport’s reconstruction pipeline.
- Dataset composition: The 1,000 scenes were selected through two-stage annotation and verification, with 15 volunteers rating reconstruction quality, furnishing, closed doors, and interactive potential.Annotators used a 1–5 quality scale and visually inspected reconstruction artifacts.
- Scale and complexity: 1.7× higher floor area and 1.4× higher navigable area distinguish HM3D from Gibson; against Gibson 4+, the increases reach 20× and 15.6×, respectively.HM3D’s scene clutter also exceeds most other datasets by approximately 1.2×, except the smaller RoboTHOR dataset.
- Reconstruction quality: 560 HM3D scenes versus 175 Gibson scenes have less than 5% of views exhibiting reconstruction artifacts, giving HM3D more than three times as many low-defect scenes.HM3D scenes overall show fewer artifacts, while Gibson and MP3D have broader defect distributions.
- Visual fidelity: HM3D-rendered images achieve the lowest FID/KID scores against MP3D real images at 20.53/15.78 and Gibson real images at 20.49/12.76.The comparison uses RGB renderings from all scene parts and real RGB images from high-resolution panoramas.
4 Experiments
Experiments evaluate PointGoal navigation agents trained on HM3D against agents trained on Gibson, MP3D, and Gibson 4+, testing convergence, generalization, scene diversity, and dataset scaling. HM3D-trained agents converge faster, generalize better across datasets and episode difficulties, and achieve perfect Gibson-test success for both RGB and depth inputs.
- PointNav performance: 100% navigation success is achieved by HM3D agents with both RGB and depth inputs on Gibson test.Table 2 reports mean and standard deviation across evaluation seeds, with HM3D agents reaching perfect Gibson-test success for both sensors.
- PointNav performance: HM3D agents converge faster and outperform corresponding agents trained on other datasets across validation settings.The HM3D agent closely follows Gibson on Gibson validation while outperforming it on MP3D and HM3D validation.
- Generalization: 3 SPL points higher performance is achieved by HM3D agents than the next-best method on MP3D test for both RGB and depth inputs.The agents had not observed Gibson or MP3D during training, indicating cross-dataset generalization.
- Dataset scaling: PointNav performance scales nearly linearly with the total navigable area in training scans.This experiment examines whether scaling up large, high-fidelity 3D datasets benefits embodied navigation.
- Generalization: HM3D agents adapt better than other agents as MP3D-test episode difficulty increases.Difficulty is measured by geodesic distance between start and goal positions, and HM3D layout diversity supports generalization to harder episodes.
5 Conclusion
HM3D is a 1,000-scene dataset of high-quality, building-scale interior reconstructions that improves on existing datasets in spatial scale, reconstruction completeness, and visual fidelity. PointGoal agents trained on HM3D match or outperform agents trained on other datasets across evaluations, while future semantic and physical annotations could support additional embodied AI tasks.
- Dataset contribution: HM3D comprises 1,000 building-scale reconstructions from real-world interiors for academic research.The paper describes HM3D as the largest dataset of high-quality 3D interior reconstructions for academic research, to the authors’ knowledge.
- Dataset contribution: HM3D improves on existing 3D reconstruction datasets through larger spatial scale, improved reconstruction completeness, and higher visual fidelity.These advantages were established through quantitative analyses.
- Embodied AI impact: Agents trained on HM3D match or outperform agents trained on other datasets when evaluated on HM3D or other datasets.The experiments used PointGoal navigation for embodied AI agents.
- Future work: Extending HM3D with object semantics and physical attributes could enable additional embodied AI tasks such as ObjectGoal.The passage presents these extensions as future work.
7 Licenses for referenced datasets · A1 Limitations of Habitat-Matterport 3D
HM3D’s coverage is constrained by Matterport Pro2 access, internet requirements, and availability across only 38 countries. The dataset currently supports only geometric tasks in static environments and lacks semantic annotations.
- A1 Limitations of Habitat-Matterport 3D: 38 countries currently provide the geographic coverage for HM3D scans.This limited coverage constrains the regions and neighborhoods represented in the dataset.
- A1 Limitations of Habitat-Matterport 3D: ∼3,000$ Matterport Pro2 sensors restrict participation to building-owners who can afford the acquisition cost.The sensor cost is one factor limiting which regions and neighborhoods can be included.
- A1 Limitations of Habitat-Matterport 3D: Internet access is required for building-owners to upload Matterport scans to the cloud.This requirement further narrows potential data sources.
- A1 Limitations of Habitat-Matterport 3D: Matterport Pro2 availability excludes regions where the sensor cannot be purchased.Together with affordability and internet requirements, this limits the types of represented regions and neighborhoods.
- A1 Limitations of Habitat-Matterport 3D: Only geometric tasks are supported, and environments are static rather than unchanging during interaction.The dataset does not currently support changing-world dynamics.
- A1 Limitations of Habitat-Matterport 3D: Semantic annotations are absent, limiting support for high-level understanding tasks such as object retrieval.The authors plan to augment HM3D with semantic annotations and study dynamic environments involving freely moving people and pets.
A2 Hyperparameters for PointNav experiments
PointNav experiments use Habitat Lab’s public DD-PPO implementation with Wijmans et al.’s hyperparameters. The agent uses a ResNet-50 backbone with GroupNorm and a two-layer, 512-D LSTM.
- Implementation and architecture: The experiments use Habitat Lab’s publicly available DD-PPO implementation and the same hyperparameters as Wijmans et al.These choices define the training setup for the PointNav experiments.
- Implementation and architecture: The policy uses a ResNet-50 backbone and a two-layer LSTM with 512-D hidden states.BatchNorm layers in ResNet-50 are replaced with GroupNorm layers.
- Optimization hyperparameters: 0.2 is the PPO clip parameter, with 2 PPO epochs, 2 mini-batches, and a value loss coefficient of 0.5.The entropy coefficient is 0.01 and the learning rate is 0.00025.
A3 Computational requirements
PointNav experiments were the most computationally expensive, using distributed training across 8 nodes with 32 Volta GPUs. Training took 2–3 days with depth inputs and 4–5 days with RGB inputs.
- A3 Computational requirements: PointNav experiments were the most computationally expensive of the experiments.The experiments were run in a distributed fashion on an internal cluster.
- A3 Computational requirements: 8 nodes with 4 GPUs per node supported the distributed experiments.The setup used 32 Volta GPUs with 16/32 GB each.
- A3 Computational requirements: 2–3 days were required to train an agent with depth inputs, compared with 4–5 days using RGB inputs.Training duration depended on the input modality.
A5 Habitat-Matterport 3D dataset collection process
HM3D’s 1,000 scans were collected by Matterport with the Habitat team through explicit user contributions for open-source Embodied AI research. Contributors granted academic or non-commercial use permissions, after which Matterport attempted to obscure personally identifiable information and used human reviewers to inspect scans.
- Collection: 1,000 scans were collected by Matterport Inc. in collaboration with Facebook AI Research’s Habitat team.Matterport contacted users directly to request scans for open-sourced Embodied AI research.
- Collection: Users contributed their Matterport scans after being explicitly asked to support open-source Embodied AI research.The outreach framed robots’ need to recognize and navigate multiple 3D environments.
- Consent: Contributors agreed to let Matterport use designated spaces and related imagery for academic and/or non-commercial purposes without payment.Contributors affirmed that they held the necessary rights, consents, and permissions.
- Privacy protection: Matterport attempted to obscure personally identifiable information, including faces, documents, addresses, contact details, and license plates.Human reviewers previewed images from every scanned location to check for such information.
A6 PointNav validation results · A7 Comparing PointNav episode datasets · A8 Dataset characteristics that impact PointNav performance
Across validation and episode-dataset analyses, HM3D-trained PointNav agents generally outperform other agents, while dataset scale and episode characteristics strongly influence performance. HM3D also provides the largest validation gains across sensors and achieves perfect Gibson validation success.
- A6 PointNav validation results: HM3D agents match Gibson on Gibson validation with RGB inputs and outperform other agents in all remaining cases, especially with RGB.These results use the final validation performance of the best checkpoint.
- A7 Comparing PointNav episode datasets: MP3D has the hardest validation episodes, whereas Gibson has the easiest.Episode difficulty is compared across Gibson, MP3D, and HM3D validation datasets.
- A8 Dataset characteristics that impact PointNav performance: The analysis evaluates how dataset characteristics affect PointNav performance, extending prior comparisons of navigable area, visual fidelity, and reconstruction quality.It examines the relationship between training-dataset characteristics and PointNav results.
- A6 PointNav validation results: 100% navigation success is achieved by HM3D agents with both depth and RGB sensors on Gibson validation.Table A1 reports mean and standard deviation from one training seed evaluated on three seeds.
- A7 Comparing PointNav episode datasets: Larger geodesic distances between start and goal positions generally indicate harder PointNav episodes.Figure A2 groups validation episodes by geodesic start-goal distance.
- A8 Dataset characteristics that impact PointNav performance: EMD(train, val) measures train-validation dissimilarity by comparing normalized geodesic-distance histograms with Earth Mover’s Distance.The histograms are computed independently for the train and validation episode splits.
- A8 Dataset characteristics that impact PointNav performance: 0.82 to 0.97 is the reported Pearson correlation range between navigable area and PointNav performance, indicating that large-scale training datasets are critical.Correlations are computed between training-dataset characteristics and navigation SPL on validation sets.
- A8 Dataset characteristics that impact PointNav performance: EMD (train, val), KID (mean), % defects, and navigable area are evaluated as dataset characteristics related to PointNav performance.The table defines these as episode-difficulty mismatch, image visual fidelity, reconstruction completeness, and total navigable training-scan area, respectively.
A9 Example scenes from Habitat-Matterport 3D
HM3D examples illustrate multi-floor residences and diverse indoor scenes across a wide range of building sizes and environments. The appendix visualizes representative views and makes all 900 training and validation scenes available online.
- Dataset visualization: 900 HM3D scenes from the train and val splits can be visualized on the dataset website.The examples extend beyond the ten appendix visualizations to cover all scenes in these splits.
- Example residences: Five example residences show multi-floor HM3D buildings spanning a wide range of sizes through top-down, cross-section, and egocentric views.The egocentric views are taken from navigable positions in each scene.
- Diverse scenes: Five diverse examples include restaurants, offices, a nightclub, an art studio, and gyms, presented through top-down, cross-section, and egocentric views.The views depict navigable positions within each scene.
A10 PointNav qualitative results
The appendix presents qualitative PointNav episodes for HM3D agents, organized by navigation difficulty and SPL performance. Separate examples illustrate agents using depth and RGB sensors, with trajectories compared against shortest-path navigation.
- Episode organization: Sample HM3D PointNav episodes are categorized by geodesic distance between start and goal and by agent SPL performance.The qualitative presentation follows a format similar to Wijmans et al. [29].
- Depth sensor: Depth-sensor examples show HM3D-agent trajectories across geodesic-distance and SPL categories, alongside navigable maps and shortest-path routes.The agent starts at the blue square, the goal is the red square, the shortest path is green, and the agent trajectory is blue.
- RGB sensor: RGB-sensor examples show HM3D-agent trajectories across geodesic-distance and SPL categories, alongside navigable maps and shortest-path routes.The agent starts at the blue square, the goal is the red square, the shortest path is green, and the agent trajectory is blue.