Source-linked AI summary
Habitat-Matterport 3D Semantics Dataset
Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Theo Gervet, John Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, Alexander William Clegg, Devendra Singh Chaplot
TL;DR
Embodied-AI research needs dense semantic annotations of realistic indoor spaces, but acquiring and labeling such data at scale is difficult. HM3DSEM supplies a large texture-based annotation layer for HM3D and yields stronger ObjectGoal navigation generalization across datasets. Its annotations remain subject to residual errors despite extensive verification.
Problem
Dense semantic annotation of acquired real-world spaces remains laborious and challenging, limiting the availability of large annotated datasets for embodied-AI research.
Method
HM3DSEM augments HM3D with texture-encoded object-instance and room semantics that correspond pixel-accurately to RGB surface textures and underlying geometry.
Results
HM3DSEM contains 142,646 object instances across 216 spaces and 3,100 rooms, and its trained policies achieve higher cross-dataset ObjectGoal generalization.
Takeaways & Limitations
The results support improving the quality and scale of dense semantic annotations for downstream embodied-AI task performance.
Takeaways & Limitations
Some annotation errors may remain despite iterative correction, author verification, and automated checks.
Abstract
from arXiv · showhide
We present the Habitat-Matterport 3D Semantics (HM3DSEM) dataset. HM3DSEM is the largest dataset of 3D real-world spaces with densely annotated semantics that is currently available to the academic community. It consists of 142,646 object instance annotations across 216 3D spaces and 3,100 rooms within those spaces. The scale, quality, and diversity of object annotations far exceed those of prior datasets. A key difference setting apart HM3DSEM from other datasets is the use of texture information to annotate pixel-accurate object boundaries. We demonstrate the effectiveness of HM3DSEM dataset for the Object Goal Navigation task using different methods. Policies trained using HM3DSEM perform outperform those trained on prior datasets. Introduction of HM3DSEM in the Habitat ObjectNav Challenge lead to an increase in participation from 400 submissions in 2021 to 1022 submissions in 2022.
1. Introduction
HM3DSEM addresses the difficulty of densely annotating real-world spaces at scale by adding a texture-based semantic layer to HM3D. Its large, diverse annotations support stronger ObjectGoal navigation generalization across datasets.
- Dense semantic annotation of real-world spaces remains laborious and challenging, despite accelerating embodied-AI research.HM3D required over 800 hours mainly for curation and verification of 3D reconstructions.
- HM3DSEM adds a dense semantic layer to HM3D, encoding object instances and rooms through textures.The layer covers architectural elements, large objects, and aggregated “stuff” categories.
- Texture-based semantics provide pixel-accurate correspondences between object labels, RGB surface textures, and underlying geometry.
- 142,646 object instances span 216 spaces and 3,100 rooms, exceeding Matterport3D by 2.8x and ARKitScenes by 2.1x.
- HM3DSEM-trained ObjectGoal policies achieve higher cross-dataset generalization and outperform policies trained on the datasets themselves on average.
2. Related Work
Prior semantic 3D datasets often focus on room-scale scenes or use coarse, mesh-based labels, while synthetic alternatives face scale and realism constraints. HM3DSEM instead targets detailed semantic annotation of diverse real-world spaces.
- Collecting, reconstructing, and annotating real-world 3D spaces at scale requires complex pipelines and annotation tools.
- Earlier real-world datasets commonly focused on room-scale reconstructions with object semantics derived from 3D mesh segments.
- ARKitScenes provides 1,661 room-scale scans but only bounding-box annotations for object instances.
- Matterport3D and related datasets use mesh-segment or back-projected labels, while Replica offers vertex-level labels for only 18 scenes.
- Synthetic datasets are often small and room-scale, and synthetic content can retain modeling biases and sim-to-real discrepancies.
3. Dataset Details
HM3DSEM provides large-scale, human-verified semantic annotations for scanned indoor scenes, using texture-based instance labels aligned with original geometry. Its dataset statistics capture diverse objects, regions, and room types, while iterative verification supports continued correction.
- Data format and contents: Texture images applied to the original HM3D geometry encode unique instance colors, raw class names, and region IDs for room-based grouping.The archival assets are packed in binary glTF format with accompanying metadata text files.
- Annotation representation: Texture-based annotations preserve pixel-accurate boundaries on the original mesh and avoid common geometry-induced misalignment and jagged-boundary errors.The approach addresses limitations of per-vertex semantic labels and avoids requiring a higher-resolution replacement mesh.
- Annotation scope: Annotators label architectural features, furniture, appliances, clutter, and smaller decorative items, segmenting individual objects whenever scan resolution permits.Examples include pillows and blankets separated from furniture, and clothing items, linens, and books separated in storage areas.
- Dataset statistics: 1,625 category tags label the dataset’s 142,646 instances across 3,100 regions, with each scene averaging 106 categories, 660 instances, and 14 regions.Synonymous raw tags remain present because annotators had freedom when defining category names.
- Dataset statistics: Each region averages 46 object instances and 20 unique categories, while room-type heuristics reveal varied residential, commercial, office, and living-space compositions.More than 30% of scenes are larger residences with at least four bedrooms and bathrooms; further statistics support subset selection and downstream analysis.
4. Experiments
The experiments evaluate ObjectNav policies trained on HM3DSEM, Gibson, and MP3D using reinforcement, imitation, and modular learning, with validation across datasets. HM3DSEM training generally provides the strongest cross-dataset performance, and larger training sets improve results.
- Experimental setup: ObjectNav policies are trained with reinforcement, imitation, and modular learning on HM3DSEM, Gibson, and MP3D, then evaluated across all three datasets.The evaluation uses the standard Success and SPL metrics and compares generalization across scene datasets.
- Reinforcement Learning: HM3DSEM-trained reinforcement-learning agents achieve the best performance averaged across validation datasets.The HM3DSEM-trained agent also outperforms the Gibson-trained agent on Gibson validation scenes.
- Imitation Learning: HM3DSEM-trained imitation-learning policies achieve the best average validation performance across datasets.The HM3DSEM policy also outperforms the Gibson policy on Gibson validation scenes.
- Imitation Learning: Increasing the number of HM3DSEM training scenes consistently improves imitation-learning validation performance.The study varies the training set from 25 to 80 scenes and suggests further gains from additional demonstrations and scenes.
- Modular Learning: HM3DSEM-trained semantic exploration policies achieve the best validation performance averaged across all evaluated datasets.The comparison uses the same exploration-policy architecture and training process across HM3DSEM, Gibson, and MP3D.
- ObjectNav Challenge 2022: 1022 submissions from 54 teams entered the 2022 Habitat ObjectNav challenge, compared with 400 submissions in 2021.The task definition remained identical while the dataset changed from Matterport3D to HM3DSEM.
5. Conclusion
HM3DSEM is presented as a large public dataset of densely annotated real-world spaces using pixel-accurate texture-based boundaries. Across reinforcement, imitation, and modular learning, HM3DSEM-trained policies outperform policies trained on prior datasets, while participation increased after its use in the 2022 ObjectNav challenge.
- Dataset and annotation: HM3DSEM is a large public dataset of real-world spaces with dense semantic annotations and pixel-accurate object boundaries encoded using textures.The dataset uses expert annotation and verification and provides standardized scene annotations for Habitat.
- ObjectNav results: Policies trained on HM3DSEM outperform policies trained on prior datasets across reinforcement, imitation, and modular learning methods.The conclusion also reports improved performance as the training dataset size increases.
- Challenge participation: The 2022 Habitat ObjectNav challenge received more submissions after introducing HM3DSEM than the 2021 challenge.The paper reports increased participation as evidence of community adoption of the dataset.
A.1. Assumptions
The annotation analysis relies on assumptions that object labels are accurate, object qualifiers indicate likely room locations, and object arrangements generally reflect natural human use.
- Annotation assumptions: Annotations are assumed to accurately describe the objects being labeled.This assumption underlies inference from object annotations to scene content.
- Annotation assumptions: Qualifiers in object names, such as “bath” or “kitchen,” are assumed to strongly indicate where those objects are found.The examples associate such qualifiers with bathrooms and kitchens, respectively.
- Layout assumptions: Object layouts are assumed to reflect natural human use rather than deliberately unnatural staging.The paper notes that this assumption supports interpreting co-occurring objects and room configurations.
A.2. Instance Segmentation and Object Detection
The paper benchmarks Mask-RCNN detection and instance segmentation across HM3DSEM, Gibson, MP3D, and ADE20k. HM3DSEM-trained models generalize best to new scenes and datasets, including ADE20k.
- Benchmark setup: Mask-RCNN models trained on each dataset are evaluated for object detection and instance segmentation across all three validation splits and ADE20k images.The benchmark uses six ObjectNav object classes and approximately 500 real-world residential images from ADE20k.
- Results: HM3DSEM-trained predictions generalize to ADE20k much better than predictions from models trained on Gibson or MP3D.Figure 7 visualizes ground-truth masks and predictions on ADE20k images.
- Results: HM3DSEM-trained models generalize best across scenes and datasets by a large margin.The result is reported for object detection and instance segmentation and is described as echoing the ObjectNav findings.
A.3. Analysis Method
The analysis maps annotator-provided category tags to heuristic room-region proposals, then organizes category presence by scene and region for downstream analysis.
- A.3. Analysis Method: 261 of 1,624 annotation tags received heuristic region proposals based on their annotation names.These proposals are suggestions rather than absolute room labels.
- A.3. Analysis Method: The candidate region names and their mapped category tags are explicitly listed for inspection.Examples include bathroom, office, and recreation-room mappings.
- A.3. Analysis Method: Category tags are recorded for every object instance and organized per region and scene with their associated region proposals.The organization is performed per tag rather than per object instance.
A.4. Scene-level Statistics
The study examines all 216 scenes using category-presence heuristics and room annotations derived from proposal votes, revealing interpretable scene-level patterns with occasional labeling errors.
- A.4. Scene-level Statistics: 12 scenes lacked bedroom-proposal objects, and all were visually verified as commercial spaces.The spaces were offices, restaurants, or stores.
- A.4. Scene-level Statistics: 7 scenes lacked bathroom-proposal tags, and these were also visually verified as non-residential spaces.One large house converted into a museum had bedroom categories but no bathroom categories.
- A.4. Scene-level Statistics: 25 scenes contained proposed garage annotations, and visual inspection verified that all contained garages.This is a direct agreement between category-derived proposals and scene content.
- A.4. Scene-level Statistics: Aggregating proposal votes by region yields potential room labels that provide more accurate suggestions of scene content.The proposal-based labels are derived from category counts within regions.
A.5. Region Label Inference
Region-level proposal votes can infer room layouts and counts from constituent object categories, producing results that closely match visual inspection while retaining some ambiguity.
- A.5. Region Label Inference: A randomly selected scene yielded 13 region proposals: 3 bathrooms, 4 bedrooms, 3 hallways or stairwells, 1 kitchen, 1 living room, and 1 ambiguous bedroom or dining room.Visual inspection produced the same count, with the ambiguous region identified as a dining room.
- A.5. Region Label Inference: In a larger scene, 3 of 21 regions lacked proposal votes because their object categories had no assigned region proposals.The remaining 18 regions received specific or ambiguous proposals.
- A.5. Region Label Inference: The larger scene’s 18 proposed regions included 5 bathrooms, 4 bedrooms, 1 living room, 2 offices, 1 recreation room, 2 kitchens, and 3 ambiguous mappings.The ambiguous mappings were bedroom or office, hallway or bedroom, and hallway or recreation room.
- A.5. Region Label Inference: Visual inspection found similar room assignments, but category weighting can reduce ambiguity while potentially misclassifying rooms containing stored beds.The weighted example assigns the bed category 10 votes instead of 1.
- A.5. Region Label Inference: Scene-level region proposals provide a reasonable estimate of each scene’s room layout and room counts.
A.6. Object-Level Analysis and Files
The object-level analysis aggregates category presence, neighboring categories, and region proposals into reusable statistics and files for studying scene and region semantics.
- A.6. Object-Level Analysis and Files: Category data are aggregated across scenes, regions, and neighboring categories so objects sharing a region can be analyzed together.The aggregation also retains region proposals for categories that have them.
- A.6. Object-Level Analysis and Files: Category prevalence across regions suggests region labels for otherwise unmapped category tags based on the categories they co-occur with.This uses the category’s regional “company they keep.”
- A.6. Object-Level Analysis and Files: The released analysis files include object counts, uncommon-category counts, scene-neighborhood statistics, region-tag mappings, and weighted-vote data.These files support independent analyses of semantic scenes.
- A.6. Object-Level Analysis and Files: Additional aggregate files report category-region neighbors, scene and region presence, instance totals, and average instances per scene or region.Aggregate categories are formed by uniting categories that share the same hand-annotated region proposal.