Source-linked AI summary
Matterport3D: Learning from RGB-D Data in Indoor Environments
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, Yinda Zhang
TL;DR
Existing RGB-D datasets provide limited views or restricted spatial coverage, constraining training data for indoor scene understanding. Matterport3D introduces a large, globally aligned dataset of panoramic RGB-D views with semantic annotations and evaluates tasks enabled by these properties. The authors report state-of-the-art performance on several scene-understanding tasks and improved cross-dataset normal-estimation generalization.
Problem
Existing RGB-D datasets offer limited image numbers, scene coverage, viewpoints, or motion quality, leaving no high-quality RGB-D dataset with diverse views across interior home environments.
Method
Matterport3D combines 194,400 RGB-D images from 10,800 panoramas across 90 buildings with globally consistent poses, surface reconstructions, and semantic segmentations.
Results
The dataset enables multiple scene-understanding tasks and achieves state-of-the-art performance on several of them, while Matterport3D-trained normal-estimation models generalize better to NYUv2.
Takeaways & Limitations
Diverse panoramic views, precise building-scale alignment, high-quality depth, and semantic context support learning representations across several indoor vision tasks.
Abstract
from arXiv · showhide
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.
1. Introduction
Matterport3D addresses the shortage of diverse, high-quality RGB-D training data with building-scale panoramic coverage and globally aligned semantic annotations. These properties support multiple indoor scene-understanding tasks and state-of-the-art baseline results.
- Indoor RGB-D scene understanding supports applications including personal robotics, augmented reality, scene modeling, and perception assistance.
- Existing datasets commonly have limited images, scenes, viewpoints, or motion quality, and none provide high-quality RGB-D views across diverse interior home environments.
- Matterport3D contains 194,400 RGB-D images from 10,800 panoramas captured across 90 building-scale scenes.
- The dataset combines 360° color and depth panoramas, human-height viewpoints sampled throughout environments, globally consistent camera poses, textured reconstructions, and instance-level region and object segmentations.
- Five investigated tasks use the dataset’s alignment, panoramic sampling, depth quality, and semantic context for keypoint matching, view overlap prediction, normal estimation, semantic segmentation, and region classification.
2. Background and Related Work
Prior RGB-D datasets span objects, rooms, apartments, offices, and public spaces, but generally lack Matterport3D’s scale, coverage, alignment accuracy, and HDR imagery. Matterport3D also expands available modalities, categories, and scene types beyond closely related Matterport-based work.
- Earlier RGB-D datasets mainly cover individual objects, standalone rooms, room views, small apartments, academic buildings, or limited public spaces.
- These datasets support tasks such as semantic segmentation, 3D object detection, normal estimation, camera relocalization, image-patch description, overlap prediction, voxel labeling, and region classification.
- Compared with a related Matterport-based dataset, Matterport3D covers 90 diverse buildings rather than 3 office buildings.
- Matterport3D panoramas are captured from viewpoints approximately 2.25m apart.
- Matterport3D additionally provides raw depth, HDR images, and 40 broadly covered categories, whereas the related dataset has RGB images, a coarse mesh, and 13 object categories.
3. The Matterport3D Dataset
Matterport3D is a building-scale RGB-D dataset with globally aligned imagery, textured reconstructions, and instance-level semantic annotations. Its comprehensive coverage, diverse views, and private-home scenes support new scene-understanding research opportunities.
- Data acquisition: Panoramas are captured approximately every 2.5m throughout each environment, with global bundle-adjusted poses and a single textured mesh reconstructed per scene.The acquisition process samples the entire walkable floor plan and aligns images to building-scale reconstructions.
- Data acquisition: 90 buildings contribute 194,400 RGB-D images, 10,800 panoramas, and 24,727,520 textured triangles.Each image is 1280x1024 with HDR color and a 6 DoF camera pose; each scene also includes a skybox and textured mesh.
- Semantic annotation: 3D semantic annotation combines annotator-specified floor-plan regions with crowdsourced object-instance labeling completed, fixed, and verified by expert annotators.Regions are extruded to ceilings, while reconstructed region meshes are painted to segment and name object instances.
- Dataset properties: The dataset provides aligned RGB-D panoramas spanning most of the sphere, globally registered imagery across 90 buildings, and stationary HDR capture without motion blur.The authors estimate average registration error at 1cm or less, while acknowledging rare misalignments of 10cm or more.
- Dataset properties: Each surface patch is observed from multiple angles and distances, averaging 11 cameras, while complete scans of 90 buildings provide long-range spatial context.These properties support learning view-dependent surface properties, view-independent representations, holistic scene understanding, and autonomous navigation.
- Dataset properties: Matterport3D covers 219,399m2 of surface area, 2,056 rooms, and 46,561m2 of floor space, providing scale for data-hungry algorithms.The dataset also includes imagery from private homes, a difficult-to-capture setting relevant to consumer-level applications.
4. Learning from the Data
Matterport3D supports learning across keypoint matching, view-overlap prediction, surface-normal estimation, and region classification by combining broad viewpoint coverage, global alignment, and high-quality depth. Experiments report stronger cross-dataset generalization and task performance, while wide-baseline overlap prediction remains more difficult.
- Keypoint Matching: 61 training scenes and 17 SUN3D scenes support keypoint-descriptor training and evaluation on 8 held-out SUN3D scenes.Performance is measured by false-positive rate at 95% recall, where lower is better.
- Keypoint Matching: Matterport3D provides wide-baselined correspondences that enable triplet-Siamese training of a ResNet-50 512-dimensional local descriptor.Training examples contain two matching patches and one non-matching patch.
- View Overlap Prediction: Matterport3D training improves loop-closure retrieval on SUN3D, while overlap-ratio regression improves performance on both SUN3D and Matterport3D.Evaluation uses normalized discounted cumulative gain on retrieval lists, excluding candidate pairs with travel distance at most 0.5m.
- View Overlap Prediction: Wide baselines make overlap prediction harder in Matterport3D than in hand-held-device data such as SUN3D.The dataset's panoramic, comprehensive viewpoint sampling creates many view-overlap and loop-closure instances.
- Surface Normal Estimation: Matterport3D's continuous rotating depth capture produces less-noisy normals, and models trained with its data generalize better from Matterport3D to NYUv2 than in the reverse direction.The model pretrained on synthetic data and Matterport3D achieves the best performance, while NYUv2 ground truth can introduce comparatively noisy results after further fine-tuning.
- Region-Type Classification: Wider image fields of view improve classification for most region types, while lounge and family-room accuracy is lower because adjacent regions become more visible and confusing.Table 5 defines classification accuracy as true positives divided by total instances per region type.
5. Conclusion
Matterport3D is a large RGB-D dataset of 90 building-scale scenes with semantic 3D annotations and diverse panoramic views. The paper reports state-of-the-art performance on several scene-understanding tasks and releases the dataset for research.
- 5. Conclusion: Matterport3D combines 90 building-scale scenes, instance-level semantic segmentations, panoramic RGB-D views, precise global alignment, and broad indoor semantic context.These properties span complete building environments and diverse living spaces.
- 5. Conclusion: The dataset enables myriad computer-vision tasks and achieves state-of-the-art performance on several scene-understanding tasks.The authors release Matterport3D for research use.
A. Learning from the Data
The dataset is partitioned into training, validation, and test scenes, with visualizations showing the diversity and scale of complete scanned environments.
- A. Learning from the Data: Bird's-eye-view textured-mesh images help convey the diversity and scale of the dataset's scenes.Each image represents one scene.
A.1. Keypoint Matching
Matterport3D's stationary-camera captures comprehensively sample viewpoints, allowing keypoints to be observed across varied views and supporting challenging correspondence learning. Visualizations show matching patches and descriptor clustering across view changes.
- A.1. Keypoint Matching: Stationary cameras comprehensively sample viewpoint space, exposing keypoints to diverse views that challenge matching and tracking.The paper specifically connects these scenarios to loop-closure detection.
- A.1. Keypoint Matching: A t-SNE embedding shows similar local keypoints clustering despite significant changes in viewpoint.The embedding is based on descriptors from the triplet Siamese network.
A.2. View Overlap Prediction
Matterport3D supports predicting the fraction of shared visible surface between pairs of views within the same scene. Its dense, diverse viewpoints create many substantial overlaps, including across different directions and regions, making prediction challenging.
- View overlap prediction estimates the fraction of pixels observing the same surface in two images.
- Matterport3D provides many overlapping views because each scene contains dense and diverse camera viewpoints.Each tripod panorama captures 18 images across six rotations and three tilt angles.
- Most examples contain around a dozen significant overlaps, including between cameras with substantially different view directions and regions.The figure encodes 0% overlap as thin dull cyan and at least 20% overlap as thick bright red.
A.3. Surface Normal Estimation
The paper evaluates surface-normal prediction from RGB using models trained or fine-tuned on synthetic data, NYUv2, and Matterport3D. Matterport3D provides cleaner depth and normal data, and its fine-tuned models produce strong results across both datasets.
- The SUNCG-MP model often produces clean planar regions with more detail, while further fine-tuning on noisy NYUv2 ground truth hurts these properties.
- Matterport3D and NYUv2 fine-tuning are compared after pretraining on synthetic data, with evaluation performed on both datasets.
- Matterport3D depth and surface normals are cleaner and more detailed than those from NYUv2, whose depth noise can overwhelm local detail.
- On Matterport3D images, the Matterport3D-fine-tuned model produces clean flat regions and local details, whereas the NYUv2-fine-tuned model does not work well.
- On NYUv2 images, the NYUv2-fine-tuned model performs well, while the Matterport3D-trained model remains reasonably good and is sometimes better.
- The model can predict surface normals in areas with missing depth, suggesting potential for improving raw sensor depth images.
A.4. Region-Type Classification
Matterport3D provides annotations for classifying the room or region containing a panorama and for labeling semantic voxels in reconstructed scenes. The examples show manually specified region boundaries and categories alongside object-instance and canonical category labels.
- Region-type classification predicts the category of the room containing a given panorama.Categories include bathroom, bedroom, closet, dining room, entryway, familyroom, garage, hallway, library, laundryroom, kitchen, livingroom, meetingroom, lounge, and office.
- For every building, a person manually outlines each region's floorplan boundary and assigns its semantic category.
- Per-voxel labeling predicts object-instance semantics after textured meshes are partitioned into region meshes and annotated through a crowdsourced paint-and-name interface.
- The dataset includes manually painted object-instance and category labels, with raw chair types mapped to a canonical chair category.
- Region annotations represent boundaries and categories through floorplans, mesh surfaces, and region instances.