Source-linked AI summary
Joint 2D-3D-Semantic Data for Indoor Scene Understanding
Iro Armeni, Sasha Sax, Amir R. Zamir, Silvio Savarese
TL;DR
Large-scale indoor understanding lacks real datasets that jointly cover diverse registered modalities and dimensions. The paper presents such a dataset with consistent instance-level annotations across 2D, 2.5D, and 3D data, providing a resource for joint, cross-modal, and potentially unsupervised learning. Its scope is six indoor areas across three buildings and 70,496 regular plus 1,413 equirectangular RGB images, alongside corresponding geometric and semantic data.
Problem
Existing real datasets are often limited in scale, diversity, modalities, or cross-dimensional coverage for indoor scene understanding.
Method
The paper constructs a mutually registered indoor dataset with RGB, depth, surface normals, global XYZ and equirectangular images, semantic annotations, meshes, and point clouds.
Results
The dataset contains 70,496 regular RGB images, 1,413 equirectangular RGB images, and a 695,878,620-point colored 3D point cloud across six areas and three buildings.
Takeaways & Limitations
The registered modalities and consistent annotations support joint and cross-modal learning, task-specific data generation, and potentially unsupervised approaches in large-scale indoor spaces.
Abstract
from arXiv · showhide
We present a dataset of large-scale indoor spaces that provides a variety of mutually registered modalities from 2D, 2.5D and 3D domains, with instance-level semantic and geometric annotations. The dataset covers over 6,000m2 and contains over 70,000 RGB images, along with the corresponding depths, surface normals, semantic annotations, global XYZ images (all in forms of both regular and 360° equirectangular images) as well as camera information. It also includes registered raw and semantically annotated 3D meshes and point clouds. The dataset enables development of joint and cross-modal learning models and potentially unsupervised approaches utilizing the regularities present in large-scale indoor spaces. The dataset is available here: http://3Dsemantics.stanford.edu/
1 Stanford University 2 University of California, Berkeley
Figure 1 presents a jointly registered dataset spanning 2D, 2.5D, and 3D modalities with semantic annotations across dimensions.
- The dataset combines RGB images, depth, surface normals, global XYZ images, and instance-level semantic annotations.
- Its modalities span 2D RGB images, 2.5D depth, and 3D meshes.
1. Introduction
The paper motivates a dataset for large-scale indoor 3D understanding by combining complementary RGB and geometric information across many modalities. It aims to support cross-modal, joint, and unsupervised learning for diverse scene-understanding tasks.
- Comprehensive 3D scans provide object geometry and surrounding context that cue semantics, layout, occlusion handling, shape completion, and amodal detection.
- RGB supplies dense appearance features that complement the geometric information available in 3D data.
- Existing RGB-D and 3D datasets are often limited in scale, diversity, or the number of modalities they provide.
- Large-scale scanning can capture entire buildings with point clouds containing hundreds of millions of points, requiring methods that handle and exploit this scale.
- The proposed dataset supports scene understanding, depth and surface-normal estimation, detection, segmentation, amodal detection, and scene reconstruction.
2. Related Datasets
The paper positions its dataset as a real-world resource that extends beyond task-specific 2.5D data by registering many image, mesh, and point-cloud modalities. It offers cross-domain learning and data generation, while remaining limited in category coverage.
- Existing real datasets are relatively limited to particular tasks and the 2.5D domain, restricting use across dimensionalities and modalities.
- The proposed dataset includes RGB, depth, equirectangular and global XYZ images, plus 3D meshes and point clouds of the same spaces.
- Its modalities can be used independently or jointly to develop models that operate across domains.
- Equirectangular images, camera parameters, and 3D meshes enable generation of additional task-specific data.
- The dataset provides more images, additional modalities, and consistent annotations across modalities and dimensions than existing real-world datasets, but has fewer object and scene categories.
3. Dataset Overview
The dataset covers six large indoor areas across three buildings and provides registered, instance-level annotations across dense 2D imagery and 3D reconstructions. Its modalities share pixel-level or cross-domain correspondence through a common reference system.
- The collection contains 70,496 regular RGB images and 1,413 equirectangular RGB images with corresponding depths, normals, semantics, global XYZ data, and camera metadata.
- The six areas come from three mainly educational and office-use buildings, with all modalities registered in one reference system.
- The dataset includes whole-building textured meshes, semantic meshes, and a colored point cloud totaling 695,878,620 points.
- Annotations are instance-level, consistent across modalities, and cover 13 object classes.
4. Collection and Processing
The dataset is built by registering scanner RGB data with textured and semantic 3D representations, then deriving consistently annotated 2D modalities and sampled views. It provides multiple 3D and image representations with shared semantic and instance information.
- Collection: The scanner produces reconstructed textured meshes, raw RGB-D images, and camera metadata used to generate additional RGB-D data and point clouds.Point clouds are created by sampling the reconstructed meshes, while semantic labels are annotated in 3D and projected into mesh and image domains.
- 3D modalities: The dataset contains textured and semantic mesh models, point clouds, and their semantic counterparts for each of six areas.The 3D modalities include reconstructed meshes and densely sampled colored point clouds.
- 3D semantics: 3D point-cloud annotations assign 13 object classes and unique instance identifiers, while scene annotations assign 11 room labels and unique indices.Annotations support object and scene semantics, including axis-aligned object bounding boxes.
- Processing: RGB and 3D data are registered to produce depth, surface normals, and projected 2D semantic images for each scan.Equirectangular images are processed first and then sampled into regular images.
- Image sampling: 72 regular images are sampled per scan location from equirectangular projections with consistent depth, surface-normal, and semantic information.Camera sampling uses yaw, pitch, and field-of-view distributions, with entropy-based selection to retain semantically diverse images while preserving some low-entropy scenes.
- Image modalities: All generated images are stored at 1080 × 1080 resolution, with camera poses provided per image and depth and surface normals rendered from the 3D mesh.Surface normals are encoded as 24-bit RGB images, and missing normal values use (128,128,128).
5. Sample Data
The sample data spans diverse indoor scenes and varies in scene category, appearance, object density, clutter, and example difficulty.
- Sample diversity: The generated samples include variation in scene category, appearance, intra-class variation, object density, and clutter.The collection contains both easy and hard examples.
6. Train and Test splits
Training and testing splits are designed to prevent visually and architecturally similar building areas from appearing across the split boundary, using six areas and 3-fold cross-validation.
- Split design: The standard splits keep areas from similarly looking buildings out of both training and testing sets.The six dataset areas are divided according to Table 5 under a 3-fold cross-validation scheme.
7. Baseline Results
The baseline evaluates 3D object detection on the dataset’s point clouds using hierarchical semantic parsing, class-specific SVMs, and contextual CRF refinement.
- Baseline method: The baseline first parses raw point-cloud data into semantically meaningful spaces and aligns them to a canonical reference coordinate system.This first stage uses an unsupervised approach.
- Baseline method: The second stage parses each space into elements from 12 available classes using one-vs-all SVMs and a CRF for contextual consistency.Quantitative results are reported in Table 6 and qualitative examples in Figure 6.
8. Conclusion
The dataset combines mutually registered 2D, 2.5D, and 3D modalities with consistent semantic annotations. It is intended to support cross-modal, joint-modal, and potentially unsupervised learning in large-scale indoor spaces.
- The dataset provides mutually registered RGB, depth, surface-normal, global XYZ, semantic, mesh, and point-cloud modalities.Semantic annotations were performed in 3D and projected consistently across modalities and dimensions.
- 2D and 2.5D data are available as both regular and 360° equirectangular images.
- The dataset is designed to fuel cross-modal, joint-modality, and potentially unsupervised approaches that exploit regularities in large-scale manmade spaces.