PaperlayerView exact revision on arXiv

Source-linked AI summary

ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner

arXiv:1702.04405v2cs.CV

TL;DR

RGB-D scene understanding lacks large datasets with broad views and rich semantic annotations. ScanNet addresses this gap with a scalable capture and crowdsourced annotation framework, producing richly annotated scans that support state-of-the-art results across several 3D scene understanding tasks.

  • Problem

    RGB-D scene understanding lacks large labeled datasets because 3D data capture is more effortful, while existing datasets provide limited views and semantic annotations.

  • Method

    ScanNet combines scalable commodity RGB-D capture, automated reconstruction, and crowdsourced semantic annotation to create richly annotated scans of real-world indoor environments.

  • Results

    ScanNet data supports state-of-the-art performance on 3D object classification, semantic voxel labeling, and CAD model retrieval.

  • Takeaways & Limitations

    The released dataset and acquisition framework are intended to benefit the community and inspire future work on additional 3D scene understanding tasks.

  • Takeaways & Limitations

    CAD alignment is limited by mismatch between available CAD models and objects observed in ScanNet scans, making exact instance-level matches difficult.

Abstract

from arXiv · show

A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available -- current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval. The dataset is freely available at http://www.scan-net.org.

1. Introduction

ScanNet addresses the limited size and scalability of existing RGB-D datasets with a large, richly annotated collection of real-world indoor scans. The paper also presents a novice-friendly capture and annotation pipeline and evaluates learned 3D models on multiple scene-understanding tasks.

  • Motivation: Existing 3D datasets are small because capture is effort-intensive and dense 3D annotation is non-trivial, limiting scalability.Prior work often uses polygon or bounding-box annotations on 2.5D RGB-D images, or relies on manual expert labeling.
  • Dataset: ScanNet contains 2.5M RGB-D images from 1513 scans across 707 distinct spaces, with camera parameters, poses, 3D reconstructions, textured meshes, and dense semantic segmentation.The dataset also provides CAD model placements for a subset of scans.
  • Capture and annotation: The capture pipeline enables novices to acquire semantically labeled 3D models using an iPad-mounted depth camera, offline processing, and returned scene reconstructions.The pipeline was designed to support large-scale RGB-D data collection and annotation by many people.
  • Evaluation: The authors train 3D deep networks on ScanNet and evaluate them on 3D object classification, semantic voxel labeling, and CAD model retrieval.They introduce a new volumetric CNN architecture for semantic voxel labeling.

2. Previous Work

Prior RGB-D datasets span isolated-object scans and indoor scenes, but commonly provide limited viewpoints, incomplete 3D annotations, or pipelines that restrict scalability and raw RGB-D research. SceneNN and PiGraphs offer densely reconstructed scenes, yet rely on expert-only scanning and labeling.

  • RGB-D datasets: 10,000+ object scans in Redwood lack scene context, so the dataset primarily evaluates surface reconstruction rather than semantic understanding of complete scenes.Only 1,781 scans were reconstructed with KinectFusion.
  • RGB-D datasets: 1,449 annotated frames in NYU v2 and 10,335 frames in SUN RGB-D provide 2D polygons, with SUN RGB-D also adding 3D bounding boxes but limited viewpoints.NYU v2 contains 464 short RGB-D sequences; the passages state that both datasets have scene diversity comparable to ScanNet.
  • RGB-D datasets: 415 SUN3D sequences cover 254 unique spaces, but most of the dataset lacks camera poses, 3D reconstructions, and semantic annotations.Some objects have manual 2D polygon annotations, while only 8 scans have camera poses estimated from user input.
  • RGB-D datasets: 265-room Matterport data provides high-quality labeled 3D meshes, but its expensive, less portable hardware and fused-point-cloud output limit raw RGB-D reconstruction research.The passage also notes that raw color and depth data are unavailable.
  • RGB-D datasets: SceneNN and PiGraphs contain 100 and 26 densely reconstructed, 3D-labeled scenes, respectively, but expert-only scanning and labeling limit scalability and dataset size.The passages contrast these systems with an acquisition framework designed for untrained users and scalable processing, although the supplied text ends before describing that framework fully.

3. Dataset Acquisition Framework

ScanNet’s acquisition framework was designed for untrained users to capture semantically labeled indoor surfaces with commodity hardware, combining simple capture, automated processing, crowdsourced annotation, and server-based data flow. The pipeline uses robust, fast reconstruction and web-based interfaces to support scalable RGB-D data collection and labeling.

  • Framework design: The framework targets untrained users, requiring trivial capture, robust automatic processing, crowdsourced semantic annotation, and tracking-server data handling.Its main goal is scalable acquisition of semantically labeled indoor surfaces using commodity hardware.
  • Hardware and capture: The capture system combines a portable, low-cost Structure RGB-D sensor with an iPad Air2, recording synchronized depth and color at 30 Hz.Depth is captured at 640 × 480 and color at 1296 × 968 pixels.
  • Automated reconstruction: Uploaded scans are processed automatically with BundleFusion and volumetric fusion to estimate 6-DoF poses and dense surface meshes at near-real-time rates.BundleFusion runs at 1 cm3 voxel resolution, while high-resolution meshes use 4 mm3 TSDF voxels and Marching Cubes.
  • Validation: The reconstruction pipeline automatically validates uploads by discarding short sequences, high-error reconstructions, and scans with few aligned frames before manual misalignment checks.This produces a cleaner dataset snapshot while preserving an unsupervised processing flow.
  • Crowdsourced annotation: Amazon Mechanical Turk workers annotate reconstructed surfaces with instance-level object categories and align 3D CAD models through web-based interfaces.The CAD-alignment task is limited by mismatches between available models and observed objects, despite ShapeNet’s 55K-object corpus.

4. ScanNet Dataset

ScanNet is a growing RGB-D dataset built from scans collected by 20 users and annotated by more than 500 Mechanical Turk workers. It spans diverse indoor spaces, provides instance-level 3D annotations, complements NYU v2, and includes crowdsourced CAD model alignments.

  • Dataset collection: ScanNet’s initial snapshot covers data collected over roughly one month by 20 users across locations in several countries, with annotations from more than 500 crowd workers.Because the framework operates unsupervised and data collection continues, the dataset grows organically.
  • Scene diversity and annotation: The dataset spans diverse real-world spaces, including offices, apartments, bathrooms, classrooms, and libraries, from small rooms to large spaces.Each scan received instance-level semantic category labels through crowdsourcing.
  • Scene diversity and annotation: 3,391 annotation tasks were deployed to annotate all 1513 scans.Crowd-worker labels were mapped to category sets including NYU v2, ModelNet, ShapeNet, and WordNet using synonym and misspelling detection.
  • Additional reconstructions: Processing all NYU v2 RGB-D sequences produced dense reconstructions with instance-level 3D object annotations complementary to existing image-based annotations.This extends the framework beyond the newly collected ScanNet scans.
  • CAD model alignment: 681 CAD model instances from 296 unique models were aligned and placed across 52 ScanNet scans in 107 virtual scene interpretations.The interpretations were produced by 106 workers, averaging 6.4 CAD model instances per annotated scan.

5. Tasks and Benchmarks

This section presents three benchmarks showing that ScanNet’s large-scale real-world RGB-D data supports supervised 3D scene understanding. The tasks include object classification, semantic voxel labeling, and correspondences between real objects and ShapeNet models for retrieval.

  • Benchmarks: ScanNet’s substantially larger train/test splits enable supervised deep learning for 3D scene understanding and comparison with existing datasets.The benchmarks use ScanNet data for object classification and dense voxel prediction.
  • 3D Object Classification: Training on ScanNet generalizes to SceneNN, whereas synthetic training transfers only limited knowledge and SceneNN training does not generalize well to ScanNet.The object-classification benchmark evaluates synthetic ShapeNet data and real-world SceneNN and ScanNet test sets.
  • Semantic Voxel Labeling: The semantic voxel-labeling network predicts labels for every surface voxel column from a 2×31×31×62 occupancy and known/unknown-space volume.Fully convolutional layers are applied at every xy position containing surface voxels.
  • Semantic Voxel Labeling: 73.0% voxel classification accuracy is achieved on 312 ScanNet test scenes using only geometric input across 20 object classes.The task predicts semantic labels for every visible surface voxel without using color.
  • CAD Model Retrieval: ScanNet establishes category-level correspondences between real-world objects and ShapeNet models, enabling mixed real-and-synthetic training for classification and model retrieval.Retrieved ShapeNetCore models are evaluated by their instance accuracy to ScanNet query regions.

6. Conclusion

ScanNet is introduced as a large-scale RGB-D dataset with richly annotated scans, surface reconstructions, object categories, and CAD model placements. The paper also presents a scalable RGB-D acquisition and semantic annotation framework and reports that ScanNet data supports state-of-the-art 3D scene understanding performance.

  • Dataset: ScanNet contains 1513 scans with surface reconstructions, instance-level object category annotations, and 3D CAD model placements.These annotations and reconstructions make the dataset richly labeled for 3D scene understanding.
  • Data collection framework: The authors designed a scalable RGB-D acquisition and semantic annotation framework to enable ScanNet’s data collection.The framework is provided for the benefit of the community.
  • Impact: ScanNet’s richly annotated scan data achieves state-of-the-art performance on several 3D scene understanding tasks.The conclusion presents this result as evidence of the dataset’s usefulness.

A. Dataset Statistics and Comparisons · A.1. Example Annotated Reconstructions

This section reports ScanNet’s construction and composition statistics while comparing it with similar prior datasets. Example reconstructions illustrate consistent semantic and instance annotations across spaces ranging from small rooms to large, object-rich areas.

  • A. Dataset Statistics and Comparisons: The section provides statistics on ScanNet’s construction and composition and compares the dataset with similar prior work.
  • A.1. Example Annotated Reconstructions: Six example annotated reconstructions cover a variety of spaces in ScanNet.
  • A.1. Example Annotated Reconstructions: Each example shows a colored surface mesh alongside a visualization assigning category labels to collected objects.
  • A.1. Example Annotated Reconstructions: Category labels remain consistent across spaces and map to WordNet synsets.
  • A.1. Example Annotated Reconstructions: Separate object-instance labels identify multiple instances belonging to the same category.
  • A.1. Example Annotated Reconstructions: The reconstructed spaces range from small rooms containing only a few objects, such as toilets.
  • A.1. Example Annotated Reconstructions: Larger scans include classrooms and studio apartments containing dozens of objects.

A.2. Dataset Construction Statistics

ScanNet’s construction framework reports reconstruction and annotation timing statistics for its 1513 scans. Reconstruction takes a median of 11.3 minutes per scene, while annotation involves 2.3 crowd workers per scan and a median of 16.8 minutes per scan.

  • Dataset scale: ScanNet contains far more annotated object instances than SceneNN among the 50 largest categories, with labels processed to remove duplicates.The consistency processing addresses duplicate labels such as “chair01” in SceneNN.
  • Reconstruction: 11.3 min is the median reconstruction processing time per scene, compared with a 14.9 min mean caused by downtime-related outliers.Processing includes data conversion, dense voxel fusion, surface mesh extraction, alignment, cleanup, and preview thumbnail rendering.
  • Annotation: 2.3 crowd workers annotate each scan on average, with a 12.0 min median annotation time per worker and a 17.3 min mean.A few workers taking significantly longer account for the higher mean.
  • Annotation: 16.8 min is the median annotation time per scan across ScanNet’s 1513 scans, compared with a 22.3 min mean.The aggregate timing sums the work of all crowd workers annotating each scan.

A.3. Dataset Composition Statistics

ScanNet addresses the scarcity of large, publicly available RGB-D datasets with dense scene-level reconstruction and semantic annotation. Its 3D annotations also support ShapeNetCore category mapping and provide 76% average pixel coverage when projected into RGB-D frames.

  • Dataset motivation: ScanNet was motivated by the lack of large, publicly available RGB-D datasets with densely reconstructed 3D scenes and comprehensive annotations.Existing datasets either annotate only subsets of frames or focus on decontextualized objects rather than scenes.
  • Category mapping: ScanNet category labels map to ShapeNetCore classes, enabling automatic CAD-model suggestions for objects in reconstructed scenes.The mapping is used during CAD model alignment and retrieval to suggest matching ShapeNet instances.
  • Object composition: ScanNet contains thousands of reconstructed instances of common objects, including chairs, tables, and cabinets, mapped among the top 25 annotation labels.These labels are mapped to ShapeNetCore classes.
  • 2D annotation projection: 76% average annotation coverage is obtained by projecting 3D annotations into each RGB-D frame using the corresponding camera pose.The coverage is measured across all pixels per scene.

A.4. NYUv2 Reconstruction and Comparison

The paper reconstructs and annotates NYUv2 RGB-D sequences with the ScanNet framework, excluding nine sequences lacking valid camera poses for more than half their frames. Compared with ScanNet, NYUv2 reconstructions are sparser and have substantially lower semantic surface coverage when annotations are projected from original frames.

  • Reconstruction and annotation: All NYUv2 RGB-D sequences were reconstructed and annotated using the ScanNet framework, except nine sequences without valid poses for > 50% of frames.The comparison also included a surface-mesh semantic annotation set created with the framework.
  • Semantic surface coverage: Figure 10 compares the percentage of reconstructed surface area annotated by projected NYUv2 frames, the ScanNet annotation pipeline, and ScanNet reconstructions.The nine sequences lacking sufficient valid camera poses were excluded from this comparison.
  • Reconstruction completeness: ScanNet sequences generally produce more complete reconstructions of real-world spaces than NYUv2 sequences.The original NYUv2 frames often provide too few viewpoints to fully reconstruct semantically meaningful objects.
  • Reconstruction completeness: NYUv2 reconstructions are much sparser than ScanNet reconstructions, making direct quantitative comparison difficult.Top-down scene views illustrate the sparsity disparity.
  • Semantic surface coverage: Projecting annotated NYUv2 RGB-D frames onto reconstructions is insufficient for semantic annotation because NYUv2 has far lower surface-coverage distributions.This conclusion is supported by the comparison in Figure 10.

B. Tasks · B.1. Semantic Voxel Labeling

The section details 3D scene-understanding benchmarks, focusing on semantic voxel labeling. The proposed network predicts class labels for voxel columns using occupied and known-space neighborhood information, then applies these predictions across voxelized scans.

  • B. Tasks: The paper provides additional details about the 3D scene-understanding tasks and benchmarks discussed in the main paper.
  • B.1. Semantic Voxel Labeling: The semantic voxel labeling network predicts class labels for each column of a voxelized scene.
  • B.1. Semantic Voxel Labeling: The network takes a 2 × 31 × 31 × 62 volume as input and predicts class scores for the center column of 62 voxels.
  • B.1. Semantic Voxel Labeling: Fully convolutional layers enable simultaneous prediction of class scores for the center voxel column.
  • B.1. Semantic Voxel Labeling: The model uses voxel-neighborhood information from occupied space and known space in front of surfaces according to the camera trajectory.
  • B.1. Semantic Voxel Labeling: At test time, the network slides through a voxelized scan along the xy-plane and predicts each column independently.
  • B.1. Semantic Voxel Labeling: Visualizations compare predicted voxel labels for ScanNet test scans with ground-truth annotations from the crowdsourced labeling task.
  • B.1. Semantic Voxel Labeling: Figure 13 presents several ScanNet test scans alongside their voxel label predictions and crowdsourced ground-truth annotations.

C. Dataset Acquisition Framework … C.6. Management UI

ScanNet’s acquisition framework combines novice-friendly RGB-D capture, calibration and cloud reconstruction with crowdsourced annotation, label consolidation, and web-based progress management. Together, these components support scalable data collection while improving reconstruction quality and maintaining visibility into scans and metadata.

  • C. Dataset Acquisition Framework: The framework targets ease of capture, robust reconstruction, rapid crowdsourcing, and visibility into collected data and metadata.It uses an iPad app, simple calibration, cloud reconstruction, and web-based interfaces to support these goals.
  • C.1. RGB-D Acquisition UI: The iPad acquisition UI lets novice users record RGB-D videos with minimal controls, room metadata entry, and a single start-stop recording button.Users can switch between color and depth views, while a featurefulness bar indicates regions likely to cause tracking loss.
  • C.2. Sensor Calibration: Calibration estimates sensor intrinsics and extrinsics, while depth undistortion uses a lookup table to correct camera-dependent distortion.The authors report that undistorting Structure-sensor depth images significantly improves tracking.
  • C.3. Surface Reconstruction: BundleFusion produces fused 3D surfaces that are cleaned by merging vertices within 1 mm and removing components with fewer than 7500 triangles.Two quadric edge-collapse decimation steps then create lower-triangle-count mesh versions.
  • C.4. Crowdsourced Annotation UI: Mechanical Turk workers paint object instances on reconstructed surfaces, annotating at least 25% of each reconstruction and being encouraged to cover at least 50%.A separate CAD retrieval task requires workers to place at least three objects and pays $1.00 per completed task.
  • C.5. Label cleaning and propagation: Labels from multiple workers are normalized to canonical names using curated synonyms, misspellings, and an edit-distance threshold, while labels with less than 5 counts are ignored.Scene consensus is computed through majority voting over over-segmented mesh regions and geometric unions with ≥50% overlap.
  • C.5. Label cleaning and propagation: Consensus labels are propagated from the coarse mesh to the high-resolution mesh by nearest-neighbor lookup in a kd-tree and projected onto input 2D frames.This extends the aggregated semantic annotation across reconstructed and image data.
  • C.6. Management UI: The web-based management UI automatically queues uploaded scans, exposes processing progress, and provides reconstruction thumbnails and statistics for filtering and sorting.Tracked stages include calibration, pose optimization, fusion, alignment, cleanup, decimation, segmentation, and crowdsourced annotation.
Loading 1702.04405v2…