PaperlayerView exact revision on arXiv

Source-linked AI summary

ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes

Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, Angela Dai

arXiv:2308.11417v1cs.CV

TL;DR

Existing 3D scene datasets lack the combination of large-scale coverage, high-fidelity capture, and comprehensive ambiguous semantic annotations needed for broad real-world evaluation. ScanNet++ pairs laser scans with registered DSLR and iPhone data, then provides benchmarks for novel view synthesis and semantic understanding. Its precise ground truth supports quantitative evaluation across diverse scenes, while capture cost and fixed DSLR brightness constrain scalability and exposure robustness.

  • Problem

    Existing datasets lack a broad combination of large-scale scenes, high-quality geometry and color, and comprehensive semantic annotations for novel view synthesis and 3D understanding.

  • Method

    ScanNet++ constructs registered indoor-scene captures from laser scans, DSLR images, iPhone RGB-D video, and open-vocabulary semantic annotations that explicitly represent ambiguity.

  • Results

    ScanNet++ supports challenging benchmarks for novel view synthesis and semantic understanding using precise real-world ground truth across diverse scenes.

  • Takeaways & Limitations

    The dataset supports high-quality and commodity-level NVS, cross-scene generalization, and fine-grained semantic understanding with label uncertainty.

  • Takeaways & Limitations

    Fixed DSLR brightness can cause overexposure or underexposure, and expensive collection prevents scaling at the rate of 2D datasets.

Abstract

from arXiv · show

We present ScanNet++, a large-scale dataset that couples together capture of high-quality and commodity-level geometry and color of indoor scenes. Each scene is captured with a high-end laser scanner at sub-millimeter resolution, along with registered 33-megapixel images from a DSLR camera, and RGB-D streams from an iPhone. Scene reconstructions are further annotated with an open vocabulary of semantics, with label-ambiguous scenarios explicitly annotated for comprehensive semantic understanding. ScanNet++ enables a new real-world benchmark for novel view synthesis, both from high-quality RGB capture, and importantly also from commodity-level images, in addition to a new benchmark for 3D semantic scene understanding that comprehensively encapsulates diverse and ambiguous semantic labeling scenarios. Currently, ScanNet++ contains 460 scenes, 280,000 captured DSLR images, and over 3.7M iPhone RGBD frames.

1. Introduction

ScanNet++ addresses the lack of datasets combining large-scale coverage with high-quality geometry and color by providing registered high-end and commodity captures, dense open-vocabulary semantics, and benchmarks for novel view synthesis and semantic understanding.

  • Existing datasets typically trade off scene quantity against high-quality color and geometry capture.
  • ScanNet++ combines sub-millimeter laser geometry, 33-megapixel DSLR images, registered iPhone RGB-D video, and dense semantic annotations.All modalities are registered to support interaction between geometric, color, commodity-level, and high-end data.
  • The dataset enables quantitative evaluation against precise real-world ground truth for diverse scene scenarios.
  • ScanNet++ introduces benchmarks for novel view synthesis using both high-quality DSLR and commodity-level iPhone images.The novel view synthesis setting includes testing images captured independently from the scanning trajectory.
  • The dataset explicitly annotates label ambiguities and supports semantic understanding across an open vocabulary.

2. Related Work

Prior datasets provide either scale with limited capture quality or high-quality data at limited scale, while ScanNet++ combines high-resolution geometry, high-quality color, diverse scenes, and fine-grained ambiguous semantics for more practical evaluation.

  • Earlier semantic datasets often use low-resolution RGB-D sequences with limited annotations, making small objects and details difficult to recognize and annotate.
  • ScanNet++ adds high-resolution laser geometry, high-quality color capture, and long-tail fine-grained multi-label annotations for ambiguous regions.
  • Existing novel view synthesis datasets commonly emphasize forward-facing, outside-in, synthetic, or small-scale scenes.Tanks and Temples, for example, has 7 training scenes and 14 test scenes.
  • ScanNet is useful for indoor NVS but suffers from motion blur, limited field of view, and testing poses subsampled from training trajectories.These properties produce a less suitable and potentially biased evaluation setting.
  • ScanNet++ provides higher-quality images across diverse real-world scenes, independent testing camera poses, and scale supporting research on cross-scene generalization.

3. Data Acquisition and Processing

ScanNet++ combines laser scanning, DSLR imaging, and iPhone RGB-D capture in a shared coordinate system, then produces high-resolution reconstructions with dense, ambiguity-aware semantic annotations.

  • Capture pipeline: The capture process uses a laser scanner, DSLR camera, and iPhone RGB-D video for each indoor scene.A typical scene takes around 30 minutes to capture, with larger scenes requiring more than two hours.
  • Capture pipeline: Laser scans provide dense point clouds for Poisson-reconstructed mesh surfaces.Each scan contains about 40 million points, with multiple scanner positions used to maximize surface coverage.
  • Capture pipeline: DSLR training images are captured with fixed lighting, wide field of view, and sharp imagery for photometric novel-view supervision.Training views follow a dense trajectory, while test poses differ in translation and orientation and may vary in height and angle.
  • Registration and alignment: COLMAP registration aligns DSLR and iPhone images with laser scans in a common coordinate system, with pose refinement and filtering of unreliable iPhone frames.Frames are filtered when their average depth difference from the rendered laser scan exceeds 0.3m.
  • Semantic annotation: Semantic labels are assigned to over-segmented meshes using free-text instance labels and can include multiple labels per segment.This captures ambiguity from occlusion and part-whole relations, such as a jacket covering a chair or a window within a door.

4. Experiments

Experiments benchmark novel-view synthesis from DSLR and iPhone inputs, cross-scene generalization, and semantic or instance segmentation. Results show useful general-prior gains but persistent failures on difficult appearance, pose, and object configurations.

  • Novel view synthesis: Novel-view synthesis is evaluated with DSLR and commodity RGB inputs using PSNR, LPIPS, and SSIM.NeRF and several state-of-the-art variants are evaluated on validation scenes.
  • Novel view synthesis: Nerfacto performs best among the compared DSLR methods, while NeRF oversmooths and TensoRF or Instant-NGP produce sharper but artifact-prone results.TensoRF shows stripes for substantially different test poses, Instant-NGP produces floaters, and Nerfacto can overfit view-dependent effects.
  • Novel view synthesis: Novel-view synthesis remains difficult for small objects and strong view-dependent effects.All methods fail to model light reflected on a table in one qualitative example.
  • Novel view synthesis: iPhone-trained synthesis performs significantly worse than DSLR-trained synthesis because of motion blur and varying brightness.Noisy structure-from-motion poses caused by blur and limited field of view add further difficulty.
  • Generalization across scenes: A pix2pix general prior learned across ScanNet++ scenes improves Nerfacto rendering quality over single-scene training.The experiment applies pix2pix to Nerfacto outputs while freezing Nerfacto weights.
  • 3D semantic understanding: Semantic and instance segmentation methods distinguish large separated objects but perform poorly on ambiguous or small objects.The semantic benchmark covers 78 classes, while the instance benchmark evaluates 75 object classes.

5. Limitations and Future Work

ScanNet++ supports large-scale DSLR-based novel-view research and multimodal radiance-semantic modeling, but its capture design imposes exposure and scalability limits.

  • Future work: Large-scale, high-quality DSLR captures support novel-view synthesis methods that generalize across multiple scenes.Registered DSLR images and semantic annotations also support combining radiance and semantic fields.
  • Limitations: Fixed DSLR brightness settings can overexpose light sources and underexpose poorly lit areas.The setting is used to ensure photometric consistency within each scene.
  • Limitations: The expensive collection process prevents ScanNet++ from scaling at the same rate as 2D datasets.

6. Conclusion

ScanNet++ provides high-fidelity multimodal indoor-scene data and challenging benchmarks for novel-view synthesis and semantic understanding. Its registered modalities, ambiguity-aware labels, and commodity capture support broader multimodal and generalization research.

  • Conclusion: ScanNet++ combines high-fidelity 3D geometry, high-resolution RGB, commodity iPhone capture, and semantic annotations into benchmarks for NVS and semantic understanding.
  • Conclusion: Independent iPhone capture introduces motion blur and noisy poses, while long-tail and multi-label annotations support fine-grained semantic understanding under label uncertainty.
  • Conclusion: Registering all modalities in one coordinate system supports multimodal semantic learning and semantic priors for novel-view synthesis.

A. Details of Data Collection

ScanNet++ captures large indoor spaces as unified scenes using a Faro laser scanner, Sony DSLR camera, and iPhone 13 Pro LiDAR sensor. The sensors are operated close together in time to reduce cross-modal lighting inconsistencies.

  • Captures target large spaces as single scenes rather than separate rooms, preserving more context and increasing downstream-task complexity.
  • The three sensors are operated as close together in time as possible to avoid lighting inconsistencies between modalities.
  • The hardware setup combines a Faro Focus Premium laser scanner, Sony Alpha 7 IV DSLR camera, and iPhone 13 Pro with LiDAR.

A.1. Laser Scan

The laser-scan pipeline merges and voxelizes point clouds at millimeter resolution, reconstructs overlapping chunks with Poisson surface reconstruction, and produces reduced meshes. DSLR capture is separately configured for consistent lighting and reliable novel-view-synthesis evaluation.

  • Laser Scan: The Faro scanner uses 1/4 resolution and 2x quality, requiring about 2 minutes per scan, with normals estimated from neighboring scan points.
  • Laser Scan: Point clouds are merged at 1mm resolution, divided into overlapping 0.5m cubes, and reconstructed per chunk using Poisson reconstruction at depth 9.
  • Laser Scan: Quadric edge collapse reduces the full-resolution mesh to 12.5%, 5%, or 1.5% of its original face count.
  • DSLR Capture: DSLR capture fixes white balance and exposure, uses a wide-field-of-view fisheye lens, and sets exposure time to 1/100s.

A.3. iPhone

Unlike the controlled DSLR setup, iPhone recordings use the phone’s default automatic mode, making novel view synthesis more challenging. The iPhone depth geometry is noisier and lower resolution than the laser-scanner reconstruction.

  • iPhone Capture: iPhone recordings use the default automatic mode rather than the controlled DSLR capture settings, increasing novel-view-synthesis difficulty.
  • Geometry Comparison: Laser-scanner geometry is much cleaner and preserves finer details than the 3D geometry generated from iPhone depth maps.
  • Capture Settings: Table 7 presents the capture-setting comparison between DSLR and iPhone.
  • Geometry Comparison: Figure 10 compares laser-scan reconstructions with the iPhone’s low-resolution point clouds, whose LiDAR depth images are noisy.

B. Benchmark

ScanNet++ provides online benchmarks for novel view synthesis and 3D semantic and instance segmentation. Its evaluation emphasizes challenging unseen viewpoints, commodity-level inputs, and multilabeled semantic ground truth across more than 100 frequent object classes.

  • Benchmark Scope: The dataset will be released with an online benchmark covering novel view synthesis, semantic segmentation, and instance segmentation.
  • Novel View Synthesis: Novel view synthesis evaluates images rendered at unseen camera poses that differ substantially from the training trajectory, using PSNR, LPIPS, and SSIM.
  • Novel View Synthesis: DSLR images serve as ground truth for iPhone-trained methods, testing high-quality output from commodity-level inputs.
  • Semantic and Instance Segmentation: Semantic and instance segmentation are evaluated on colored meshes and posed RGB images using vertex labels, instance masks, IoU, mean IoU, AP25, AP50, and AP.
  • Semantic and Instance Segmentation: More than 100 frequent object classes use multilabeled ground truth, allowing submissions to provide multiple predictions per vertex.
  • Evaluation Protocol: A hidden test set and public evaluation website support comparisons, while separate test sets prevent semantic input meshes from guiding novel view synthesis.
  • Semantic Annotation: Annotators assign free-text labels to mesh segments, verify them in a second pass, and follow guidelines covering ambiguous multilabel cases.

D. Novel View Synthesis on iPhone Data

The iPhone-data novel view synthesis comparison uses DSLR ground truth, with output color correction applied before visual evaluation. Compared with DSLR results, commodity iPhone inputs produce more artifacts and blur.

  • Color correction based on optimal transport aligns iPhone-generated outputs with DSLR ground truth before comparison.
  • Compared with DSLR results, novel view synthesis from commodity iPhone data produces more artifacts and blur.
  • Figure 13 presents qualitative comparisons of novel view synthesis methods on iPhone data against DSLR ground truth.
Loading 2308.11417v1…