Source-linked AI summary
ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, Elad Shulman
TL;DR
Indoor scene understanding needs datasets that better match the diversity of real homes and the mobile depth sensors used today. ARKitScenes provides large-scale Apple LiDAR RGB-D data with high-quality depth and furniture annotations, then evaluates it on 3D object detection and color-guided depth upsampling. The dataset advances state-of-the-art methods while revealing generalization limitations in realistic scenarios.
Problem
Existing indoor datasets lack sufficient diversity and often use depth hardware unlike widely available mobile sensors, creating a gap for realistic scene-understanding research.
Method
ARKitScenes collects Apple LiDAR RGB-D sequences with poses and reconstructions, adds laser-scanner depth and oriented furniture boxes, and evaluates two downstream tasks.
Results
The dataset advances state-of-the-art methods while highlighting their limitations in generalizing to realistic scenarios.
Takeaways & Limitations
ARKitScenes is intended to support novel algorithms and evaluation on diverse homes captured with a mobile RGB-D sensor.
Takeaways & Limitations
Lighting can change during the roughly six-hour collection of a venue, causing potentially inconsistent illumination across sequences and scans.
Abstract
from arXiv · showhide
Scene understanding is an active research area. Commercial depth sensors, such as Kinect, have enabled the release of several RGB-D datasets over the past few years which spawned novel methods in 3D scene understanding. More recently with the launch of the LiDAR sensor in Apple's iPads and iPhones, high quality RGB-D data is accessible to millions of people on a device they commonly use. This opens a whole new era in scene understanding for the Computer Vision community as well as app developers. The fundamental research in scene understanding together with the advances in machine learning can now impact people's everyday experiences. However, transforming these scene understanding methods to real-world experiences requires additional innovation and development. In this paper we introduce ARKitScenes. It is not only the first RGB-D dataset that is captured with a now widely available depth sensor, but to our best knowledge, it also is the largest indoor scene understanding data released. In addition to the raw and processed data from the mobile device, ARKitScenes includes high resolution depth maps captured using a stationary laser scanner, as well as manually labeled 3D oriented bounding boxes for a large taxonomy of furniture. We further analyze the usefulness of the data for two downstream tasks: 3D object detection and color-guided depth upsampling. We demonstrate that our dataset can help push the boundaries of existing state-of-the-art methods and it introduces new challenges that better represent real-world scenarios.
1 Introduction
ARKitScenes addresses the mismatch between existing indoor RGB-D datasets and widely available mobile depth sensors by providing large-scale, diverse home captures with rich geometric annotations. It also demonstrates the dataset’s usefulness for advancing scene-understanding methods and evaluating realistic scenarios.
- Motivation: Existing indoor datasets often lack diversity and use hardware unlike today’s consumer depth sensors, limiting practical generalization.The paper identifies both data diversity and depth-sensing technology gaps as challenges for applying recent research to day-to-day use.
- Dataset contribution: ARKitScenes is the first large-scale RGB-D dataset captured with Apple’s LiDAR scanner on handheld devices.It is presented as the largest RGB-D dataset in terms of sequences and scene diversity collected in people’s homes.
- Dataset contribution: 5,048 RGB-D sequences cover 1,661 unique scenes, exceeding the current largest available indoor dataset by more than three times.The dataset includes estimated camera poses and LiDAR-based ARKit scene reconstructions for all sequences.
- Dataset contribution: ARKitScenes provides per-frame RGB, depth, IMU, camera poses, and scene reconstructions from iPad Pro sequences.These assets combine raw and processed mobile-device data with reconstructed scene geometry.
- Dataset contribution: High-quality ground truth includes RGB-registered depth and oriented 3D bounding boxes for room-defining objects.The dataset’s annotations support realistic geometric supervision for indoor scene understanding.
- Impact: The dataset advances state-of-the-art methods while exposing limitations in current methods and datasets when generalizing to realistic scenarios.The authors position ARKitScenes as a resource for developing algorithms that better represent real-world use.
2 Related work
Prior indoor RGB-D datasets enabled scene-understanding research but often lacked viewpoint diversity, scene-level labels, oriented boxes, or depth sensor data. ARKitScenes addresses these gaps with larger, more diverse mobile captures and additional annotations.
- Indoor scene understanding: ScanNet provides 1,513 scans of 707 unique scenes, while ARKitScenes contains three times as many captures and 1,661 unique scenes.ARKitScenes also uses Apple’s LiDAR scanner rather than Kinect and provides assets absent from ScanNet.
- Indoor scene understanding: Sun RGB-D offers over 10,000 RGB-D frames but lacks scene-level labels and viewpoint diversity because its labels are frame-level.Long indoor captures were introduced to address viewpoint diversity.
- 3D object detection: Indoor 3D detection datasets remain limited: SunRGB-D lacks scene-level labels, whereas ScanNet lacks oriented 3D bounding boxes.Many outdoor detection assumptions, such as Bird’s Eye View projection, do not generalize well to indoor scenes.
- 3D object detection: ARKitScenes provides the largest set of 3D oriented bounding boxes for 17 room-defining object categories while retaining depth sensor data.This addresses gaps in datasets that provide 3D labels without depth sensor data.
- Color-guided depth upsampling: Color-guided depth upsampling generates high-resolution depth from a high-resolution color image and a low-resolution depth map, but prior datasets provide few high-resolution ground-truth images.Examples include 34 Middlebury images and 58 synthetic MPI-Sintel images.
3 ARKitScenes dataset
ARKitScenes combines mobile RGB-D capture in real homes with stationary laser scans, automatic registration, dense ground-truth depth, and manually annotated oriented 3D boxes. Its released data supports whole-scene and per-frame detection evaluations.
- Data collection: ARKitScenes acquisition combines raw mobile sensor data, stationary laser scans, automatic spatial registration, and manual 3D bounding-box annotation.The 2020 iPad Pro records IMU, Wide and Ultra Wide RGB, and LiDAR depth, while a Faro Focus S70 captures XYZRGB point clouds.
- Data collection: Data collection spans rural, suburban, and urban homes across London, Newcastle, and Warsaw, including low-, medium-, and high-SES households.These selection criteria were intended to increase indoor scene diversity and coverage.
- Ground truth generation: Laser-scan triangulation creates a watertight textured mesh whose foreground and occlusion triangles support unobstructed-line-of-sight rendering.The triangle sets are separated using the angle between each triangle normal and its ray to the scanner’s nodal point.
- Ground truth generation: Local feature matching between camera keyframes and laser-scan renderings estimates camera poses, enabling frame-aligned orthographic ground-truth depth maps.The refined poses and dense depth maps are provided in the dataset.
- Dataset split: The dataset includes 5,048 released sequences, with venue-level splits assigning 80% to training, 10% to validation, and 10% to a held-out test set.All scans and iPad sequences from a venue remain in the same split, shared across downstream tasks.
4 Tasks and benchmarks
ARKitScenes is evaluated on indoor 3D object detection and color-guided depth upsampling, using realistic mobile LiDAR data and adapted training procedures. The benchmarks show strong performance gains for learned upsampling methods and state-of-the-art detection models, while exposing challenges from real-world depth artifacts and small objects.
- 3D object detection: ARKitScenes evaluates state-of-the-art models on single-frame and whole-scene 3D object detection.Single-frame detection operates on individual RGB-D frames, whereas whole-scene detection uses the reconstructed 3D scene.
- Color-guided depth upsampling: ARKitScenes makes depth upsampling a twofold challenge: increasing resolution and correcting artifacts from mobile LiDAR sensing.Prior work commonly evaluates low-resolution maps produced by downsampling high-resolution ground truth, which does not reproduce these sensing artifacts.
- Color-guided depth upsampling: Occluded regions in projected laser-scan ground truth require adaptations to existing losses and handling of missing-depth pixels.The SSIM loss from MSPF cannot be applied directly, and its image-wide edge loss is replaced with a more robust alternative.
- Color-guided depth upsampling: MSG and MSPF produce sharper edges and more realistic structure than classical upsampling methods on ARKitScenes.The comparison includes Bilinear interpolation, JBU, FGI, MSG, and MSPF, with ℓ1 and RMSE results supporting the learned methods’ advantage.
5 Conclusions
The paper presents ARKitScenes as the first Apple LiDAR indoor RGB-D dataset and the largest indoor mobile RGB-D dataset collected to that point. Its downstream evaluations show that the data supports object detection and depth upsampling while targeting better real-world generalization.
- ARKitScenes is the first dataset captured with Apple’s LiDAR scanner and the largest indoor RGB-D dataset collected with a mobile device.
- The dataset is demonstrated on 3D object detection and color-guided depth upsampling.
- ARKitScenes is intended to support methods that generalize better to real-world scenarios.