PaperlayerView exact revision on arXiv

Source-linked AI summary

DIML/CVL RGB-D Dataset: 2M RGB-D Images of Natural Indoor and Outdoor Scenes

Jaehoon Cho, Dongbo Min, Youngjung Kim, Kwanghoon Sohn

arXiv:2110.11590v1cs.CV

TL;DR

Outdoor depth acquisition is challenged by strong illumination and longer ranges, while existing options described here have limitations for dense natural-scene depth. DIML/CVL presents 2M RGB-D images across indoor and outdoor scenes, constructed with Kinect v2 indoors and stereo cameras outdoors. The authors anticipate the dataset’s scale and quality could help bridge indoor and outdoor simulations for depth-related tasks.

  • Problem

    Outdoor depth acquisition is challenged by strong illumination and longer ranges; LiDAR produces sparse maps, while synthetic ground truth incurs domain adaptation issues for natural outdoor scenes.

    Source: §III · p. 4

  • Method

    The dataset combines indoor Kinect v2 RGB-D capture with hand-held outdoor stereo acquisition and provides confidence maps for outdoor stereo depth.

    Source: 1) Hardware setup · p. 1, §I · p. 1, 3) Confidence Estimation · p. 6

  • Results

    DIML/CVL provides 1M indoor and 1M outdoor RGB-D data, for 2M total images.

    Source: §I · p. 1

  • Takeaways & Limitations

    The authors anticipate that the dataset’s scale and quality could help bridge indoor and outdoor simulations for depth-related tasks.

    Source: §IV · p. 6

    The comprehensive performance analysis was planned for release with the complete 2M RGB-D data, after an initial subset release.

    Source: §I · p. 1

Abstract

from arXiv · show

This manual is intended to provide a detailed description of the DIML/CVL RGB-D dataset. This dataset is comprised of 2M color images and their corresponding depth maps from a great variety of natural indoor and outdoor scenes. The indoor dataset was constructed using the Microsoft Kinect v2, while the outdoor dataset was built using the stereo cameras (ZED stereo camera and built-in stereo camera). Table I summarizes the details of our dataset, including acquisition, processing, format, and toolbox. Refer to Section II and III for more details.

I. OVERVIEW

DIML/CVL comprises indoor and outdoor RGB-D data collected in South Korea from fall 2015 to summer 2017, with 1M images in each setting. Indoor capture used a mobile Kinect v2 setup and covered varied commercial and residential scenes.

I. OVERVIEW

  • The dataset was collected in South Korea from fall 2015 to summer 2017 to promote depth-related applications based on deep learning.A subset was released before the analysis was complete; the comprehensive performance report was planned for release with the complete 2M data.Source: §I · p. 1
  • The dataset provides 1M indoor and 1M outdoor RGB-D data, captured outdoors with hand-held stereo cameras and accompanied by confidence maps.The maps quantify the accuracy of outdoor stereo depth maps.Source: §I · p. 1

1) Hardware setup:

  • Indoor RGB-D videos were captured with a Microsoft Kinect v2, whose external battery enhanced mobility in the hand-held setup.A tripod and slider were also used, and the average camera velocity was 10 frames per second to avoid motion blur in color images.Source: 1) Hardware setup · p. 1, Figure 1 · p. 1
  • Indoor scenes span 18 categories and 283 scenes across nine districts in four cities.The color and depth streams were originally captured at 1920×1080 and 512×424, respectively; depth measurements were limited to 0.5–7 meters due to near- and far-range inaccuracies.Source: 1) Hardware setup · p. 1

B. Data Processing

Indoor processing co-aligns color images and depth maps by projecting raw depth into RGB coordinates, then filling holes with color-guided depth upsampling. Calibration used either Kinect SDK or the ‘iai kinect v2 libary,’ with differing warped-map sizes and crop outputs.

B. Data Processing

  • Indoor processing projects raw depth into RGB coordinates and then fills holes in the warped map using color-guided depth upsampling.This is the overall processing sequence.Source: B. Data Processing · p. 1

1) Calibration:

  • The Kinect SDK processed approximately 476,000 image pairs, while the ‘iai kinect v2 libary’ processed the remaining pairs.The SDK warped maps to 480×270 RGB coordinates; the library warped them into cropped RGB coordinates, after which regions outside the IR camera field of view were discarded and color images were cropped to 1408×792.Source: 1) Calibration · pp. 1–2
  • Calibration estimated the IR-to-depth shift and IR-to-color projection matrix, then warped depth through IR coordinates into RGB coordinates.Checkerboard images used a 5×7 pattern with 0.03 m corner spacing; images should cover most of the frame at varied ranges for accurate calibration.Source: 1) Calibration · p. 2

2) Depth upsampling:

Indoor depth-map holes were filled using a color-guided depth upsampling algorithm.

2) Depth upsampling:

C. Data Format

Indoor depth maps are stored in millimeters as 16-bit PNG files, alongside cropped RGB images and multiple depth-map versions. File identifiers encode camera, date, and acquisition order.

C. Data Format

  • Indoor depth maps are stored in millimeters using 16-bit PNG files.Source: C. Data Format · p. 2
  • The indoor color image is a 24-bit cropped RGB image at 1408×792, while the original depth map is 512×424 and projected depth maps are 480×270 or 1408×792.The filled depth map is 1408×792.Source: C. Data Format · pp. 2–3
  • Indoor RGB-D image-pair identifiers encode a two-digit camera number, six-digit date, and six-digit acquisition order.Four Kinect v2 cameras were used.Source: C. Data Format · p. 3

D. Indoor data toolbox

The indoor toolbox fills holes in depth maps, while the outdoor dataset uses stereo-based depth estimation with confidence maps for natural scenes.

D. Indoor data toolbox

  • The indoor toolbox function “fill depth.m” fills holes in the depth map.Source: D. Indoor data toolbox • · p. 3
  • Indoor scenes were captured across 9 districts in 4 South Korean cities, including offices, rooms, dormitories, exhibition centers, and streets.Source: Figure 2 · p. 3
  • Indoor calibration used at least 150 images with the pattern tilted vertically and horizontally to cover all image areas.Source: Figure 4 · p. 4

III. OUTDOOR RGB-D DATASET

  • Strong illumination and longer outdoor ranges make Kinect v2 infeasible, while LiDAR depth is sparse and accumulating scans requires substantial manual cleanup.The cited passage notes LiDAR covers less than 6% of pixels in one example and has a limited vertical field of view.Source: §III · pp. 3–4
  • Synthetic dense ground-truth depth maps have a domain-adaptation issue when directly used for natural outdoor scenes.Source: §III · p. 4
  • Stereo matching generates outdoor depth maps, with an associated confidence map provided to reduce the side effects of incorrect estimates.The dataset uses the stereo-based approach for large-scale outdoor RGB-D acquisition.Source: §III · p. 4

1) Hardware setup:

Outdoor data were acquired with ZED and built-in stereo cameras, and the dataset covers varied scenes collected with a hand-held setup.

1) Hardware setup:

  • The built-in stereo system uses a 40 cm baseline for a sensing range up to 80 m, compared with the ZED camera’s stated range of up to 20 m.The ZED camera has a 12 cm baseline; the built-in system uses mvBlueFox3 sensors.Source: 1) Hardware setup · p. 4
  • The outdoor processing pipeline captures stereo images and generates depth and per-pixel confidence maps; stereo calibration includes distortion correction and rectification.Source: Figure 7 · p. 5, Figure 8 · p. 6

2) Data acquisition:

  • Tables IV and V summarize indoor and outdoor dataset categorization, respectively.Their captions distinguish data acquired using the SDK and data acquired with [12] using ZED, and note calibration data or data would be posted later.Source: Table IV · p. 5, Table V · p. 5
  • The dataset contains 0.92M ZED and 0.08M built-in-camera stereo images, captured at 1920×1080 or 1280×720 resolution.Average camera velocity was 10 frames per second to avoid motion blur.Source: 2) Data acquisition · p. 5
  • Hand-held acquisition covers 9 categories and 132 scenes across four South Korean cities, complementing road- and traffic-focused vehicle-mounted outdoor data.Scenes were captured steadily using a tripod and slider while walking.Source: 2) Data acquisition · p. 5

B. Data Processing

Outdoor processing includes stereo rectification, followed by disparity estimation and confidence-map generation.

B. Data Processing

  • The outdoor processing task rectifies stereo images and estimates a disparity map with its associated confidence map.Source: B. Data Processing · p. 6

1) Calibration and Rectification:

2) Stereo Matching:

Outdoor depth is derived from stereo disparity, confidence maps are estimated using two methods, and depth maps are stored in millimeters.

3) Confidence Estimation:

  • MC-CNN matching costs are filtered and processed with semi-global matching to produce disparity, then calibration parameters recover depth with a maximum disparity range of 228.Source: 3) Confidence Estimation · p. 6
  • Two methods, produce confidence maps; a threshold of 0.9 × 255 removed the majority of inaccurate depth values.Confidence is normalized to [0, 255], with higher values indicating better confidence; the methods were used for different acquisition periods.Source: 3) Confidence Estimation · p. 6

C. Data Format

  • Outdoor depth maps are stored in millimeters as 16-bit PNG files, alongside 8-bit disparity and confidence maps and rectified left and right RGB images.Source: C. Data Format · p. 6

D. Outdoor data toolbox

The outdoor toolbox computes confidence maps for stereo depth, while the concluding remarks identify training the dataset for broad vision tasks as future work.

D. Outdoor data toolbox

  • Confidence maps for 400,000 stereo pairs were computed using, and the remainder using.Method takes a cost volume and its associated disparity map, computed using, as inputs.Source: D. Outdoor data toolbox • · p. 6

IV. CONCLUDING REMARKS

  • The authors plan to demonstrate successful training of large DNNs on the dataset across computer-vision and image-processing applications.They anticipate its scale and quality could help bridge indoor and outdoor simulations, especially for depth-related tasks.Source: §IV · p. 6
  • The anticipated applications include monocular depth estimation, RGB-D feature learning, outdoor scene understanding, and dehazing.Source: §IV · p. 6
Loading 2110.11590v1…