Source-linked AI summary
RGBD Datasets: Past, Present and Future
Michael Firman
TL;DR
RGBD research lacks a consolidated account of available datasets and the gaps that remain across tasks. This paper surveys datasets across eight categories, extracts information to guide data selection, and examines future directions. It finds a diverse and growing field with opportunities in synthetic data and dense reconstructions of static and dynamic scenes, while synthetic data remains limited in its fidelity to the real world.
Problem
Researchers need a consolidated view of RGBD datasets and underexplored areas to select appropriate data and compare progress across tasks.
Method
The paper reviews state-of-the-art RGBD datasets across eight categories and extracts relevant information for dataset selection and comparison.
Results
The field contains considerable, diverse RGBD dataset coverage, while future opportunities include synthetic data and dense reconstructions of static and dynamic scenes.
Takeaways & Limitations
RGBD datasets can support transparent comparisons and research in underexplored directions, with future collection increasingly involving dense static and dynamic scene reconstructions.
Takeaways & Limitations
Synthetic scenes may not capture the true properties of the real world, and turntable datasets differ from real scenes in occlusion, lighting, and object-distance variation.
Abstract
from arXiv · showhide
Since the launch of the Microsoft Kinect, scores of RGBD datasets have been released. These have propelled advances in areas from reconstruction to gesture recognition. In this paper we explore the field, reviewing datasets across eight categories: semantics, object pose estimation, camera tracking, scene reconstruction, object tracking, human actions, faces and identification. By extracting relevant information in each category we help researchers to find appropriate data for their needs, and we consider which datasets have succeeded in driving computer vision forward and why. Finally, we examine the future of RGBD datasets. We identify key areas which are currently underexplored, and suggest that future directions may include synthetic data and dense reconstructions of static and dynamic scenes.
1. Introduction
RGBD dataset collection became more accessible after the Kinect, but assembling and labeling useful data remains difficult. Public datasets improve comparability and coverage, while the field is expanding toward dense reconstructions of static and dynamic scenes.
- Motivation: Before the Kinect, depth datasets were costly to collect and typically contained small, isolated static objects or laboratory scenes.Researchers used custom active stereo systems and expensive 3D scanners, whose lack of portability limited collection settings.
- Motivation: Kinect-era datasets became larger and more ambitious, while annual RGBD dataset releases continued to increase.Examples include video-based semantic labels, dense object-surface reconstructions, and synthetic scene generation.
- Motivation: Despite available sensors, collecting RGBD data remains nontrivial because researchers still develop battery devices, drivers, and custom formats.These engineering requirements constrain practical data collection beyond simply owning a depth sensor.
- Why datasets matter: Public RGBD datasets remove repeated capture, make results transparent, enable comparison on common data, and can direct research toward underexplored areas.Shared datasets can also create competition for better-performing algorithms.
- Paper contribution: The review catalogs 101 public datasets, substantially broadening earlier surveys and identifying opportunities for new data.The paper compares this coverage with earlier efforts describing 14, 19, and 44 datasets.
2. State-of-the-art in RGBD datasets
The paper reviews state-of-the-art RGBD datasets across eight categories using explicit inclusion criteria. It organizes the survey from isolated objects through tracking and reconstruction to human-focused datasets.
- Scope: The review covers RGBD datasets across eight categories, with some datasets appearing in multiple categories depending on content and labeling.The categories include objects, camera tracking, scene reconstruction, object pose, semantics, object tracking, actions, faces, and identity recognition.
- Inclusion criteria: The survey includes actively captured depth data such as time-of-flight and structured light, while excluding passive stereo and lidar datasets.It focuses on commodity depth capture rather than those separate sensing regimes.
- Inclusion criteria: The review prioritizes datasets with human-applied labeling and excludes very small datasets and datasets created mainly to demonstrate acquisition systems.These criteria follow the paper’s emphasis on information-bearing data rather than raw data volume.
- Organization: The survey proceeds from isolated objects to camera tracking, scene reconstruction, object pose, semantics, tracking, actions, gestures, faces, and identity recognition.This ordering provides the structure for the state-of-the-art review.
2.1. Objects in isolation
RGBD turntable datasets provide multiple unoccluded views of isolated objects and support recognition tasks, but their controlled conditions differ substantially from real-world scenes. Newer datasets improve scale or camera-pose accuracy while retaining different trade-offs.
- Turntable datasets: Turntable datasets capture multiple unoccluded views of the same object from different angles.Their controlled viewpoints make them useful for object-centered RGBD research.
- Representative datasets: The 2011 RGB-D Object Dataset contains 300 objects but lacks accurate camera poses, whereas BigBIRD uses calibrated Kinects and DSLRs to address this issue.BigBIRD is smaller, so the datasets trade scale against pose calibration.
- Limitations: Turntable datasets support tasks such as object detection and discovery, but their lack of occlusion, lighting variation, and distance variation creates a domain gap from real scenes.The paper identifies these controlled conditions as a limitation for understanding real-world environments.
- Representative datasets: A real-world object dataset contains 10,000 items ranging from books to cars, making it two orders of magnitude larger than earlier real-life object collections.The dataset exploits improved camera tracking to scan individual objects in real-world settings.
2.2. Camera tracking and scene reconstruction
Camera tracking and dense reconstruction require reliable ground-truth poses, which are difficult to obtain with external hardware. Existing benchmarks combine motion-tracking systems, synthetic control, or specialized scans, each with distinct coverage and realism trade-offs.
- Ground truth: Ground-truth camera poses are necessary for validating tracking algorithms but difficult to acquire because they require external hardware.The paper links advances in camera tracking and dense reconstruction to this ground-truth challenge.
- Camera tracking benchmarks: The TUM benchmark is a de-facto standard with motion-tracking ground truth, varied scenes, and varied camera motions.The review summarizes TUM alongside similar camera-tracking datasets.
- Ground-truth alternatives: Kinect-derived manually verified poses suit tasks an order of magnitude harder than tracking, such as camera relocalization or voxel occupancy prediction.They are not presented as equivalent ground truth for ordinary tracking evaluation.
- Synthetic data: Synthetic data can avoid difficult ground-truth acquisition while allowing controlled camera trajectories, motion blur, and other scene factors.ICL-NUIM provides eight trajectories across two synthetic indoor scenes, using paths taken from real handheld camera trajectories.
- Real-world reconstruction: A real-world dataset combines ground-truth camera motions with laser-scanned reconstructions, but its scenes are less diverse than TUM.The paper identifies it as the only known real-world dataset with both forms of ground truth.
- Voxel occupancy: Tabletop objects scanned on every visible surface provide ground truth for estimating unobserved voxel occupancy from a depth image.The reconstruction design supplies complete surface observations for this task.
2.3. Object pose estimation
Object pose datasets address 6-DoF alignment using object models and pose annotations, but ground truth remains difficult, especially at room scale.
- 2.3. Object pose estimation: Object pose estimation seeks the transformation that best aligns a known 3D object model with an observed scene.Depth cameras aid this task by providing absolute scale.
- 2.3. Object pose estimation: Most summarized datasets contain tabletop-sized objects because acquiring models and ground-truth poses for larger objects is difficult.Room-scale studies instead use alternative evaluation methods or human annotations as approximate ground truth.
- 2.3. Object pose estimation: Synthetic data is identified as a possible avenue for addressing the difficulty of obtaining large-object pose annotations.
2.4. Semantic labeling
Semantic RGBD datasets support increasingly general scene understanding, but the reviewed datasets depict static worlds even when labels are propagated through video.
- 2.4. Semantic labeling: Semantic datasets provide labels for images and videos, with realism discretized into three categories for comparison.A low realism score does not imply a dataset is worse or less useful, because constructed scenarios can provide high-quality ground truth.
- 2.4. Semantic labeling: The 1449-frame densely labeled NYUv2 subset became a de-facto standard for indoor scene labeling because of its label quality and variety.
- 2.4. Semantic labeling: SUN3D extends beyond NYUv2’s single static-frame modality by propagating object labels through Kinect videos, but includes only 8 annotated sequences.
- 2.4. Semantic labeling: All reviewed semantic datasets depict a static world, contrasting with the dynamic worlds represented by tracking datasets.
2.5. Tracking
RGBD tracking datasets cover dynamic objects and humans, while related semantic and pose datasets can also support tracking; activity data remains biased toward controlled settings.
- 2.5. Tracking: Only four datasets are explicitly designed for tracking, all using bounding-box annotations for moving objects or humans.They include the Princeton Tracking Benchmark, RGB-D people, Kinect Tracking Precision, and RGBD Pedestrian datasets.
- 2.5. Tracking: Some semantic and 6-DOF pose datasets also provide labels that can support tracking tasks.These include objects labeled through video and object pose annotations.
- 2.5. Tracking: Activity and gesture datasets cover sign language, hand gestures, daily actions, falls, and specialized activities such as salad preparation.
- 2.5. Tracking: Many activity datasets were recorded in offices or laboratories with researchers performing the actions, motivating data collection from people at work and home.The paper links such settings to dataset bias and less believable recognition baselines.
2.7. Faces
RGBD face research has expanded from small or acquisition-focused datasets to identity recognition, pose regression, and expression or emotion inference.
- 2.7. Faces: Face datasets now support identity recognition, pose regression, and expression or emotion inference beyond early small or acquisition-focused collections.The paper summarizes these datasets in Table 6.
2.8. Recognition
Human-recognition datasets use activities such as walking as input, but target subject attributes such as identity or gender rather than the activity itself.
- Human-recognition datasets record activities such as walking while labeling identity, gender, or other subject attributes.
3. Future areas for datasets
Future RGBD datasets should address underexplored gaps through synthetic data and richer geometric representations, including dense reconstructions with surface- and voxel-level labels, dynamic geometry, and correspondences.
- 3.1. Synthetic data: Synthetic RGBD data offers perfect task ground truth and controllable scene parameters without expensive human labeling.It can support segmentation, reconstruction, tracking, and camera or object pose tasks.
- 3.1. Synthetic data: Synthetic scenes may fail to capture the true properties of the real world, even when sensor noise is emulated.
- 3.2. Full voxel occupancy: Most semantic datasets expose only 2.5D views, while full scene surface geometry remains difficult to capture in large real-world environments.
- 3.2. Full voxel occupancy: Dense reconstructions with labeled surfaces would enable semantic segmentation directly on meshes rather than only on images or videos.
- 3.2. Full voxel occupancy: Voxel-level semantic labeling would require every scene voxel to be annotated, potentially making synthetic data necessary because manual labeling is difficult.
- 3.3. Dynamic scenes: RGBD datasets with ground-truth dense geometry for dynamic scenes are nearly absent, leaving deforming objects and frame correspondences as an open challenge.
4. Conclusion
The review finds a diverse RGBD dataset landscape and argues that improved reconstruction and labeling now enable datasets of dense static and ultimately dynamic scenes.
- The RGBD dataset field is diverse across applications, suggesting depth information is useful in many sectors.
- Most reviewed datasets use single frames or videos from static cameras, but improved reconstruction and labeling create opportunities for dense static and dynamic scene datasets.