PaperlayerView exact revision on arXiv

Source-linked AI summary

RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos

Hongchi Xia, Yang Fu, Sifei Liu, Xiaolong Wang

arXiv:2401.12592v3cs.CV

TL;DR

Real-world object-video datasets often lack depth, while synthetic data can be difficult to transfer to cluttered real scenes. WildRGB-D collects and annotates in-the-wild RGB-D object videos, then benchmarks four downstream tasks. The reported experiments find benefits for depth-informed reconstruction and label-free 6D pose learning, and show the dataset can support generalizable camera-pose estimation.

  • Problem

    Real-world object-video datasets may lack depth for accurate 3D annotations and downstream tasks, while synthetic data poses challenges for transfer to cluttered real scenes.

    Source: §1 · p. 2

  • Method

    WildRGB-D collects iPhone RGB-D object videos in 360-degree views and automatically annotates masks, real-world-scale camera poses, and aggregated point clouds.

    Source: §1 · p. 2

  • Results

    The dataset can serve as training data for generalizable camera-pose estimation; depth improves surface-reconstruction precision, and its RGB-D images support self-supervised 6D pose estimation without training labels.

    Source: §1 · p. 3

  • Takeaways & Limitations

    WildRGB-D provides a real-world RGB-D resource for benchmarking novel-view synthesis, camera-pose estimation, 6D pose estimation, and surface reconstruction.

    Source: §1 · p. 2

Abstract

from arXiv · show

We introduce a new RGB-D object dataset captured in the wild called WildRGB-D. Unlike most existing real-world object-centric datasets which only come with RGB capturing, the direct capture of the depth channel allows better 3D annotations and broader downstream applications. WildRGB-D comprises large-scale category-level RGB-D object videos, which are taken using an iPhone to go around the objects in 360 degrees. It contains around 8500 recorded objects and nearly 20000 RGB-D videos across 46 common object categories. These videos are taken with diverse cluttered backgrounds with three setups to cover as many real-world scenarios as possible: (i) a single object in one video; (ii) multiple objects in one video; and (iii) an object with a static hand in one video. The dataset is annotated with object masks, real-world scale camera poses, and reconstructed aggregated point clouds from RGBD videos. We benchmark four tasks with WildRGB-D including novel view synthesis, camera pose estimation, object 6d pose estimation, and object surface reconstruction. Our experiments show that the large-scale capture of RGB-D objects provides a large potential to advance 3D object learning. Our project page is https://wildrgbd.github.io/.

1. Introduction

WildRGB-D addresses the limited depth and real-world coverage of existing object-video datasets with category-level RGB-D videos captured across diverse scenes. Its benchmarks report benefits for generalizable camera-pose estimation, depth-informed surface reconstruction, and self-supervised 6D pose estimation.

1. Introduction

  • Synthetic data can be difficult to transfer to real scenes, while RGB-only real-world datasets may lack depth-based annotations, clutter, or broad object-view coverage.CO3D relies on COLMAP for annotations that work on only 20% of collected videos; Wild6D covers six categories and a smaller range of object views.Source: §1 · p. 2
  • WildRGB-D collects category-level RGB-D object videos using iPhones to circle objects through 360 degrees.The introduction reports 8,500 tabletop objects across 44 categories in 20K videos.Source: §1 · p. 2
  • The dataset includes single-object, multi-object, and static-hand videos to represent clutter and human-hand occlusion in real-world scenes.The paper identifies these varied occlusion cases as useful study settings for some tasks.Source: §1 · p. 2
  • Automatic annotations use RGB-D SLAM for real-world-scale camera poses and aggregated point clouds, with detection, segmentation, and tracking for object masks.The described mask pipeline combines Grounding-DINO, Segment-Anything, and XMem.Source: §1 · p. 2
  • The paper benchmarks novel-view synthesis, sparse relative camera-pose estimation, object surface reconstruction, and object 6D pose estimation.The benchmarks include depth-assisted NeRF training and generalization to unseen categories.Source: §1 · pp. 2–3
  • WildRGB-D can serve as training data for generalizable camera-pose estimation; depth improves reconstruction precision, and the dataset supports label-free 6D pose learning.For surface reconstruction, SDF-based Neusfacto performs better in this setting; the pose model is evaluated on Wild6D.Source: §1 · p. 3

2. Related Work

Related work spans synthetic and real-world object datasets, neural scene representations, sparse camera-pose estimation, and category-level 6D pose estimation. WildRGB-D is positioned as a real-world RGB-D resource with full 360-degree videos and annotations relevant to these research directions.

3D Object Datasets

  • Existing object datasets range from synthetic category-level collections to real-world datasets with specialized scopes, while WildRGB-D combines 360-degree RGB-D videos with real-world-scale camera poses, masks, and aggregated point clouds.Prior resources include category-specific datasets, multiview-stereo datasets without category-level classification, and specialized-hardware scans without real captured RGB-D wild videos or background depths.Source: §2 · p. 3

Neural Radiance Field and Object Surface Reconstruction

  • NeRF maps sampled points along each ray to density and color for volume-rendered views, while SDF-based methods represent scenes as implicit surfaces with clearer object boundaries.The cited work also describes approaches that improve visual quality or efficiency, generalize representations across scenes, and use 3D Gaussian Splatting as an alternative.Source: §2 · p. 3

Camera Pose Estimation

  • Sparse-view camera-pose estimation remains challenging, and WildRGB-D's full 360-degree multiview videos provide a large-scale training database for these algorithms.The related work contrasts sparse-view methods with mature SfM and SLAM approaches for dense views.Source: §2 · pp. 3–4

Object 6D Pose Estimation

  • Category-level 6D pose methods include NOCS estimation, direct regression, keypoint estimation, and self-supervision; WildRGB-D's category-level RGB-D images are described as potential self-supervised training data.The cited self-supervised approach learns correspondences across 2D and 3D without annotations.Source: §2 · p. 4

3. The WildRGB-D Dataset

WildRGB-D captures iPhone RGB-D videos by circling objects in 360 degrees, using single-object, multi-object, and hand-object setups. Its annotations include real-world-scale camera poses, point clouds, and central object masks; the final collection contains 8,367 objects in 23,049 videos.

3.1. Data Collections, Processing, and Annotation

  • iPhones record RGB and corresponding depth while rotating around each object, with three videos per object: single-object, multi-object, and hand-object.The capture setup controls rotation speed to reduce blur and selects 46 common categories; videos with poor SLAM pose estimates are excluded.Source: §3.1 · p. 4
  • RGB-D SLAM uses captured depth to recover real-world-scale camera poses, which are projected with depth images into aggregated point clouds.Videos with inaccurate SLAM poses are excluded after manual quality checks; BAD SLAM and Open3D SLAM together raise the reported success rate to over 90%.Source: §3.1 · p. 4, Figure 5 · p. 5
  • Central object masks are generated by prompting SAM with Grounding-DINO boxes, then tracking the first-frame mask through each video with XMem..Source: §3.1 · p. 5

3.2. Statistics and Distribution

  • 8,367 objects across 23,049 videos remain after SLAM failures are excluded, with 33.1% single-object, 63.0% multi-object, and 3.9% hand-object videos.The reported object and video retention rates are 99.3% and 91.0%, respectively.Source: §3.2 · p. 5

4. Experiments

Experiments benchmark novel view synthesis, camera-pose estimation, object surface reconstruction, and category-level 6D pose estimation. Results report benefits from depth supervision for several reconstruction settings and assess generalization across scenes, categories, and datasets, while identifying limits in translation prediction and out-of-distribution pose evaluation.

4.1. Novel View Synthesis

  • For single-scene NVS, across ten selected scenes per category with uniformly sampled validation images, Mip-NeRF 360 and Instant-NGP outperform original NeRF on visual-quality metrics (PSNR, SSIM, and LPIPS).Source: §4.1 · p. 5
  • IBRNet outperforms PixelNeRF and MVSNeRF in visual-quality metrics across all three cross-scene difficulty levels.The experiments also report that learned geometry quality is not highly correlated with novel-view rendering quality.Source: Table 3 · p. 5, §4.1 · p. 6
  • Depth supervision improves Instant-NGP and PixelNeRF NVS performance, while the additional depth loss brings limited improvement to MVSNeRF, which already uses depth to guide ray construction.The experiments include single-scene and cross-scene NVS settings; the reported results describe depth priors as improving representations in both settings.Source: Table 4 · p. 5, §4.1 · p. 6

4.2. Camera Pose Estimation

  • RelPose and RelPose++ generalize to known and unseen categories with relative-rotation errors described as reasonable, but RelPose++ has comparatively large translation-prediction error.The evaluation uses sparse views and category-level training and testing splits.Source: §4.2 · pp. 6–7, Table 5 · p. 8, Table 6 · p. 8

4.3. RGBD Object Surface Reconstruction

  • RGB-D priors improve object surface reconstruction, and Neusfacto with RGB-D outperforms Instant-NGP, with depth helping the SDF-based method learn object boundaries.Evaluation uses Chamfer distance against aggregated object point clouds derived from object-masked depth images.Source: §4.3 · p. 7, Table 7 · p. 8

4.4. RGBD 6D Object Pose Estimation

  • Out-of-distribution 6D pose evaluation shows some decreased metrics when training only on WildRGB-D, while WildRGB-D training benefits IoU and joint-dataset training improves rotation-plus-translation evaluation in particular categories.The reported distribution difference is that Wild6D does not cover full 360 degrees, whereas WildRGB-D covers a larger pitch-angle range.Source: §4.4 · pp. 7–8, Table 8 · p. 8

5. Conclusion

WildRGB-D provides a large-scale collection of real-world RGB-D object videos, annotations, and four evaluation tracks. The dataset does not yet include object 6D pose annotations, which the authors identify as future work requiring further crowdsourcing.

5. Conclusion

  • WildRGB-D contains around 8,500 objects and nearly 20,000 RGB-D videos across 46 categories, with masks, real-world-scale camera poses, aggregated point clouds, and four evaluation tracks.The conclusion characterizes this large-scale capture as having potential to advance 3D object learning.Source: §5 · p. 8
  • The dataset lacks object 6D pose annotations; the authors identify crowdsourcing these annotations for supervised training and evaluation as future work..Source: §5 · p. 8
Loading 2401.12592v3…