PaperlayerView exact revision on arXiv

Source-linked AI summary

TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline

Hongjie Fang, Hao-Shu Fang, Sheng Xu, Cewu Lu

arXiv:2202.08471v2cs.ROcs.CV

TL;DR

Transparent-object reflection and refraction make ordinary RGB-D depth unreliable, limiting depth-dependent grasping. The paper builds the TransCG real-world dataset and proposes DFNet for depth completion. Experiments report improved cross-domain robustness and efficiency, and real-robot trials apply the method to novel-object grasping.

  • Problem

    Reflection and refraction make ordinary depth sensors produce inaccurate transparent-object depth, while synthetic datasets lack real sensor depth maps that matter in real-world applications.

    Source: §I · pp. 1–2

  • Method

    The paper builds TransCG, a real-world dataset with 57,715 RGB-D images from 130 scenes, and proposes DFNet to refine inaccurate depth using RGB and depth inputs.

    Source: §I · p. 2, A. Overview · p. 3

  • Results

    DFNet generalizes better across scenes, improves most reported metrics, and has the highest inference speed and least computation overhead among compared methods.

    Source: §I · p. 2

  • Takeaways & Limitations

    Cross-domain tests report TransCG-trained methods generalizing well to other testing datasets, and robot experiments demonstrate applicability to grasping novel transparent objects.

    Source: E. Discussion · p. 5, §VI · p. 7

Abstract

from arXiv · show

Transparent objects are common in our daily life and frequently handled in the automated production line. Robust vision-based robotic grasping and manipulation for these objects would be beneficial for automation. However, the majority of current grasping algorithms would fail in this case since they heavily rely on the depth image, while ordinary depth sensors usually fail to produce accurate depth information for transparent objects owing to the reflection and refraction of light. In this work, we address this issue by contributing a large-scale real-world dataset for transparent object depth completion, which contains 57,715 RGB-D images from 130 different scenes. Our dataset is the first large-scale, real-world dataset that provides ground truth depth, surface normals, transparent masks in diverse and cluttered scenes. Cross-domain experiments show that our dataset is more general and can enable better generalization ability for models. Moreover, we propose an end-to-end depth completion network, which takes the RGB image and the inaccurate depth map as inputs and outputs a refined depth map. Experiments demonstrate superior efficacy, efficiency and robustness of our method over previous works, and it is able to process images of high resolutions under limited hardware resources. Real robot experiments show that our method can also be applied to novel transparent object grasping robustly. The full dataset and our method are publicly available at www.graspnet.net/transcg

I. INTRODUCTION

Transparent-object depth is difficult to recover with ordinary RGB-D sensors, limiting depth-dependent grasping. The paper introduces TransCG, a large real-world dataset, and DFNet, a depth-completion network evaluated for generalization, efficiency, and grasping.

I. INTRODUCTION

  • Ordinary depth sensors produce noisy or incomplete maps for transparent objects, making many depth-based methods unsuitable for these scenes.Reflection and refraction distort light; some prior geometry-recovery approaches instead require specialized hardware, unlike the common robot-arm and RGB-D-camera setting.Source: §I · p. 1
  • Synthetic transparent-object datasets lack real sensor depth maps, which degrades their methods in real-world applications.The cited work also describes prior real-world depth datasets as small or limited in diversity and scene settings.Source: §I · pp. 1–2
  • TransCG is a real-world dataset for transparent-object depth completion containing 57,715 RGB-D images of 51 transparent objects and around 200 opaque objects across 130 scenes.The dataset also provides 3D mesh models of the transparent objects.Source: §I · p. 2
  • DFNet takes RGB images and inaccurate depth maps to produce refined depth maps and assist 6-DoF grasping methods.The authors report better cross-scene generalization, improvement on most metrics, highest inference speed, and least computation overhead among compared methods.Source: §I · p. 2
  • Real-robot experiments report promising performance when applying the method to grasp novel transparent objects.The dataset, source code, and pretrained models are publicly released.Source: §I · p. 2

II. RELATED WORKS

Prior transparent-object datasets trade off real-world scale, sensor-grounded depth, and diversity, while depth-completion methods vary in input setting and generalization. TransCG is proposed as a real-world dataset built with an efficient collection pipeline.

A. Transparent Datasets

  • Existing transparent-object datasets include large collections for tasks not requiring depth, synthetic depth datasets, and smaller or restricted real-world datasets.The cited real-world datasets have limited diversity or environments, and ClearGrasp real-world data contains 286 samples because generating missing information is time- and labor-intensive.Source: A. Transparent Datasets · p. 2
  • TransCG uses a novel pipeline to construct a large-scale real-world dataset for transparent-object depth completion.The paper frames this pipeline as a response to the difficulties of building transparent-object datasets.Source: A. Transparent Datasets · p. 2

B. Depth Completion and Estimation

  • Depth estimation methods span RGB-only, RGB with sparse depth, and RGB with inaccurate depth; transparent-object completion falls into the last category.Completion in this setting can use multiple views or a single view.Source: B. Depth Completion and Estimation · p. 2
  • Single-view approaches include two-stage pipelines, ClearGrasp adaptations, self-attentive adversarial replacement, and local implicit depth refinement.The cited account notes unacceptable real-time inference speed for ClearGrasp and limited cross-domain generalization for LIDF-Refine.Source: B. Depth Completion and Estimation · p. 2
  • The paper argues that better real-world generalization and applicability require a tiny, robust model supported by substantial real-world data.Multi-view alternatives cited include NeRF-based depth recovery and physical-based reconstruction using multiple images.Source: B. Depth Completion and Estimation · p. 2

C. 6-DoF Grasping

6-DoF grasping predicts gripper position and rotation in 3D, but grasping methods rely heavily on depth images that are unreliable for transparent objects. The paper motivates real-world dataset construction and describes TransCG’s collection pipeline and scene coverage.

C. 6-DoF Grasping

  • Depth-dependent 6-DoF grasping methods are unsuitable for transparent objects, motivating color-based depth recovery to support grasping.6-DoF methods predict gripper position and rotation in 3D and allow grasping from various angles.Source: C. 6-DoF Grasping · pp. 2–3

A. Overview

  • The collection pipeline uses limited object-level manual annotation and robot data collection to address the scarcity of large real-world datasets.Prior accurate-depth datasets were often synthetic, while real-world collection was constrained by annotation workload.Source: A. Overview · p. 3
  • Optical tracking and scanned 3D models provide object poses during collection, which the pipeline uses to restore geometry during annotation.The objects are wrapped in opaque materials for scanning; IR markers support real-time pose tracking.Source: A. Overview · p. 3
  • TransCG contains 57,715 RGB-D images from 130 scenes, with refined ground-truth depth, transparent masks, and surface normals.The scenes comprise 65 simple isolated scenes and 65 challenging cluttered scenes; the dataset was collected within a week using two cameras.Source: A. Overview · p. 3
  • The dataset includes 51 objects that can produce inaccurate depth, including transparent, translucent, reflective, and dense tiny-hole objects.The scenes use varied background settings and are presented as closer to real-world grasping environments in the cluttered subset.Source: A. Overview · p. 3

B. System Setup

The system tracks transparent objects using IR markers mounted on an attached fixer, then uses scanned models and manual tracker-to-object annotation to support pose tracking. A GUI enables real-time evaluation, and processing an object takes about an hour of human effort.

B. System Setup

  • The tracking system is the only part of dataset building that requires human effort, and processing one object takes around 1 hour on average.The system enables recovery of ground-truth depth for transparent objects afterward.Source: B. System Setup · pp. 3–4
  • An IR-marker fixer is tracked optically, while the object is temporarily wrapped in opaque material for 3D scanning and model acquisition.The flat fixer makes tracking more robust than attaching markers directly to the object.Source: B. System Setup · pp. 3–4
  • Manual tracker-to-object annotation provides the marker-to-object transformation, with a GUI for real-time evaluation in a PST tracker and two-camera system.The cameras are an Intel RealSense D435 and L515; the tracker records the marker pattern for subsequent tracking.Source: Figure 3 · p. 4, B. System Setup · p. 4

C. Data Collection

A robot-mounted tracking system collects images along a fixed 240-viewpoint trajectory, with calibration and pose recovery supporting collection across viewpoints. The collected poses are then used to render ground-truth depth, transparent masks, and surface normals.

C. Data Collection

  • The robot-mounted system captures RGB-D images and tracking results along a fixed trajectory of 240 viewpoints, with randomly placed transparent objects and varied table covers and opaque objects.Two cameras provide images at different quality levels.Source: C. Data Collection · p. 4
  • Camera extrinsic calibration lets object poses be recovered at viewpoints where tracking fails, using a successfully tracked viewpoint and camera poses across views.The first viewpoint is always successfully detected and can serve as the reference.Source: C. Data Collection · p. 4
  • Collected 6D poses are used to render ground-truth depth maps, transparent masks, and surface normals.The dataset also captures RGB-D images using two cameras.Source: C. Data Collection · p. 4

D. Dataset Postprocessing

Dataset postprocessing filters unsuitable images and validates the remaining scene annotations before recording valid viewpoints.

D. Dataset Postprocessing

  • Blurry and improperly exposed samples are automatically removed, then rendered object meshes are manually checked for overlap with the original objects and valid viewpoints are recorded per scene.OpenCV Laplacian operators and histograms support automatic sample detection.Source: 1) Data Verification · p. 4

2) Dataset Split:

Cross-dataset experiments indicate that methods trained on TransCG generalize well, while the dataset split holds out scenes containing selected objects. The section also describes the grasping pipeline and DFNet’s architecture and loss.

2) Dataset Split:

  • All scenes containing 12 randomly selected objects form the test set, leaving 34,191 training samples and 23,524 testing samples.The remaining scenes are used for training.Source: 2) Dataset Split · p. 4

E. Discussion

  • Methods trained on TransCG generalize well to other testing datasets, and the fixer-mounted markers have minimal impact on network training.Markers are placed on an attached fixer rather than directly on transparent-object surfaces, preserving the transparent part to the greatest extent.Source: E. Discussion · pp. 4–5

A. Overview

  • The grasping baseline applies completed depth to a grasp-pose detection network that takes point clouds and outputs grasp poses.The overview states that the completed depth can support depth-based grasping of transparent objects.Source: A. Overview · p. 5

B. Depth Completion

  • DFNet is a four-layer U-Net with dense-based CDCD, CDC, and CDCU blocks; original depth is supplied to every block, skip paths retain high-resolution information, and DUC replaces deconvolution in upsampling.The network takes RGB and inaccurate partial depth as input and predicts a full depth map.Source: B. Depth Completion · p. 5, Figure 4 · p. 5, A. Overview · p. 5
  • DFNet combines a depth-inaccuracy loss with a weighted surface-normal cosine-distance loss that penalizes unsmoothness, excluding depth pixels outside [0.3, 1.5].The surface normals are computed from predicted and ground-truth depth maps.Source: B. Depth Completion · p. 5

C. Object Grasping

The grasping pipeline refines camera depth with DFNet before point-cloud construction and grasp-pose detection. The section also includes TransCG experiment results and visual comparisons of refined depth maps and error maps.

C. Object Grasping

  • RGB and camera depth are resized for DFNet refinement, restored to original size, converted to a scene point cloud, and passed to GraspNet-baseline for grasp candidates and robot execution.Other depth-based grasping methods are also applicable.Source: C. Object Grasping · p. 5
  • Table II presents experiment results on TransCG; its GPU-memory and inference-time measurements use an NVIDIA GeForce RTX 3090.Source: Table II · p. 6, C. Object Grasping · p. 6
  • Fig. 5 compares refined depth maps and error maps from different methods across TransCG testing scenes.Source: Figure 5 · p. 6

V. EXPERIMENTS

The experiments compare DFNet with representative transparent-object depth-completion methods on TransCG. The reported results cover accuracy, qualitative depth completion, and computational efficiency.

A. Depth Completion Experiments

  • ClearGrasp, LIDF-Refine [41], and TranspareNet are evaluated using their released code and optimal hyperparameters, with all baselines trained on TransCG for fair comparison.Source: A. Depth Completion Experiments · p. 6
  • DFNet uses 64 hidden channels and trains for 40 epochs with batch size 32, AdamW, a multi-step learning-rate schedule, and data augmentation.The initial learning rate is 10^-3; augmentation includes flipping, rotation, added noise, and HLS color transformations.Source: A. Depth Completion Experiments · p. 6
  • All methods use 320 × 240 images for training and testing; training uses four NVIDIA GeForce RTX 3090 GPUs and testing uses one.Average training time per epoch is approximately 20 minutes.Source: A. Depth Completion Experiments · p. 6
  • Depth-completion metrics are calculated on transparent areas according to transparent masks unless otherwise specified; reported measures include RMSE, REL, MAE, and threshold δ.Threshold δ uses 1.05, 1.10, and 1.25.Source: A. Depth Completion Experiments · p. 6
  • DFNet outperforms ClearGrasp and LIDF-Refine [41] on both metrics and quality, and performs better than TranspareNet on some metrics; it also has the smallest size, fastest inference time and lowest GPU memory occupation.Source: A. Depth Completion Experiments · p. 6

B. Cross-Domain Experiments

Cross-domain experiments assess both model robustness and dataset generality. The reported comparisons favor the proposed method across domains and TransCG over prior synthetic datasets in real-world testing.

B. Cross-Domain Experiments

  • The experiments test methods across training and testing domains, and use the proposed method to compare dataset generality on TOD [20] and ClearGrasp real-world data.Source: B. Cross-Domain Experiments · p. 6
  • In cross-domain method tests, the proposed method is reported as the most robust; it is less sensitive to domain changes than LIDF-Refine [41], whose performance drops substantially across domains.The experiments include synthetic-to-TransCG and TransCG-to-ClearGrasp-real testing; see Table III.Source: B. Cross-Domain Experiments · p. 7, Table III · p. 7
  • TransCG is reported as more universal than previous synthetic datasets when tested on TOD [20] and ClearGrasp real-world data, including despite the latter's similarity to ClearGrasp synthetic training data.See Table IV for the dataset cross-domain results.Source: B. Cross-Domain Experiments · p. 7, Table IV · p. 7

C. Real Robot Experiments

Real-robot experiments evaluate grasping of transparent objects in a cluttered scene. The conclusion states that the method applies to novel objects and is compatible with depth-based manipulation methods.

C. Real Robot Experiments

VI. CONCLUSION

  • The conclusion reports that TransCG is more general than previous synthetic datasets, DFNet is more efficient and robust, and real-robot tests demonstrate applicability to novel objects.It also states that compatibility with depth-based manipulation methods allows the method to serve as preprocessing for downstream transparent-object tasks.Source: §VI · p. 7
Loading 2202.08471v2…