Source-linked AI summary
ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation
Shreeyak S. Sajjan, Matthew Moore, Mike Pan, Ganesh Nagaraja, Johnny Lee, Andy Zeng, Shuran Song
TL;DR
Transparent objects often yield inaccurate or missing depth because their refractive and specular appearance violates standard sensing assumptions. ClearGrasp predicts normals, masks, and boundaries from RGB-D images and globally refines the sensor depth using synthetic training data. It generalizes to real and novel objects and improves robotic picking, achieving 86% suction and 72% parallel-jaw success rates.
Problem
Transparent materials produce inaccurate or missing depth estimates because their refractive and specular appearance violates classic stereo assumptions.
Method
ClearGrasp uses deep networks to infer surface normals, transparent-surface masks, and boundaries, then combines them with initial depth in global optimization.
Results
ClearGrasp generalizes to real-world and unseen transparent objects and achieves 86% suction and 72% parallel-jaw picking success on a real robot.
Takeaways & Limitations
The refined geometry can be used out-of-the-box with robotic manipulation algorithms for transparent-object picking.
Takeaways & Limitations
Depth remains indeterminate for regions completely enclosed by occlusion boundaries, allowing such regions to receive random values.
Abstract
from arXiv · showhide
Transparent objects are a common part of everyday life, yet they possess unique visual properties that make them incredibly difficult for standard 3D sensors to produce accurate depth estimates for. In many cases, they often appear as noisy or distorted approximations of the surfaces that lie behind them. To address these challenges, we present ClearGrasp -- a deep learning approach for estimating accurate 3D geometry of transparent objects from a single RGB-D image for robotic manipulation. Given a single RGB-D image of transparent objects, ClearGrasp uses deep convolutional networks to infer surface normals, masks of transparent surfaces, and occlusion boundaries. It then uses these outputs to refine the initial depth estimates for all transparent surfaces in the scene. To train and test ClearGrasp, we construct a large-scale synthetic dataset of over 50,000 RGB-D images, as well as a real-world test benchmark with 286 RGB-D images of transparent objects and their ground truth geometries. The experiments demonstrate that ClearGrasp is substantially better than monocular depth estimation baselines and is capable of generalizing to real-world images and novel objects. We also demonstrate that ClearGrasp can be applied out-of-the-box to improve grasping algorithms' performance on transparent objects. Code, data, and benchmarks will be released. Supplementary materials available on the project website: https://sites.google.com/view/cleargrasp
I. INTRODUCTION
ClearGrasp addresses unreliable depth sensing for transparent objects by combining deep learning, synthetic data, and RGB-D inputs to estimate geometry for manipulation. It predicts geometric cues and refines sensor depth, generalizing to real and unseen objects.
- Transparent objects’ refractive and specular materials violate classic stereo assumptions, producing noisy or distorted depth estimates.
- ClearGrasp predicts surface normals, transparent-surface masks, and occlusion or contact edges, then uses them to refine initial RGB-D depth.
- Synthetic rendered images with domain randomization can produce reasonable real-world results, while mixing real out-of-domain data improves generalization to real and unseen objects.
- The benchmark includes over 50,000 synthetic RGB-D images and 286 real-world RGB-D images with ground-truth geometries.
- ClearGrasp outperforms monocular depth baselines and achieves 86% suction and 72% parallel-jaw picking success on a real robot.
II. RELATED WORK
Prior work estimates geometry, recognizes transparent objects, or learns from synthetic data, but transparent-object reconstruction remains difficult because standard sensors provide inaccurate or missing depth. Existing approaches often rely on constrained capture settings or simplified synthetic imagery.
- Estimating geometry from color images: Color-image geometry methods infer or complete depth, but they generally do not explicitly handle transparent objects with inaccurate or missing sensor depth.
- Recognizing transparent objects: Earlier transparent-object recognition methods relied on specular reflections or refractive edge characteristics, while later methods used deep learning detectors.
- Estimating geometry of transparent objects: Transparent-object geometry methods often assume a specific capture procedure, background pattern, sensor type, or known object model.
- Learning from synthetic data: Synthetic datasets for transparent objects require high-quality rendering and models to reproduce artifacts such as specular highlights and caustics.
III. METHOD
ClearGrasp takes a single RGB-D image, predicts transparent-object geometry cues with deep networks, and combines those predictions with the sensor depth in a global optimization to produce a refined depth map.
- ClearGrasp uses the color image to infer surface normals, transparent-surface masks, and occlusion boundaries before global depth refinement.
A. Estimating 3D Geometry of Transparent Object
The geometry-estimation module adapts depth completion for transparent objects by removing unreliable measurements and predicting normals plus both occlusion and contact boundaries.
- Transparent-surface masks remove unreliable depth measurements, including missing values and background-depth readings caused by refraction.
- Surface-normal estimation predicts pixel-wise normals from RGB and L2-normalizes them as unit vectors.
- Boundary detection distinguishes non-edge pixels, occlusion boundaries, and contact edges to improve depth discontinuity estimates.
B. Synthetic Training data generation
ClearGrasp’s synthetic data uses physically based rendering to model transparent-object effects and provides comprehensive geometry ground truth across known and held-out objects.
- Figure 4 pairs rendered images with surface-normal, boundary, depth, and mask ground truth across different objects.
- The synthetic pipeline uses Blender’s physics engine and Cycles ray tracing to simulate refraction, reflection through multiple surfaces, and soft shadows.
- The dataset contains 9 CAD models of transparent plastic objects, with 4 held out during training to test generalization.
- Each rendered scene varies lighting, ground-plane textures, camera settings, and the number of CAD objects.
- Ground truth includes RGB, aligned metric depth, transparent-object segmentation, camera and object poses, and scene surface normals.
C. Real-World Benchmark
The real-world benchmark captures transparent objects alongside spray-painted counterparts to obtain accurate depth references, covering known and novel objects under varied backgrounds.
- The benchmark uses photography backgrounds made from five wooden laminates and five cloths, with ambient lighting and camera distances of 40–100 cm.
- Transparent objects are replaced by identical spray-painted instances, and overlaid camera frames provide depth ground truth for the original objects.
- The validation set contains 173 images of 5 known objects, while testing contains 113 images of 5 novel objects, including 3 unseen glass objects.
- Each image contains 1–6 objects, averaging 2 objects per image.
D. Grasp planning
The evaluation connects ClearGrasp’s depth refinement to grasp planning and tests generalization, ablations, training-data choices, and downstream robot performance.
- Grasp planning: The grasping system predicts pixel-wise success for scripted grasp primitives over end-effector locations and orientations using ClearGrasp-completed depth images.
- Grasp planning: Parallel-jaw grasping evaluates 16 orientations by rotating heightmaps in 22.5° increments and selecting the pixel-orientation pair with the highest predicted probability.
- Generalization: The model generalizes to real images with similar depth RMSE and Rel scores on known objects, although surface-normal accuracy decreases on real images.
- Generalization: Novel-object depth estimation performs better than known-object estimation on synthetic and real-world data, likely because the novel objects are smaller.
- Ablations: Removing noisy transparent-object depth values is important, with true-positive mask rates above 95% critical for removing incorrect initial depths.
- Ablations: Weighted losses and adding contact edges both significantly improve depth-completion results.
- Robot manipulation: ClearGrasp improves grasping success from 64% to 86% with suction and from 12% to 72% with parallel-jaw grasping.
- Limitations: The system remains imperfect under varying lighting, cluttered scenes, sharp caustics, and shadows.
A. Additional Details on Dataset
The datasets distinguish known from novel objects and use different cameras across portions of the real-world benchmark, supporting cross-object and cross-camera evaluation.
- Camera setup: Known-object images use a RealSense D435 camera, while 80% of novel-object images use a RealSense D415 camera.
- Synthetic dataset: The synthetic dataset uses 5 known training objects and 4 novel test objects, including thick-glass and complex-shaped examples.
- Real-world benchmark: The real-world benchmark contains 5 known and 5 novel objects, including novel glass objects not modeled in the synthetic dataset.
B. Limitations and Failure Cases
ClearGrasp has failure modes tied to depth indeterminacy, cluttered or varying backgrounds, and bright directional lighting. These conditions can corrupt surface-normal or occlusion-boundary predictions and produce erroneous depth.
- The paper presents four examples illustrating these model failure cases.
- Regions completely enclosed by occlusion boundaries can have indeterminate depth and may be assigned random values.This occurs when the region cannot be reconstructed from surface normals relative to the rest of the scene.
- Cluttered scenes with mutually occluding transparent objects make surface-normal and occlusion-boundary prediction difficult, causing output-depth errors.
- Varying backgrounds behind transparent objects, such as table edges or partially occluded opaque objects, are challenging cases.
- Bright directional lighting and caustics can make the model mistake transparent-object shadows for transparent objects.The synthetic dataset lacks accurate caustics, increasing susceptibility to this error.
C. Additional Training Details
ClearGrasp uses shared pretrained convolutional architectures and varied training data and augmentations to improve robustness beyond the limited synthetic scene composition. Bright-patch augmentation specifically targets lighting, caustics, and background variation.
- All three neural networks use Deeplabv3+ with a DRN-D-54 backbone and COCO-pretrained semantic-segmentation weights.The networks estimate surface normals, occlusion boundaries, and transparent-surface segmentation.
- Training augments the synthetic data with Matterport3D and ScanNet images to address the synthetic dataset’s limited scene composition.The synthetic scenes contain a flat plane and up to five transparent objects, without walls or random opaque objects.
- The training pipeline uses image augmentations including flips, rotations, color-space changes, blur, noise, and coarse dropout.
- Random translucent white patches are overlaid on images to approximate bright caustics or directional-light patches absent from the synthetic data.The patches use Simplex Noise blended with a white image.
- The white-patch augmentation produced significant improvement with patterned backgrounds, bright caustics, directional lights, and varying backgrounds.
D. Experiment on Network Architectures
The architecture experiment evaluates surface-normal networks across backbones and input image sizes, motivated by better performance on bottles farther from the camera. Qualitative comparisons also examine ablations and monocular-depth baselines.
- Network architecture experiment: Surface-normal estimation was better for bottles farther from the camera, motivating experiments with larger receptive fields.
- Network architecture experiment: Table IV compares Deeplabv3+ models using ResNet-101 and DRN-54 backbones across different input image sizes.
- Qualitative evaluation: The ablation study evaluates ClearGrasp without transparent-surface masks, without contact edges, and without weighted contact-edge loss terms.
- Qualitative evaluation: Qualitative baseline comparisons contrast ClearGrasp with DeepCompletion and DenseDepth.
- Failure analysis: Failure-case depth errors are mostly attributed to erroneous occlusion-boundary predictions or surfaces lacking contact edges because of occlusion.