ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation

• Rendered synthetic images with domain randomization enable real-world results, while mixing in real-world images without transparent objects improves generalization to real images and unseen transparent objects.Source: §I · pp. 1–2
• ClearGrasp predicts transparent-surface masks, occlusion and contact edges, and surface normals, then combines them in global optimization to adjust and complete depth.Source: §I · p. 1, Figure 2 · p. 2
• ClearGrasp outperformed monocular depth-estimation baselines and achieved 86% suction and 72% parallel-jaw picking success on a real-world robot.Source: §I · p. 2
• The work contributes a synthetic dataset of over 50,000 RGB-D images and a real-world benchmark of 286 images with ground-truth geometries.Source: §I · p. 2

Estimating geometry from color images

• Existing image-based depth inference and RGB-D completion methods do not explicitly handle transparent objects, whose commodity-camera depth is often inaccurate or missing.Surface normals alone do not provide the depth images or point clouds required by manipulation algorithms.Source: §II · p. 2

Recognizing transparent objects

• Classic transparent-object detectors use reflections or refractive edges, while later approaches localize objects or recover piece-wise planar depth.ClearGrasp instead recovers detailed non-planar geometry needed for manipulation.Source: §II · p. 2

Estimating geometry of transparent objects

• Earlier transparent-object geometry methods often assume controlled capture, known backgrounds, sensors, or 3D models; ClearGrasp generalizes to unseen objects without prior models or camera-position knowledge.Lysenkov et al. fit training-stage object models to edges extracted using a segmentation mask.Source: §II · p. 2

Learning from synthetic data

• Synthetic datasets rarely support transparent objects; existing examples target refractive flow [8], segmentation, or relative depth in simplified settings, unlike ClearGrasp’s realistic-environment absolute-depth reconstruction.Representative transparent imagery and artifacts such as specular highlights and caustics require high-quality rendering and 3D models.Source: §II · pp. 2–3

• ClearGrasp predicts surface normals, transparent-surface masks, and occlusion boundaries from color, then uses these cues with initial depth in a global optimization to refine depth across transparent surfaces.The input is a single RGB-D image.Source: §III · p. 3
• The work constructs a synthetic dataset for training and testing and a real-world benchmark for transparent objects.Source: §III · p. 3
• ClearGrasp is applied to a real-world robotic pick-and-place system.Source: §III · p. 3

A. Estimating 3D Geometry of Transparent Object

• ClearGrasp removes transparent-surface depth readings and predicts normals plus occlusion and contact edges before applying depth completion adapted from Zhang and Funkhouser [59].The additional transparent-surface mask removes unreliable measurements; predicting both edge types helps distinguish boundaries for reconstruction.Source: A. Estimating 3D Geometry of… · p. 3
• Commodity RGB-D sensors produce missing depth and misleading background-surface depths on transparent objects, so a Deeplabv3+ model with a DRN-D-54 backbone masks those pixels [9, 53].The mask removes both absent readings and nonzero readings caused by light refracting through the object and reflecting from surfaces behind it.Source: A. Estimating 3D Geometry of… · p. 3
• A Deeplabv3+ network with a DRN-D-54 backbone predicts pixel-wise RGB surface normals, and L2 normalization makes the outputs unit vectors.The final convolutional layer has three output classes.Source: A. Estimating 3D Geometry of… · p. 3
• Boundary detection classifies pixels as non-edge, occlusion boundary, or contact edge; contact edges improve occlusion-boundary predictions and boundary pixels receive 5x background loss weight.Contact edges are not directly used by optimization, but help prevent erroneous boundaries around entire objects that would obstruct normal-guided depth recovery.Source: A. Estimating 3D Geometry of… · p. 3
• Global optimization fills masked depth using predicted normals and observed depth, while occlusion boundaries guide depth discontinuities through a weighted squared-error objective.The objective combines observed-depth error, neighboring-pixel depth differences, and normal consistency; the experiment uses λ_D = 1000, λ_S = 0.001, and λ_N = 1.0.Source: A. Estimating 3D Geometry of… · p. 3

B. Synthetic Training data generation

• Synthesis AI generated the data using Blender’s physics engine [6] and Blender Cycles, which simulates refraction, reflection through multiple surfaces, and soft shadows.The rendering engine was selected for its configurability and its ability to simulate effects important for transparent objects.Source: B. Synthetic Training data generation · pp. 3–4
• Scenes combined 1–5 dropped CAD objects, optionally a gray tote box, 33 HDRI lighting environments, and 65 ground-plane textures, using Intel RealSense D415 camera intrinsics.Objects came to rest under physics simulation before random lighting environments and ground textures were applied.Source: B. Synthetic Training data generation · p. 4
• The dataset used 9 CAD models, with 4 held out for testing; it contained over 13,000 images of 3 objects each and 5,000 images each of 2 other objects.One hundred images of each model were reserved for validation, and around 100 images were rendered for each of the 4 test objects.Source: B. Synthetic Training data generation · p. 4
• Each scene included RGB, aligned depth in meters, transparent-object segmentation, camera and object poses, and scene surface normals as ground truth.The rendered examples and their corresponding ground truth geometry are shown in Fig. 4.Source: B. Synthetic Training data generation · p. 4, Figure 4 · p. 4

C. Real-World Benchmark

• The benchmark used ambient-lit scenes with five wooden laminates and five background cloths, while a tripod-mounted camera was positioned 40–100 cm from the objects.The backgrounds were spread across a flat surface against a wall; ambient lighting was used to avoid sharp caustics.Source: C. Real-World Benchmark · p. 4
• Ground-truth transparent-object depth was captured by replacing each object with an identical rough-stone-textured spray-painted instance and aligning the images, enabling sub-millimeter positioning accuracy.The setup overlaid a frozen frame of the transparent objects and surrounding opaque objects with subsequent frames of their spray-painted replacements.Source: C. Real-World Benchmark · p. 4
• Validation comprised 173 images of 5 training-known objects, while testing comprised 113 images of 5 novel objects, including 3 glass objects absent from the synthetic dataset.Images contained 1–6 objects, averaging 2 per image.Source: C. Real-World Benchmark · p. 4

D. Grasp planning

• The grasp planner predicts pixel-wise success from ClearGrasp surface normals and completed depth, then selects the highest-probability pixel for top-down suction or parallel-jaw grasping.For parallel-jaw grasps, it evaluates 16 rotated heightmaps, selecting the pixel and rotation with the highest predicted probability; training uses binary cross-entropy.Source: D. Grasp planning · p. 4
• On known objects, real-world depth RMSE and Rel scores are similar to synthetic-domain results, while surface-normal accuracy decreases on real images.Large normal errors occur when transparent objects occlude novel opaque objects.Source: D. Grasp planning · p. 5, Table I · p. 6
• For novel shapes, depth estimation generalizes well on synthetic and real-world data, with better results than on known objects, likely because the novel objects are smaller.Real-novel metrics also exceed Syn-novel; the passage attributes this to three new, thicker glass objects with more evident refraction than the thin plastic objects.Source: D. Grasp planning · p. 5, Table I · p. 6
• ClearGrasp outperforms DenseDepth [2] by a large factor when both methods are trained on the same data.DenseDepth directly predicts depth from color.Source: D. Grasp planning · p. 5, Table II · p. 6
• Removing transparent-object pixels matters: retaining noisy input depths significantly increases final error, while mask true-positive rate above 95% is critical; contact edges and weighted loss also improve depth completion.The contact-edge comparison is illustrated in Fig. 7.Source: D. Grasp planning · pp. 5–6, Table II · p. 6, Figure 7 · p. 6
• Out-of-domain real-world pretraining alone does not detect transparent objects, but combining it with in-domain synthetic fine-tuning improves results, especially on real-world tests.The training-data analysis compares procedures with and without pretraining on 80k ScanNet and Matterport images and with and without synthetic-data fine-tuning.Source: D. Grasp planning · p. 6, Table III · p. 6
• ClearGrasp improves real-world grasp success from 64% to 86% for suction and from 12% to 72% for parallel-jaw grasping.For each end-effector, the system was trained with 500 trial-and-error attempts and evaluated over 50 attempts.Source: D. Grasp planning · p. 6

APPENDIX

• The appendix contains additional system details, analysis, and experimental results.Source: APPENDIX · p. 10

• All known-object real-world images use a RealSense D435, while 80% of novel-object images use a RealSense D415.Source: A. Additional Details on Dataset · p. 10
• The synthetic dataset uses 5 known training objects and 4 novel test objects; the novel objects include 2 thick-glass objects and 2 with complex shapes.Source: Figure 8 · p. 10
• The real-world benchmark has 5 known and 5 novel objects; the 5 known and 2 novel objects were modeled synthetically, while 2 other novel objects are glass rather than plastic.Source: Figure 9 · p. 10

B. Limitations and Failure Cases

• When an occlusion boundary completely encloses a region, surface normals cannot determine its depth, which may therefore be assigned a random value [59].Examples include a partially occluded bottle with invisible contact edges and the enclosed inner portions of glasses.Source: B. Limitations and Failure Cases · p. 10
• Overlapping transparent objects make surface normals and occlusion boundaries harder to predict, while a nonconstant background behind an object also challenges the models.These errors can propagate to the output depth; examples include a bottle at a table edge or partially occluding an opaque object.Source: B. Limitations and Failure Cases · p. 10
• Bright directional lighting and caustics can cause the model to mistake transparent-object shadows for transparent objects.The synthetic dataset lacks accurate caustics because of Cycles rendering limitations, making the model particularly susceptible to this failure.Source: B. Limitations and Failure Cases · p. 10

C. Additional Training Details

• All three networks used Deeplabv3+ with a DRN-D-54 backbone, COCO-pretrained weights, and shared SGD hyperparameters.Training used a constant learning rate of 1e-6, momentum 0.9, weight decay 5e-4, and up to 128 images per batch at 256x256 resolution on 8 Nvidia V100 GPUs.Source: C. Additional Training Details · p. 10
• Surface-normal training mixed 2k Matterport3D and 2k ScanNet images into each transparent-object-training epoch, following pretraining on 80k images from those datasets.The added images helped retain knowledge of surfaces such as walls; training progressed through increasingly large data subsets and resolutions, from 128x128 to 256x256.Source: C. Additional Training Details · p. 10
• Augmentations included flips and 90-degree rotations except for surface normals, color-space changes, blur, and element-wise, Gaussian, Laplacian, and dropout noise.The listed color augmentations included RGB addition and multiplication, hue and saturation changes, contrast normalization, and grayscale conversion.Source: C. Additional Training Details · p. 10
• Augmentation significantly improved performance with patterned cloth, bright caustics or directional lights, and varying backgrounds behind transparent objects.White patches generated by blending Simplex noise with a white image were overlaid transparently to mimic caustics or directional lights; channel-wise coarse dropout targeted varying backgrounds.Source: C. Additional Training Details · p. 11

D. Experiment on Network Architectures

• Smaller input images and a DRN-54 backbone improve surface-normal results over larger inputs and Resnet-101, supporting the larger-receptive-field hypothesis.The authors report that smaller inputs effectively increase receptive-field width and that replacing Resnet with DRN increases it further.Source: Table IV · p. 11, D. Experiment on Network Architectures · p. 11

E. Additional Qualitative Results

• Fig. 10 attributes most output-depth errors to erroneous occlusion-boundary predictions or surfaces lacking contact edges because of occlusion.The figure highlights depth errors in red and occlusion-boundary errors in black.Source: Figure 10 · p. 12
• Fig. 11 presents qualitative ablations removing masks, contact edges, or weighted loss terms for contact edges.These are the three ablation conditions shown in the figure.Source: E. Additional Qualitative Results · p. 11, Figure 11 · p. 12
• Fig. 12 qualitatively compares ClearGrasp with DeepCompletion and DenseDepth.The paper directs readers to its website for additional qualitative results.Source: E. Additional Qualitative Results · p. 11, Figure 12 · p. 13