Source-linked AI summary
Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects
Jeffrey Ichnowski, Yahav Avigal, Justin Kerr, Ken Goldberg
TL;DR
Robots struggle to detect and infer the geometry of transparent objects because existing depth cameras have difficulty sensing them. Dex-NeRF uses NeRF density, strategically placed lights, and transparency-aware depth rendering to feed Dex-Net grasp planning, achieving at least 90% grasp success in physical experiments while retaining long training time as a drawback.
Problem
Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of transparent objects, limiting robotic grasping of them.
Method
Dex-NeRF uses NeRF to recover transparent-object geometry through learned density, added lights, and transparency-aware depth rendering for Dex-Net grasp planning.
Results
90% or better grasp success rates were achieved on real transparent objects with NeRF-based grasp planning, including synthetic- and real-image-trained models.
Takeaways & Limitations
Recovered NeRF geometry was sufficient to compute grasps for robot manipulation tasks involving transparent objects.
Takeaways & Limitations
NeRF's long training time remains a main drawback of the approach.
Abstract
from arXiv · showhide
The ability to grasp and manipulate transparent objects is a major challenge for robots. Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of such objects. We propose using neural radiance fields (NeRF) to detect, localize, and infer the geometry of transparent objects with sufficient accuracy to find and grasp them securely. We leverage NeRF's view-independent learned density, place lights to increase specular reflections, and perform a transparency-aware depth-rendering that we feed into the Dex-Net grasp planner. We show how additional lights create specular reflections that improve the quality of the depth map, and test a setup for a robot workcell equipped with an array of cameras to perform transparent object manipulation. We also create synthetic and real datasets of transparent objects in real-world settings, including singulated objects, cluttered tables, and the top rack of a dishwasher. In each setting we show that NeRF and Dex-Net are able to reliably compute robust grasps on transparent objects, achieving 90% and 100% grasp success rates in physical experiments on an ABB YuMi, on objects where baseline methods fail.
1 Introduction
Dex-NeRF uses NeRF within a robot grasp-planning pipeline to recover transparent-object geometry and compute grasps. Experiments on synthetic and real images report at least 90% grasp success on real objects.
- Transparent objects challenge depth cameras because reflection and refraction make their appearance vary with viewpoint and illumination.The paper motivates geometry sensing for robotic interaction with transparent objects across household, industrial, and hazardous-environment tasks.
- Dex-NeRF integrates NeRF with robot grasp planning to compute and execute grasps on transparent objects.NeRF can represent view-dependent effects associated with transparency and reconstruct scene geometry from calibrated multi-view images.
- Fixed workcell cameras or accurately encoded manipulator-mounted cameras make camera extrinsics readily determinable for NeRF.The paper identifies extrinsic-camera determination as an important practical condition for the approach.
- 90% or better grasp success rates were achieved on real transparent objects using NeRF-based grasp planning.The evaluation includes models trained from photorealistic synthetic images and real images.
- The paper contributes a transparency-aware NeRF depth-rendering method, its integration with Dex-Net, and synthetic and real transparent-object datasets.
2 Related Work
Prior work detects transparent objects with data-driven vision and represents scenes with NeRF, while robotic grasping generally depends on sufficiently accurate RGB-D surface observations. NeRF offers view-dependent modeling of reflections and transparency but remains slow to train and data-inefficient.
- Detecting Transparent Objects: Data-driven prior methods detect or segment transparent objects using CNNs, transformers, and other learned models.Other approaches also predict object pose from visual input.
- Neural Radiance Fields: NeRF maps 3D coordinates and view directions to volume density and view-dependent radiance, enabling representation of specularities, reflections, and transparent-object geometry.
- Neural Radiance Fields: NeRF is slow to train and has low data efficiency, motivating methods such as Plenoctrees, JaxNeRF, and depth-supervised acceleration.The cited approaches reduce training time or improve efficiency through sparse representations, implementation changes, or depth supervision.
- Robotic Grasping: Traditional and data-driven grasping methods rely on sufficiently accurate observations of target surfaces, including depth maps, point clouds, octrees, or TSDFs.
3 Problem Statement
The task assumes rigid transparent objects observed by a fixed camera array or a robot-manipulated camera. The goal is to compute a gripper frame that produces a stable grasp of a transparent object.
- The environment contains rigid transparent objects and provides multiple scene images from fixed known cameras or a manipulable camera.
- The system must compute a robot-gripper frame that results in a stable grasp of a transparent object.
4 Method
Dex-NeRF recovers transparent-object geometry with NeRF density and uses transparency-aware depth rendering to produce grasp-planning inputs. Additional lighting improves visibility of transparent surfaces, while Vanilla-NeRF rendering yields noisy or hole-filled depth maps.
- 4.2 Recovering Geometry of Transparent Objects: Compared with an Intel RealSense depth camera, the proposed pipeline recovers depth across most of a real-world scene despite a few holes.The RealSense fails to recover depth from a large portion of the scene, especially around transparent-object effects.
- 4.1 Preliminary: Training NeRF: NeRF learns view-dependent color and view-independent density, allowing transparent-object geometry to be recovered from multi-view images.The model maps spatial location and viewing direction to density and color; its density is used to identify occupied space.
- 4.3 Rendering Depth for Grasp Analysis: Vanilla-NeRF depth rendering weights sampled distances by ray-termination probabilities, but produces noisy depth maps on transparent objects.The resulting depth maps support poorer grasp predictions than the transparency-aware rendering shown in Fig. 3.
- 4.3 Rendering Depth for Grasp Analysis: Dex-NeRF uses the first ray sample whose density exceeds threshold m to render transparency-aware depth for grasp analysis.Low thresholds produce noise, whereas high thresholds create holes; experiments use m = 15.
- 4.4 Improving Visibility with Additional Lights: Additional lights create more viewing angles for specular reflections, helping NeRF fill holes in the learned geometry.The paper reports improved learned geometry when increasing from one light to a 5x5 light array, with optimal placement depending on object and camera distributions.
5 Experiments
Experiments evaluate Dex-NeRF across synthetic and real transparent-object scenes, including physical grasping, lighting, depth rendering, and camera-grid workcell configurations. The results show usable grasp planning from NeRF-derived depth, with performance sensitive to training time, illumination, thresholds, and camera count.
- Datasets: The experiments use synthetic and physical transparent-object datasets collected across multiple camera poses and graspable settings.The datasets include cluttered tables, a dishwasher scene, and singulated transparent objects, with over 70 camera poses per dataset.
- Synthetic Grasping Experiments: 80%–98% grasp success rates were reached after approximately 50k–60k iterations in synthetic singulated-object simulations.Success improved with training time before plateauing, suggesting a practical iteration limit for high grasp success.
- Physical Grasping Experiments: 90% and 100% grasp success rates were achieved on physical transparent-object experiments using Dex-NeRF with an ABB YuMi.Each object was tested with 10 computed grasps, and success required lifting the object.
- Comparison to RealSense Depth: RealSense depth sensing often produced unknown-depth regions for transparent objects, whereas the proposed pipeline produced high-quality depth maps.The comparison was qualitative and used real-world transparent-object scenes.
- One vs Many Lights: Additional lights improved transparent-surface depth recovery when total light wattage was held constant.With a 5x5 light array, glasses were nearly fully recovered compared with missing closer surfaces under a single light.
- Workcell Setup: A minimum of approximately 9 cameras produced a usable depth map for grasp planning, while a 7x7 grid performed best among tested configurations.Increasing camera count improved PSNR and SSIM, with no statistically significant improvement beyond 7x7 cameras.
6 Conclusion
Dex-NeRF recovers transparent-object geometry accurately enough for robot grasp planning and achieves high physical grasp success, while long NeRF training time remains a key drawback.
- 90% and 100% grasp success rates were achieved in physical experiments on an ABB YuMi robot using recovered transparent-object geometry.The geometry was sufficient to compute grasps for robot manipulation tasks.
- NeRF recovers transparent geometry by combining additional lights that create specular reflections with thresholding of visible transparent points.The method uses NeRF’s learned density to identify occupied transparent points.
- Synthetic and real transparent-object datasets were created for experiments in geometry recovery and potential future NeRF research.
- Long training time required to obtain a NeRF model is identified as a main drawback for future work.Proposed acceleration directions include depth supervision, manipulator-mounted cameras, and visio-spatial foresight.