PaperlayerView exact revision on arXiv

Source-linked AI summary

GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF

Qiyu Dai, Yan Zhu, Yiran Geng, Ciyu Ruan, Jiazhao Zhang, He Wang

arXiv:2210.06575v3cs.ROcs.CV

TL;DR

Transparent and specular objects challenge depth-based 6-DoF grasp detection because their geometry is sensed unreliably. GraspNeRF uses sparse multiview RGB and jointly learns a generalizable NeRF with grasp detection, outperforming all cited baselines in real-world sequential grasping while operating in real time.

  • Problem

    Transparent and specular objects remain difficult for vision-based 6-DoF grasp detection because depth sensors produce wrong or missing geometry.

  • Method

    GraspNeRF jointly learns a generalizable NeRF and volumetric grasp detector from sparse multiview RGB inputs, using TSDF geometry and synthetic domain-randomized training data.

  • Results

    GraspNeRF outperforms depth-restoration and separate-NeRF baselines by over 20% grasp success rate, reaching 82.2% in packed and 65.9% in pile real-world scenes.

  • Takeaways & Limitations

    The system enables material-agnostic, sparse-input, real-time 6-DoF grasping that directly generalizes to novel real-world scenes.

  • Takeaways & Limitations

    The volume grid's limited resolution may bottleneck geometry reconstruction and grasping performance.

Abstract

from arXiv · show

In this work, we tackle 6-DoF grasp detection for transparent and specular objects, which is an important yet challenging problem in vision-based robotic systems, due to the failure of depth cameras in sensing their geometry. We, for the first time, propose a multiview RGB-based 6-DoF grasp detection network, GraspNeRF, that leverages the generalizable neural radiance field (NeRF) to achieve material-agnostic object grasping in clutter. Compared to the existing NeRF-based 3-DoF grasp detection methods that rely on densely captured input images and time-consuming per-scene optimization, our system can perform zero-shot NeRF construction with sparse RGB inputs and reliably detect 6-DoF grasps, both in real-time. The proposed framework jointly learns generalizable NeRF and grasp detection in an end-to-end manner, optimizing the scene representation construction for the grasping. For training data, we generate a large-scale photorealistic domain-randomized synthetic dataset of grasping in cluttered tabletop scenes that enables direct transfer to the real world. Our extensive experiments in synthetic and real-world environments demonstrate that our method significantly outperforms all the baselines in all the experiments while remaining in real-time. Project page can be found at https://pku-epic.github.io/GraspNeRF

I. INTRODUCTION

GraspNeRF addresses transparent and specular object grasping with sparse multiview RGB, generalizable NeRF construction, and grasp-oriented scene representation. It reports real-time performance and higher real-world sequential grasp success than competing methods.

  • Depth sensors often produce wrong or missing geometry for transparent and specular objects, undermining depth-based grasp detection.
  • Generalizable NeRF aggregates multiview observations to construct novel-scene representations without per-scene training and with sparse inputs.
  • GraspNeRF predicts TSDF geometry from aggregated multiview features and uses it for volumetric 6-DoF grasp detection.
  • The model is trained on 2.4 million photorealistically rendered images from 100K domain-randomized cluttered scenes containing diverse object materials.
  • By over 20% grasp success rate, GraspNeRF outperforms depth-restoration and separate-NeRF baselines, reaching 82.2% in packed and 65.9% in pile real-world scenes.
  • The system uses six sparse RGB views, supports material-agnostic 6-DoF grasping, generalizes to novel scenes, and runs at 11 FPS.

II. RELATED WORK

Prior approaches address transparent-object grasping through depth restoration, multiview geometry, or NeRF-based representations, but retain limitations in occlusion handling, input density, optimization time, or grasp dimensionality. GraspNeRF instead uses a differentiable generalizable NeRF jointly trained with grasping.

  • A. Transparent and Specular Object Grasping: Depth-restoration methods correct incomplete or erroneous depths before grasping, while single-view approaches struggle with occlusion and limited 3D geometry.
  • A. Transparent and Specular Object Grasping: GlassLoc and GhostPose use multiview observations for transparent-object scene representation or pose estimation, whereas GraspNeRF represents geometry with a differentiable generalizable NeRF.
  • B. Implicit Representations for Grasping: DexNeRF avoids corrupted sensor depths but requires dense images, hours of per-scene NeRF training, and only 3-DoF top-down grasping.
  • B. Implicit Representations for Grasping: Implicit scene representations are increasingly used for robotic tasks, but prior NeRF-based methods can require per-scene optimization and dense inputs.

A. Problem Statement and Method Overview

The framework maps sparse multiview RGB observations into a volumetric TSDF representation and then predicts dense 6-DoF grasps. Its generalizable NeRF-inspired construction, rendering branch, and joint optimization align scene reconstruction with grasping.

  • A. Problem Statement and Method Overview: The task maps known-calibrated multiview images of cluttered tabletop scenes to 6-DoF grasps for grasp-to-remove operations.
  • A. Problem Statement and Method Overview: GraspNeRF combines scene representation construction Fscene with volumetric grasp detection Fgrasp in a two-module framework.
  • B. Scene Representation Construction: NeuRay-inspired multiview features are projected onto a 3D grid and aggregated into a volumetric feature representation for grasping.
  • B. Scene Representation Construction: Unlike ray-based NeuRay features, the proposed construction forms a whole-scene volumetric representation suitable for grasping.
  • B. Scene Representation Construction: A TSDF prediction network maps aggregated features to a truncated signed distance field encoding geometric information for grasp detection.
  • B. Scene Representation Construction: The TSDF is coupled to volume density for rendering, allowing grasp detection and NeRF rendering to share the representation.
  • B. Scene Representation Construction: The network is trained end-to-end and directly optimized for grasping, rather than using staged NeRF and grasping training.

C. Volumetric Grasping Detection

The volumetric detector converts the predicted TSDF grid into dense grasp candidates, predicting quality, orientation, and opening width at voxel-center positions. Candidate filtering uses geometry, quality, collision-related width constraints, and non-maximum suppression.

  • A 3D CNN maps the TSDF grid to a dense grasp candidate volume, using voxel centers as position candidates.
  • At each candidate position, separate decoder heads predict grasp quality, rotation, and gripper opening width.
  • Candidates far from the surface are discarded using TSDF, while Gaussian smoothing, thresholding, and non-maximum suppression refine quality predictions.
  • The detector retains grasps within the gripper's maximum opening width and randomly selects among remaining candidates for execution.

D. Domain Randomization-based Synthetic Data Generation

The paper introduces a synthetic multiview grasping dataset covering diffuse, transparent, and specular objects, and trains scene representation and grasp prediction jointly with complementary losses.

  • D. Domain Randomization-based Synthetic Data Generation: 100K scenes provide 2.4 million RGB images and 2 million 6-DoF grasp poses across diffuse, specular, and transparent materials.The dataset addresses the absence of grasp annotations for transparent and specular objects in existing multiview resources.
  • D. Domain Randomization-based Synthetic Data Generation: Domain randomization varies object materials, textures, backgrounds, illumination, and camera poses to support transfer from photorealistic simulation to real scenes.
  • E. Network End-to-End Training: End-to-end training jointly optimizes scene representation construction and grasping because geometry quality and grasp prediction support each other.
  • E. Network End-to-End Training: The objective combines grasp supervision for quality, orientation, and width with color reconstruction and TSDF geometry losses.The grasp loss excludes zero-quality grasps, while geometry learning uses L1 supervision with Eikonal regularization.

IV. EXPERIMENTS

The experiments evaluate GraspNeRF in simulation and on a real robot, including baseline comparisons and ablations of its main design choices.

  • IV. EXPERIMENTS: Experiments cover simulation and real-robot grasping, with simulation ablations analyzing the impact of the framework’s principal designs.

A. Implementation Details

Implementation uses a fixed volumetric workspace, six wrist-mounted RGB viewpoints, and separate simulated and real object sets spanning the target materials.

  • A. Implementation Details: A Franka Panda with a parallel-jaw gripper operates in a 30 × 30 × 30 cm3 workspace observed from six uniformly sampled RGB viewpoints.The wrist-mounted RealSense D415 depth is not used by the method.
  • A. Implementation Details: Simulation uses 473 meshes with 417 training and 56 testing objects, while real evaluation uses 20 household objects arranged in piles or upright packed scenes.Materials and textures are randomized across transparent, specular, and diffuse categories.

C. Baseline Methods

The experiments compare volumetric, depth-restoration, and NeRF-based baselines across retrieval and clutter-removal tasks, with GraspNeRF achieving the strongest reported performance.

  • Evaluation Metrics: Success Rate measures successful grasps divided by attempts, while Declutter Rate measures the average percentage of objects removed across rounds.
  • D. Simulation Grasping Experiments: 86.1% average success over 36 single-object retrieval trials establishes GraspNeRF as the best-performing method among the compared baselines.The result is reported in Table I for simulation.
  • D. Simulation Grasping Experiments: GraspNeRF significantly improves transparent and specular object grasping over competing methods in sequential clutter removal.Table II reports sequential clutter-removal results for transparent, specular, and mixed materials.
  • Baseline Analysis: VGN suffers from noisy or missing multiview depths, whereas SwinDR-VGN can oversmooth surfaces and erase small objects after depth restoration.
  • Baseline Analysis: NeRF-VGN requires 49 views and lengthy per-scene optimization, performs poorly for single-object retrieval, and struggles with textureless backgrounds.
  • Baseline Analysis: GraspNeRF’s randomized-background training enables geometry generalization to both textureless and textured tabletops during inference.Figure 3 is cited as demonstrating superior geometry reconstruction over competing methods.

E. Real Robot Experiments

Real-robot experiments show that GraspNeRF achieves strong grasping performance with sparse multiview RGB inputs and real-time inference, while retaining some failure cases on upright cylindrical objects.

  • GraspNeRF achieves the highest grasping success and declutter rates across real-world pile and packed sequential clutter-removal settings.The supplied table passage identifies the evaluation scope, while the results passage reports GraspNeRF as the top-performing method across materials and settings.
  • 88.9% success over 18 trials is reported for real-world single-object retrieval, outperforming competing baselines.
  • Six RGB views support 6-DoF grasp prediction, compared with 49 views required by NeRF-VGN.
  • 11 FPS inference without per-scene optimization makes GraspNeRF substantially faster than NeRF-VGN in the real-world evaluation.
  • Insufficient friction can cause the gripper to slip from upright cylindrical objects, such as hard glass bottles, in packed scenes.

F. Ablation Studies

Ablations show that end-to-end learning, supervised TSDF prediction, and the NeRF rendering branch each contribute to grasping performance, especially for challenging transparent and specular objects.

  • End-to-end training outperforms separate or sequential scene-and-grasp training by reducing the impact of TSDF errors on grasping.
  • Removing TSDF prediction and feeding aggregated scene features directly to grasp detection significantly degrades performance, supporting supervised geometric representation.
  • Removing volume rendering and its color loss reduces performance, particularly for transparent and specular object decluttering.

V. CONCLUSIONS

GraspNeRF extends generalizable NeRF to sparse-input, real-time 6-DoF grasping for transparent and specular objects, while jointly learning scene representation and grasp prediction. Experiments support its advantages over competing methods, but limited volume-grid resolution remains a bottleneck.

  • GraspNeRF uses generalizable NeRF for sparse-input, direct real-time 6-DoF grasp detection without per-scene optimization.
  • Jointly learning scene representation and grasping exploits their synergy, while domain-randomized synthetic data helps bridge the sim-to-real gap.
  • Limited volume-grid resolution may bottleneck geometry reconstruction and grasping performance.
Loading 2210.06575v3…