PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-Grasp: a 6-Dof Interactive Grasp Policy for Language-Oriented Objects in Cluttered Indoor Scenes

Yuhao Lu, Yixuan Fan, Beixing Deng, Fangfu Liu, Yali Li, Shengjin Wang

arXiv:2308.00640v1cs.RO

TL;DR

Interactive grasping must locate a language-specified target and predict a grasp pose despite restrictions in prior approaches and limited robotics-focused grounding data. VL-Grasp combines visual grounding with 6-Dof grasp detection and uses RoboRefIt for training; its average success rate across different scenes is 72.5%.

  • Problem

    Prior interactive grasping approaches restrict observation views, and popular grounding datasets have limited alignment with robotics and few ambiguous same-category samples.

    Source: §I · p. 1, 2) Comparative Results on Different… · p. 6

  • Method

    VL-Grasp combines visual grounding and 6-Dof grasp pose detection, using RoboRefIt as a visual grounding dataset for robot reasoning.

    Source: §I · p. 2

  • Results

    72.5% average success rate was achieved by VL-Grasp in different indoor scenes.

    Source: 2) Comparative Results on Different… · p. 6

  • Takeaways & Limitations

    The authors conclude that VL-Grasp has expansibility and adaptability for practical grasping.

    Source: §VI · p. 6

    Open-world interactive grasping with unseen objects remains future work.

    Source: §VI · p. 6

Abstract

from arXiv · show

Robotic grasping faces new challenges in human-robot-interaction scenarios. We consider the task that the robot grasps a target object designated by human's language directives. The robot not only needs to locate a target based on vision-and-language information, but also needs to predict the reasonable grasp pose candidate at various views and postures. In this work, we propose a novel interactive grasp policy, named Visual-Lingual-Grasp (VL-Grasp), to grasp the target specified by human language. First, we build a new challenging visual grounding dataset to provide functional training data for robotic interactive perception in indoor environments. Second, we propose a 6-Dof interactive grasp policy combined with visual grounding and 6-Dof grasp pose detection to extend the universality of interactive grasping. Third, we design a grasp pose filter module to enhance the performance of the policy. Experiments demonstrate the effectiveness and extendibility of the VL-Grasp in real world. The VL-Grasp achieves a success rate of 72.5\% in different indoor scenes. The code and dataset is available at https://github.com/luyh20/VL-Grasp.

I. INTRODUCTION

The paper targets language-directed grasping in settings where existing 2D approaches restrict viewpoints and available grounding datasets poorly match robotic grasping. It proposes VL-Grasp and RoboRefIt to support grasping across more views and indoor scenes.

I. INTRODUCTION

  • Prior 2D-plane interactive grasping restricts robot viewpoints and object placement, while available grounding datasets offer limited robotic relevance and ambiguous-object samples.The paper notes that these limits constrain use in diverse environments such as under shelves.Source: §I · p. 1
  • RoboRefIt is a new visual grounding dataset for robot reasoning, built around 66 categories of graspable desktop objects in varied indoor scenes.The paper reports that experiments validate its functionality and usability.Source: §I · p. 2
  • VL-Grasp combines visual grounding with 6-Dof grasp pose detection to support interactive grasping across multiple views and scenes.Its point cloud filter converts scene-level depth into an object-level point cloud.Source: §I · pp. 1–2
  • The proposed point cloud filter module is designed to improve grasping success in complex scenes.Source: §I · p. 2

II. RELATED WORKS

Related work covers visual grounding methods and datasets, alongside 2D-plane and 6-Dof grasp pose detection. The discussion describes how prior grounding approaches and datasets have been constructed across computer vision and robotics.

a) Visual Grounding

  • Visual grounding locates a target from a sentence using a bounding box for REC or a mask for RES, with methods combining visual and language features.The section describes CNN-LSTM and transformer approaches, as well as unified frameworks with shared backbones and task-specific heads.Source: §II · p. 2

b) Visual Grounding Datasets

  • Visual grounding datasets include manually annotated expressions, game-based collection, and automatic generation from scene graphs or structured templates.ScanRefer and SUNRefer annotate 3D target boxes; OCID-Ref adapts OCID images and labels for robotics.Source: §II · p. 2

c) Grasp Pose Detection

  • Grasp pose detection is divided into 2D-plane grasping and 6-Dof grasping, with the former limiting gripper posture perpendicular to the desktop.Source: §II · p. 2
  • The related work describes deep-learning 6-Dof methods, including a large-scale grasp dataset and approaches for proposals, affordance constraints, transparent objects, and novel objects.GraspNet-1Billion provides a large-scale grasp dataset and a baseline for learning grasp poses.Source: §II · p. 2

III. ROBOREFIT DATASET

RoboRefIt is a real-world indoor visual grounding dataset designed for robotic perception and reasoning. Its data collection and expression-generation process covers varied scenes, objects, and referring expressions.

III. ROBOREFIT DATASET

  • RoboRefIt contains 10,872 RGB-D images and 50,758 referring expressions from 187 real-world indoor scenes and 66 object categories.The dataset includes 5,636 images with multiple objects of the same kind, and provides 2D bounding boxes and segmentation masks.Source: §III · p. 2

A. Data collection

  • The collection uses 23 object categories from GraspNet-1Billion and 43 additional categories, placing objects across six kinds of daily-life scenes.Scenes contain three to ten objects, and multiple scenes include two or three objects of the same category; images are captured from different directions.Source: A. Data collection · p. 2
  • In the scenario examples, red boxes identify the referred targets, while expression colors distinguish templates, attribution and object, and location.Source: Figure 2 · p. 3
  • RoboRefIt uses a semiautomatic referring-expression generation engine that combines text synthesis with manual annotation.The engine creates diverse, complex, unambiguous expressions for objects in scene images.Source: A. Data collection · p. 3
  • Robot directives are composed from a beginning template, attribution, object, and location components.The location component helps distinguish objects when multiple similar or identical instances appear; the engine randomly samples and combines phrase embeddings.Source: A. Data collection · p. 3

B. Data annotation

The section describes RoboRefIt’s annotation pipeline: REC receives 2D boxes and RES masks, with RGB annotations applicable to aligned depth images.

B. Data annotation

  • RoboRefIt provides 2D bounding-box labels for REC and segmentation masks for RES, applicable to aligned depth images as well as RGB images.Source: B. Data annotation · p. 3
  • Experienced annotation experts mark object boxes and masks from multiple image descriptions, supplementing expressions until each identifies a unique, unambiguous object.Source: B. Data annotation · p. 3
  • Professional checkers inspect preliminary annotations in at least three rounds.Source: B. Data annotation · p. 3

C. Comparison with Existing Datasets

RoboRefIt is distinguished from prior visual-grounding datasets by its focus on robotic indoor grasping, annotation choices, scale, object diversity, and ambiguous samples. The paper highlights its relevance to graspable desktop objects and cluttered scenes.

C. Comparison with Existing Datasets

  • RoboRefIt emphasizes graspable desktop objects and cluttered indoor scenes, unlike datasets with few human-robot interaction scenes.The paper describes RoboRefIt as the largest indoor dataset by image count among the compared datasets.Source: C. Comparison with Existing Datasets · p. 3
  • RoboRefIt includes segmentation masks, which the paper identifies as important for post-filtering 6-Dof grasp poses.The comparison notes that several indoor datasets do not annotate segmentation masks.Source: C. Comparison with Existing Datasets · p. 3
  • RoboRefIt includes many manually set ambiguous samples, whereas the compared indoor datasets’ source datasets contain few such samples.The paper identifies content relevance, annotation and image-data format, scale, object diversity, and ambiguity as comparison characteristics.Source: C. Comparison with Existing Datasets · p. 3

IV. VL-GRASP

VL-Grasp is a two-stage policy for predicting a target object’s 6-Dof grasp pose from RGB-D input and a natural-language command. It first locates the target, then uses its position results and depth information to produce a grasp configuration.

IV. VL-GRASP

A. Problem Statement

  • Given RGB and depth images plus a command referring to one object, the task is to predict that target’s optimal 6-Dof grasp pose.The first stage outputs a 2D bounding box and segmentation mask; the second uses those results with depth to produce the grasp configuration.Source: A. Problem Statement · p. 4

B. Overview

  • Stage I grounds the command in the RGB image, while Stage II converts the target’s position results and depth into an optimal candidate grasp configuration.The grounding model is trained on RoboRefIt; the second stage first obtains an object-region point cloud before grasp-pose detection.Source: B. Overview · p. 4

C. Visual Grounding Network

VL-Grasp grounds language in an RGB image, filters depth-derived point clouds around the target, and predicts and filters 6-Dof grasp proposals. Its stages combine visual grounding with point-cloud-based pose detection.

C. Visual Grounding Network

  • Stage I uses Referring Transformer to predict a target bounding box and segmentation mask from the command and RGB image.BERT and ResNet extract language and image features; cross-modal attention fuses them before REC and RES decoders produce the outputs.Source: C. Visual Grounding Network · pp. 4–5

D. Point Cloud Filter

  • The point cloud filter crops depth using a dilated bounding box, then converts the crop into a single-view point cloud.Dilation retains more complete object geometry and edge-background geometry; experiments report that the filter benefits grasp-pose detection and real-robot grasping.Source: D. Point Cloud Filter · p. 5

E. 6-Dof Grasp Pose Detection Network

  • The grasp network predicts multiple 6-Dof proposals and selects the highest-confidence candidate after filtering poses through the object mask.Low-resolution features predict approach directions and a foreground mask; high-resolution features predict rotation, depth, and confidence.Source: E. 6-Dof Grasp Pose Detection Network · p. 5

V. EXPERIMENTS

The experiments describe RoboRefIt’s train, testA, and testB split and identify the purpose of each test set. The supplied table caption specifies that it reports REC and RES results under RefTR on RoboRefIt.

A. Data Split

  • RoboRefIt divides its original data into train, testA, and testB sets to evaluate adaptability across scene distributions.The supplied passages do not provide the split sizes or table values.Source: A. Data Split · p. 5

V. EXPERIMENTS

  • Table II reports REC and RES results under RefTR on RoboRefIt.The supplied passage gives the table’s scope but no result values.Source: Table II · p. 5

A. Data Split

  • TestA measures basic prediction ability, while testB measures generalization to a different scene distribution.Train and testA use stratified random sampling from selected scenes; testB uses another set of scenes, including scenes absent from training.Source: A. Data Split · p. 5

B. REC and RES Experiments

The RoboRefIt experiments benchmark referring-expression comprehension and segmentation, using the results to select VL-Grasp’s Stage I backbone and input. The real-robot experiments describe the deployment setup for evaluating grasping.

B. REC and RES Experiments

  • ResNet50 with RGB input is selected for VL-Grasp Stage I after ablations of image backbones and input types.The experiments also consider depth-only and RGB-D inputs.Source: B. REC and RES Experiments · pp. 5–6
  • REC is evaluated with prec@0.5 and RES with mIoU on RoboRefIt’s testA and testB sets.prec@0.5 measures the share of test images with bounding-box IoU above 0.5; mIoU compares predicted and ground-truth masks.Source: B. REC and RES Experiments · pp. 5–6

C. Real Robot Experiments

  • Real-robot evaluation uses a UR3 arm, RG2 gripper, D455 camera, and mobile robot; VL-Grasp trains grounding on RoboRefIt and GPD on GraspNet-1Billion.The experiments use practical object scenes and natural-language commands similar to the datasets.Source: C. Real Robot Experiments · p. 6

1) Comparative Results on Object Categories:

VL-Grasp generalizes across RoboRefIt object categories, including similar-shape objects unseen by the trained GPD model. In varied indoor scenes, its average grasp success rate is 72.5%.

1) Comparative Results on Object Categories:

  • VL-Grasp generalizes to diverse RoboRefIt object categories and can grasp similar-shape objects unseen by the trained GPD model.The evaluation distinguishes public objects seen in both RoboRefIt and GraspNet-1Billion from GPD-unseen objects seen in RoboRefIt but not GraspNet-1Billion.Source: 1) Comparative Results on Object… · p. 6

2) Comparative Results on Different Scenes:

3) Ablation Results on Point Cloud Filter:

The point cloud filter improves grasping in real scenes compared with the two tested filtering strategies. The conclusion presents VL-Grasp and RoboRefIt as contributions and identifies open-world grasping with unseen objects as future work.

3) Ablation Results on Point Cloud Filter:

  • The point cloud filter predicts grasp poses far better than no pre-filter or mask pre-filter and improves real-environment grasp success.Without pre-filtering, clutter can produce inaccurate poses; mask pre-filtering removes environmental context useful for judging object position and posture.Source: 3) Ablation Results on Point Cloud… · p. 6, Figure 4 · p. 7

VI. CONCLUSION

  • VL-Grasp combines visual grounding with 6-Dof grasp detection, while open-world grasping with unseen objects remains future work.The paper also introduces RoboRefIt as a visual grounding dataset for robotic perception and reasoning in indoor environments.Source: §VI · p. 6

Related papers

Loading 2308.00640v1…