PaperlayerView exact revision on arXiv

Source-linked AI summary

PGA: Personalizing Grasping Agents with Single Human-Robot Interaction

Junghyun Kim, Gi-Cheon Kang, Jaein Kim, Seoyun Yang, Minjoon Jung, Byoung-Tak Zhang

arXiv:2310.12547v2cs.ROcs.CVcs.LG

TL;DR

LCRG systems generally use generic language, while personal-object requests require users to describe objects in robot-suitable terms. PGA introduces GraspMine and uses interaction data with propagation through Reminiscence to personalize grounding; it outperforms the one-sample Direct model and is comparable to a model trained on nearly 9k annotations.

  • Problem

    Current LCRG systems predominantly use generic expressions, so requests for personal objects such as “my wallet” may require users to specify color or location.

    Source: §I · p. 1

  • Method

    PGA personalizes object grounding by assigning personal indicators to Reminiscence objects through visual-feature-based propagation.

    Source: C. Propagation through Reminiscence · p. 3

  • Results

    Approximately 30% average absolute increment over Direct, with comparable object-grounding results to Supervised trained on nearly 9k annotated samples.

    Source: §I · p. 2

  • Takeaways & Limitations

    The conclusion presents PGA as a way to shift interaction beyond generic shared knowledge toward more natural, intuitive exchanges.

    Source: §VII · p. 7

    Online grasping accuracy drops around 13% from grounding accuracy, with the motion-planning strategy potentially ineffective for objects such as wine bottles.

    Source: §VI · p. 7

Abstract

from arXiv · show

Language-Conditioned Robotic Grasping (LCRG) aims to develop robots that comprehend and grasp objects based on natural language instructions. While the ability to understand personal objects like my wallet facilitates more natural interaction with human users, current LCRG systems only allow generic language instructions, e.g., the black-colored wallet next to the laptop. To this end, we introduce a task scenario GraspMine alongside a novel dataset aimed at pinpointing and grasping personal objects given personal indicators via learning from a single human-robot interaction, rather than a large labeled dataset. Our proposed method, Personalized Grasping Agent (PGA), addresses GraspMine by leveraging the unlabeled image data of the user's environment, called Reminiscence. Specifically, PGA acquires personal object information by a user presenting a personal object with its associated indicator, followed by PGA inspecting the object by rotating it. Based on the acquired information, PGA pseudo-labels objects in the Reminiscence by our proposed label propagation algorithm. Harnessing the information acquired from the interactions and the pseudo-labeled objects in the Reminiscence, PGA adapts the object grounding model to grasp personal objects. This results in significant efficiency while previous LCRG systems rely on resource-intensive human annotations -- necessitating hundreds of labeled data to learn my wallet. Moreover, PGA outperforms baseline methods across all metrics and even shows comparable performance compared to the fully-supervised method, which learns from 9k annotated data samples. We further validate PGA's real-world applicability by employing a physical robot to execute GrsapMine. Code and data are publicly available at https://github.com/JHKim-snu/PGA.

I. INTRODUCTION

GraspMine asks robots to locate and grasp personal objects from personal indicators after minimal interaction, addressing LCRG systems that rely on generic descriptions. PGA combines user interaction, robot inspection, and propagation through environmental images to learn these objects, outperforming the one-sample baseline and approaching a nearly 9k-sample supervised model.

I. INTRODUCTION

  • The task targets the gap between personal requests such as “Get my wallet” and generic descriptions that require users to specify color or location.The paper frames this as a move toward more intuitive, user-centric interaction.Source: §I · pp. 1–2
  • GraspMine is a personalized LCRG task and benchmark for grounding and grasping personal objects from indicators after a one-time verbal introduction.Its dataset covers 96 personal objects and five test splits.Source: §I · pp. 1–2
  • PGA collects unlabeled environmental images, acquires personal-object information through human-robot and robot-object interaction, then propagates indicators to adapt an object-grounding model.Robot-object interaction captures multiple views; pseudo-labeling uses visual features of objects in Reminiscence.Source: §I · p. 2
  • PGA improves object grounding by an average absolute increment of approximately 30% over Direct and shows comparable results to Supervised, trained with nearly 9k annotated samples.The experiments also report real-world execution on a physical robot.Source: §I · p. 2
  • The paper reports comprehensive baseline experiments and demonstrates PGA’s practical applicability by deploying it on a physical robot.Visual examples are presented from different phases of PGA.Source: §I · p. 2

II. RELATED WORK

Prior LCRG work includes supervised systems and an unsupervised personalization framework, but these approaches leave limitations for personal-object language and grounding. Related personalization work addresses visual and linguistic representations, while differing in its grounding task or annotation requirements.

A. Language-Conditioned Robotic Grasping (LCRG)

  • Many LCRG systems use fully supervised training and public datasets, while GVCCI adapts to personal objects unsupervised but relies on generic categories and attributes, leaving users with robot-centric instructions.PGA instead learns personal objects and their language descriptors from a single user-robot interaction.Source: A. Language-Conditioned Robotic… · p. 2

B. Personalization

  • Personalization work includes temporal grounding from videos that omits spatial grounding and spatial grounding that assumes fine-grained annotations and personal object names.PGA is distinguished by requiring a single human-robot interaction without further annotations.Source: B. Personalization · p. 2

A. Reminiscence Construction

PGA builds Reminiscence from detected objects in raw images of the user’s environment, then acquires personal-object information through a verbal introduction and inspection from multiple views.

A. Reminiscence Construction

  • PGA collects environmental images, detects objects, and represents each detected object as a visual-encoder embedding linked to its source image and bounding box.The Reminiscence nodes initially lack personal indicators.Source: A. Reminiscence Construction · p. 3

B. Object Information Acquisition

  • During human-robot interaction, the user displays and verbally describes a personal object with a general indicator and a personal indicator; PGA predicts its bounding box and creates a tagged object node.The general indicator can identify the object by position, while the personal indicator can be a phrase such as “my sleeping pills.”Source: B. Object Information Acquisition · p. 3
  • During robot-object interaction, PGA grasps and rotates the object to capture multiple views, creating view-specific nodes that share the personal indicator with the introduced-object node.The introduced object and its view nodes together form A + 1 indicator-tagged nodes for each personal object.Source: B. Object Information Acquisition · p. 3

C. Propagation through Reminiscence

PGA propagates personal indicators from known objects to unlabeled Reminiscence objects using affinity scores. It iteratively adds confident pseudo-labels to the known set until fewer than 10% of nodes change labels.

C. Propagation through Reminiscence

  • Propagation transfers personal indicators from known nodes in OK to nodes in Reminiscence OR, whose indicators are missing.The known set is updated with propagated nodes for subsequent iterations.Source: C. Propagation through Reminiscence · p. 3
  • For each Reminiscence node, PGA averages cosine similarity to known object nodes for each personal indicator and assigns the highest-affinity indicator when it exceeds a threshold.The affinity score is computed separately for each possible indicator.Source: C. Propagation through Reminiscence · p. 3
  • Propagation repeats until the percentage of relabeled nodes falls below 10%, automatically gathering pseudo-labeled examples to alleviate labeled-data scarcity.The stopping condition is based on the ratio of changed indicators.Source: C. Propagation through Reminiscence · p. 4

D. Personalized Object Grounding Model

The Personalized Object Grounding Model is a Transformer-based vision-language model that predicts personal-object bounding boxes from images and language indicators. It is optimized on object nodes after propagation.

D. Personalized Object Grounding Model

  • The Personalized Object Grounding Model takes an image and natural-language indicator as input and infers the personal object's bounding-box coordinates.It is a Transformer-based vision-language model.Source: D. Personalized Object Grounding Model · p. 4
  • The model minimizes the negative log-likelihood of the bounding box for every object node in OK.The training nodes are the set after Propagation through Reminiscence, not the earlier set from Sec. III-B.Source: D. Personalized Object Grounding Model · p. 4
  • The optimization uses OK after Propagation through Reminiscence rather than the earlier OK from Sec. III-B.This specifies which object-node set supplies the bounding-box training examples.Source: D. Personalized Object Grounding Model · p. 4

E. Personalized Object Grasping

Personalized Object Grasping starts by grounding the queried object in 2D, then uses point-cloud processing to identify its 3D points and plans a robot-arm trajectory to grasp it.

E. Personalized Object Grasping

IV. GraspMine

GraspMine combines training examples from human-robot interaction, optional unlabeled Reminiscence images, and a test set for personal-object grounding. Its dataset includes personal and everyday objects, with test splits designed to assess varied scenes and indicators.

IV. GraspMine

  • GraspMine contains 96 personal objects and more than 100 everyday objects, totaling around 200 individual objects.The curated dataset includes a Training set, Reminiscence, and Test set.Source: §IV · p. 4
  • The Training set contains 96 image–personal-indicator pairs collected through human-robot interaction.The passage also mentions associated general indicators.Source: §IV · p. 4
  • Reminiscence consists of 400 raw images containing multiple objects; its annotations support analysis and Supervised training but are not used to train PGA.The unlabeled images can optionally aid learning.Source: §IV · p. 4
  • The test set includes heterogeneous, homogeneous, cluttered, paraphrased, and generic splits, varying object similarity, scene clutter, indicator wording, and generic-object coverage.The cluttered split comes from IM-Dial, while the generic split uses VGPI's Test-E split.Source: §IV · p. 4

A. Compared Methods

PGA is compared with single-sample and fully supervised baselines for personal-object grounding. Offline, it outperforms Direct by about 30% and achieves results comparable to Supervised.

B. Offline Experiment

A. Compared Methods

  • Direct uses 96 interaction labels, PassivePGA adds 4.7k pseudo-labeled examples, and Supervised trains on 8.7k Reminiscence annotations as an upper-bound benchmark.Direct averages 1.1 human-robot interactions per object; the Supervised model requires approximately 100 times more manual effort than PGA.Source: A. Compared Methods · p. 5

B. Offline Experiment

  • Robot-object interaction improves PGA by up to 8.5% over PassivePGA; PGA's propagation accuracy is 79.4% versus 64.5%.The paper attributes the improvement to capturing objects from different views.Source: B. Offline Experiment · pp. 5–6
  • After learning personal indicators, PGA retains its ability to ground objects from generic instructions.Source: B. Offline Experiment · p. 5
  • PGA reduces propagation to invalid boxes to 0.07%, versus 83.7% for PassivePGA, and to ambiguous boxes to 41.2%, versus 50.4%.The paper identifies off-the-shelf detector noise as the main source of propagation errors.Source: B. Offline Experiment · p. 6
  • Across trials using 25, 100, and 400 images, offline performance improves incrementally with Reminiscence size; zero images is equivalent to Direct.The figure describes scores across raw-image counts from 0 to 400.Source: B. Offline Experiment · p. 6, Figure 3 · p. 6

C. Online Experiment

In physical-robot tests, PGA outperformed Direct in grasping across both evaluated splits. Qualitative examples show how object inspection supports propagation across viewpoints, while ambiguous boxes and cluttered scenes remain challenging.

C. Online Experiment

  • PGA's grasping performance gap over Direct was 23.4% in the Heterogeneous split and 33.3% in the Homogeneous split, with enhanced grounding accuracy.The comparisons used scenes reproduced from randomly selected images and human-evaluated grounding success and grasping accuracy.Source: C. Online Experiment · p. 6
  • The physical-robot evaluation used a Kinova Gen3 Lite with a Realsense D435, and assessed human-rated grounding success and grasping accuracy.Learning and inference ran on a remote server, while manipulation planning ran locally.Source: C. Online Experiment · p. 6

D. Qualitative Analysis

  • PGA's object inspection covered varied viewpoints, allowing it to propagate labels to differently viewed samples that PassivePGA often missed.The charger example shows PGA handling views unlike those seen during human-robot interaction, while PassivePGA missed some of them.Source: D. Qualitative Analysis · p. 6, Figure 4 · p. 7
  • PassivePGA sometimes propagated toward ambiguous boxes, which could lead to inaccurate grounding.The qualitative analysis points to an incorrect red-box prediction in sample A and suggests effects on some results in panel B.Source: D. Qualitative Analysis · p. 7
  • Highly cluttered environments remained challenging, as reflected in the lower scores for the Cluttered split.The paper highlights scenario E as an example of objects situated in clutter.Source: D. Qualitative Analysis · p. 7

VI. DISCUSSION

The discussion identifies boundaries in PGA's setup, including a fixed table area and pre-acquired Reminiscence images, as well as grasping limits from motion planning. The conclusion presents GraspMine and PGA as a step toward user-centric interaction.

VI. DISCUSSION

  • The arm robot confines execution to an 80*60 cm table with objects at least partially visible, excluding retrieval from handbags or shelves.The authors leave deeper, dynamically adaptive motion planning for future work.Source: §VI · p. 7
  • PGA assumes access to pre-acquired raw Reminiscence images; autonomous acquisition by a mobile robot is left for future study.The paper notes random navigation as a feasible way to obtain such images.Source: §VI · p. 7
  • A roughly 13% drop from grounding accuracy to grasping accuracy was attributed to motion-planning limits, including horizontal-first alignment that may fail for wine bottles.The authors identify object-specific grasping strategies as a future need.Source: §VI · p. 7

VII. CONCLUSION

  • GraspMine and PGA aim to support learning a user's personalized knowledge through one human-robot interaction and promote a shift toward user-centric interactions.The authors anticipate more natural, intuitive interaction and an improved non-expert user experience.Source: §VII · p. 7
Loading 2310.12547v2…