PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping

Adam Rashid, Satvik Sharma, Chung Min Kim, Justin Kerr, Lawrence Chen, Angjoo Kanazawa, Ken Goldberg

arXiv:2309.07970v2cs.ROcs.CV

TL;DR

Task-oriented grasping requires selecting safe, useful object parts, but conventional grasp planners typically optimize lifting success without semantic part awareness. LERF-TOGO combines DINO-based object masks with conditional LERF queries for zero-shot language-guided grasping, selecting the correct object part in 81% of trials and successfully grasping in 69%.

  • Problem

    Learning-based grasp planners typically measure success by lifting an object and therefore do not account for semantic object parts needed for safe, task-oriented grasping.

  • Method

    LERF-TOGO extracts a coarse 3D object mask with DINO features, then conditionally queries a multi-scale LERF to produce language-guided semantic grasp distributions over object parts.

  • Results

    LERF-TOGO grasps the correct object 96% of the time, directs grasps to the correct object part 81% of the time, and successfully grasps in 69% of trials.

  • Takeaways & Limitations

    The method supports zero-shot language-guided grasping of semantically meaningful object parts without requiring object-part training data.

  • Takeaways & Limitations

    The end-to-end process takes a few minutes, limiting practicality for time-sensitive applications, and connected foreground groups such as flower bouquets remain challenging.

Abstract

from arXiv · show

Grasping objects by a specific part is often crucial for safety and for executing downstream tasks. Yet, learning-based grasp planners lack this behavior unless they are trained on specific object part data, making it a significant challenge to scale object diversity. Instead, we propose LERF-TOGO, Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which uses vision-language models zero-shot to output a grasp distribution over an object given a natural language query. To accomplish this, we first reconstruct a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D language field queryable with text. However, LERF has no sense of objectness, meaning its relevancy outputs often return incomplete activations over an object which are insufficient for subsequent part queries. LERF-TOGO mitigates this lack of spatial grouping by extracting a 3D object mask via DINO features and then conditionally querying LERF on this mask to obtain a semantic distribution over the object with which to rank grasps from an off-the-shelf grasp planner. We evaluate LERF-TOGO's ability to grasp task-oriented object parts on 31 different physical objects, and find it selects grasps on the correct part in 81% of all trials and grasps successfully in 69%. See the project website at: lerftogo.github.io

1 Introduction

Task-oriented grasping requires selecting semantically appropriate object parts, but conventional learning-based planners generally optimize whether an object is lifted. LERF-TOGO addresses this gap with zero-shot language-guided grasp ranking and achieves strong physical-robot results.

  • Task-oriented grasping selects object parts such as knife handles or flower stems to satisfy safety and downstream-task constraints.
  • LERF-TOGO takes an object and natural-language part query as input, then ranks viable grasps on that object.
  • 96% of generated grasps target the correct object, 82% target the correct object part, and 69% successfully grasp the object across 39 household objects.
  • LERF-TOGO extracts a coarse DINO-based 3D object mask and conditions LERF queries on it to isolate semantic object parts.

2 Related Work

Prior task-oriented grasping methods commonly rely on affordance data, demonstrations, or object-part annotations, whereas LERF-TOGO uses free-form language over a 3D reconstruction. Its foundation is a multi-scale CLIP field that supports object- and part-level queries but requires explicit spatial grouping.

  • Task-Oriented Grasping: Task-oriented grasping has used human-grasp models, geometric part features, simulated affordance datasets, category transfer, and object-part grasp networks.
  • Task-Oriented Grasping: Unlike approaches requiring canonical images, part labels, or demonstrations, LERF-TOGO isolates object parts with free-form language queries in a 3D reconstruction.
  • NeRF and Language Fields: NeRF-based semantic and feature fields have also been developed for consistent 3D labels, feature fusion, editing, navigation, and region-proposal-based language mapping.
  • LERF Preliminaries: LERF distills CLIP features into a NeRF, producing scale-conditioned 3D language features that can be queried at different physical scales.
  • LERF Preliminaries: LERF supports long-tail queries without fine-tuning but produces nonuniform object activations because it lacks spatial grouping, motivating DINO-based object masks.

3 Problem and Assumptions

The task is to grasp a single language-specified object and part on a planar workspace using a robot with calibrated wrist-mounted vision. The pipeline reconstructs the scene, extracts an object mask, and ranks grasps by part relevance.

  • The robot must grasp and lift a single target object specified by natural-language object and part queries, such as “sunglasses; ear hooks.”
  • The pipeline reconstructs the scene, renders an object-centric point cloud, flood-fills DINO features into a 3D object mask, and re-ranks sampled grasps by part relevance.
  • The assumed setup uses a parallel-jaw gripper, a calibrated wrist-mounted RGB camera, and graspable objects in the scene.

4 Method

LERF-TOGO reconstructs a language-embedded 3D scene, groups points into an object mask, conditionally queries object parts, and ranks sampled grasps using semantic and geometric scores.

  • 4 Method: LERF-TOGO generates a 3D object mask with DINO-based grouping, conditionally queries LERF for part relevancy, and ranks viable grasps on the object part.The pipeline combines scene reconstruction, object extraction, semantic part localization, and grasp ranking.
  • 4.1 3D Object Extraction: DINO features produce a coarse foreground localization whose neighboring points are iteratively grown into a complete object mask using similarity thresholds.The process begins from a top-down LERF localization and expands a point-cloud region around the most relevant 3D point.
  • 4.2 Conditional LERF Queries: Conditional LERF queries search only within the object mask, yielding a semantic distribution over the object that represents the likelihood of each point being the desired part.This addresses CLIP’s bag-of-words behavior and limits part queries to the target object’s geometry.
  • 4.3 Grasping: GraspNet samples 6-DOF parallel-jaw grasps from multiple virtual-camera views to improve coverage of object geometry and target parts.The virtual-camera hemisphere leverages the full 3D geometry reconstructed by NeRF.
  • 4.3 Grasping: Grasp ranking combines median LERF relevancy within each grasp volume with GraspNet confidence using s = 0.95s_sem + 0.05s_geom.The weighting emphasizes semantic relevance while slightly favoring geometrically confident grasps.
  • 4.4 Scene Reconstruction: The method uses a wrist-mounted camera and hemispherical capture trajectory to collect multi-view imagery for LERF reconstruction.The system captures approximately 60 images per scene while extracting DINO, multi-scale CLIP, and depth features.

5 Experiments

Experiments evaluate LERF-TOGO on diverse object-part queries, language-model-generated task queries, and comparisons with alternative semantic localization methods.

  • 5 Experiments: LERF-TOGO is evaluated on 31 objects and 49 object parts, with natural-language object and part queries chosen to identify graspable locations.Queries include parts such as handles, plant stems, ear hooks, and frames.
  • Task-Oriented Grasping: An LLM generates object-part pairs from tasks using few-shot prompting, and majority voting selects the pair appearing in most of seven candidates.The setup investigates whether an LLM can generate the object and part query automatically.
  • 5.1 Comparisons: Single-view comparisons evaluate ConceptFusion, Semantic Abstraction, and OWL-ViT using inputs designed to expose all relevant object parts.The comparisons use 20 prompts across 5 scenes and adapt each method’s query or masking procedure to its interface.
  • Integration with an LLM Planner: LERF-TOGO integrates with an LLM planner that selects robotic primitives and object-part pairs for tasks such as uncorking wine.The supported primitives are grasp, press, twist, pick&place, and pour.
  • 5.1 Comparisons: ConceptFusion is tested with NeRF-derived depth and a larger OpenCLIP ViT-H/14 model, while Semantic Abstraction uses concatenated or separate object-part queries.OWL-ViT receives an RGB image, an object-prompt bounding box, and a part query within that region.

6 Results

LERF-TOGO supports language-guided grasping of objects and parts, outperforming comparison methods on task-oriented grasping while retaining strong object recognition. Its remaining failures include sensitivity to wording, subtle geometry, and similar parts.

  • 92% of LLM-generated task specifications identified the correct primitive, while grasps reached the correct object in 71% of 49 tasks.LLM object selection matched human performance, but part selection reached 71% versus 82% for humans.
  • LERF-TOGO outperformed ConceptFusion by 43% at task-oriented grasping through multi-scale semantics.ConceptFusion performed similarly on object selection but struggled with object-part selection because it used one CLIP embedding per point.
  • Semantic Abstraction achieved 80% object detection and 35% part detection, with smaller part features potentially diluted across scales.Concatenated object-and-part queries highlighted entire objects, reflecting CLIP bag-of-words behavior addressed by compositional queries.
  • OWL-ViT achieved 85% object localization accuracy but struggled more on long-tail object-part queries.Examples included “measuring tape” and “ethernet dongle.”
  • Primary failures involved visually similar parts, subtle geometries, close object categories, and prompt-sensitive part localization.Examples include confusing a teapot spout with its handle and “body” with “base.”
  • Without 3D object masking and conditional querying, LERF produced fragmented part-query results that could ignore relevant regions on oblong objects.The ablation compared separate object and part queries with multiplying their relevancy results.

7 Limitations and Future Work

LERF-TOGO is constrained by runtime, connected foreground groups, ambiguous multiple instances, and unsupported referring expressions. Future work must improve efficiency and hierarchical scene disambiguation.

  • End-to-end processing takes a few minutes, which can be impractical for time-sensitive applications.The authors suggest additional LERF-training regularization and optimization as possible remedies.
  • DINO flood-fill can group connected flowers together, making it difficult to isolate one flower’s stem within the foreground group.The authors identify hierarchy within foreground groups as critical for such cases.
  • When multiple objects match a prompt, the system arbitrarily selects one, and it is not designed for referring or comparative expressions.Examples include “mug next to the plate” and “biggest mug.”

8 Conclusion

LERF-TOGO uses zero-shot vision-language models with Language Embedded Radiance Fields to grasp objects and their parts through language. By improving spatial grouping, it supports hierarchical part queries conditioned on the full object.

  • LERF-TOGO uses zero-shot vision-language models with Language Embedded Radiance Fields to grasp objects and parts through language.
  • Improved spatial grouping enables hierarchical part queries conditioned on the full object.
  • The method grasps the correct object 96% of the time and directs grasps to the correct object part 81% of the time.

A.1 Implementation Details

The implementation builds LERF-TOGO on Nerfacto and modifies the NeRF configuration for faster convergence, smoother optimization, and better handling of specularities and robot shadows.

  • LERF-TOGO is implemented on Nerfacto with a 16-level hashgrid, maximum resolution 256, larger output-head MLPs, and weight decay of 1e-7.The modifications target faster convergence, smoother optimization, specularity handling, and robot-shadow robustness.

B Robot Capture

The robot capture pipeline reconstructs object-centric 3D representations for grasping, with capture coverage and training choices affecting semantic quality and fine-grained feature emergence.

  • Robot Capture: Reduced robot scan coverage degrades the 3D object mask and can select the incorrect object.LERF’s semantic field quality depends heavily on the distribution of scene views.
  • Robot Capture: Fine-grained or small parts may require 2–3k LERF training steps, whereas larger objects and parts can emerge by 1k steps.The representation is trained to 2k steps in the reported experiments.
  • Robot Capture: Object-centric point clouds are built by rendering six views around the highest-activation 3D point and using DINO similarity for floodfill extraction.The extracted point cloud supports object-focused processing for grasping.
  • Robot Capture: Staged training delays depth regularization until camera poses settle, preserving thin features and speeding LERF optimization.The system trains without regularization for 500 steps, then anneals the depth loss over the next 1500 steps.
  • Robot Capture: Ambiguous or repeated-object queries select one object rather than proposing grasps for every instance in the category.This is a scope boundary for scene interpretation with identical objects or ambiguous language.

D Experiments

The experiments integrate the system with a physical robot, language-driven task prompts, semantic grasp weighting, and object or part localization evaluations across varied queries and scenes.

  • D Experiments: The physical-robot system parallelizes DINO, CLIP, and ZoeDepth computation across three NVIDIA 4090 GPUs for real-time capture processing.The robot uses a UR5 arm and Logitech BRIO webcam at 1600×896 resolution.
  • D Experiments: An LLM converts scene object lists and natural-language tasks into a basic action plus an object-part sequence for the robot.Supported basic actions include press, grasp, twist, and pick & place.
  • D Experiments: A high semantic grasp weight of w = 0.95 is required because geometric scoring can bias grasps away from small, fine-grained object parts.Both geometric and semantic grasp scores range from 0 to 1.
  • D Experiments: Example task prompts pair actions with target parts, such as grasping a knife by its handle or lifting a salt shaker by its base.These examples illustrate how language specifies the intended contact region.
  • D Experiments: The evaluation covers object and part queries, task-specific examples, and semantic localization results from Semantic Abstraction and OWL-ViT.The supplied tables enumerate object-part queries and tasks for each scene.
Loading 2309.07970v2…