PaperlayerView exact revision on arXiv

Source-linked AI summary

GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Grasping

Yuhang Zheng, Xiangyu Chen, Yupeng Zheng, Songen Gu, Runyi Yang, Bu Jin, Pengfei Li, Chengliang Zhong, Zengmao Wang, Lina Liu, Chao Yang, Dawei Wang, Zhen Chen, Xiaoxiao Long, Meiqing Wang

arXiv:2403.09637v1cs.ROcs.CV

TL;DR

Language-guided robotic manipulation needs scene representations that support open-ended queries, but existing approaches have limitations in view requirements, localization, inference, and scene changes. GaussianGrasper builds an explicit Gaussian feature field from RGB-D views and uses it for language-based localization and grasping. The authors report improved segmentation and localization over other approaches, faster querying than LERF, and real-world manipulation effectiveness.

  • Problem

    Existing scene-understanding approaches for language-guided manipulation face limitations including imprecise localization, dense-view requirements, slow inference, and difficulty handling scene changes.

    Source: §I · p. 1

  • Method

    GaussianGrasper uses an explicit 3D Gaussian field with efficiently distilled language features and rendered geometry to support language-guided object localization and grasping.

    Source: §I · p. 2, 2) Geometry Restruction · p. 5

  • Results

    The method significantly outperforms other approaches on segmentation and localization and achieves an approximately 180× query-speedup over LERF at 640 × 480 resolution.

    Source: B. Results of Efficient Feature… · p. 6

  • Takeaways & Limitations

    Real-world experiments demonstrate GaussianGrasper’s effectiveness for language-guided object querying and grasping.

    Source: §VI · p. 8

    The reconstructed scene remains static between recorded updates, and the method cannot estimate depth or normals for transparent objects.

    Source: §V · pp. 7–8

Abstract

from arXiv · show

Constructing a 3D scene capable of accommodating open-ended language queries, is a pivotal pursuit, particularly within the domain of robotics. Such technology facilitates robots in executing object manipulations based on human language directives. To tackle this challenge, some research efforts have been dedicated to the development of language-embedded implicit fields. However, implicit fields (e.g. NeRF) encounter limitations due to the necessity of processing a large number of input views for reconstruction, coupled with their inherent inefficiencies in inference. Thus, we present the GaussianGrasper, which utilizes 3D Gaussian Splatting to explicitly represent the scene as a collection of Gaussian primitives. Our approach takes a limited set of RGB-D views and employs a tile-based splatting technique to create a feature field. In particular, we propose an Efficient Feature Distillation (EFD) module that employs contrastive learning to efficiently and accurately distill language embeddings derived from foundational models. With the reconstructed geometry of the Gaussian field, our method enables the pre-trained grasping model to generate collision-free grasp pose candidates. Furthermore, we propose a normal-guided grasp module to select the best grasp pose. Through comprehensive real-world experiments, we demonstrate that GaussianGrasper enables robots to accurately query and grasp objects with language instructions, providing a new solution for language-guided manipulation tasks. Data and codes can be available at https://github.com/MrSecant/GaussianGrasper.

I. INTRODUCTION

Language-guided manipulation requires both locating objects from language and understanding their geometry well enough to grasp them. GaussianGrasper uses an explicit 3D Gaussian field with distilled language features and geometry to support localization, grasp selection, and scene updates.

I. INTRODUCTION

  • For the “hamburger” query, GaussianGrasper produces more precise 3D localization than 2D feature fusion and LERF in the figure comparison.The paper reports zero-shot generalization in real-world household tabletop scenes and common-object manipulation tasks.Source: Figure 1 · p. 1, §I · p. 2
  • Distilled feature fields can have ambiguous boundaries, require many views, infer slowly, and handle manipulation-driven scene changes poorly.The paper also identifies inconsistent semantics across views as a challenge for 2D-to-3D fusion.Source: §I · pp. 1–2
  • GaussianGrasper represents scenes as 3D Gaussian primitives and reconstructs a feature field for language-guided localization.Its design also renders depth and surface normals for grasping and updates the scene by operating on Gaussians and fine-tuning 3DGS.Source: §I · p. 2
  • Its EFD module uses SAM segmentation priors and contrastive learning to distill CLIP features while addressing computational expense and boundary ambiguity.The system scans a few RGB-D viewpoints, then uses open-vocabulary queries and a pre-trained grasping module; rendered normals screen grasp proposals using Force-closure theory.Source: §I · p. 2

II. RELATED WORK

Prior grasp detection approaches use 2D predictions or 3D geometry, while 3D feature-field methods for manipulation face view, update, and feature-computation constraints. GaussianGrasper combines explicit 3D Gaussians with SAM segmentation priors in an efficient feature-distillation module.

A. Grasp Pose Detection

  • Earlier 2D grasp detectors predict 3-DoF poses, whereas 6-DoF methods use depth or point clouds and can lose performance on photometrically challenging objects such as transparent ones.RGB-D grasp-pose generation is used here, with estimated normals screening infeasible poses using Force-closure theory.Source: A. Grasp Pose Detection · p. 2

B. Reconstructing 3D Feature Field For Manipulation

  • Feature-field methods for manipulation can require many viewpoints and lack rapid scene updates; explicit feature fusion reduces viewpoint needs but incurs memory and computation costs.GaussianGrasper’s explicit Gaussian distillation module uses SAM segmentation priors to speed feature reconstruction and reduce memory use.Source: B. Reconstructing 3D Feature Field For… · p. 2

III. METHODOLOGY

The method initializes a 3D Gaussian field from multi-view RGB-D images, reconstructs a feature field, and supports language-guided manipulation. Its differentiable rasterizer projects and depth-sorts Gaussian primitives before accumulating pixel values.

III. METHODOLOGY

  • The pipeline takes multi-view RGB-D images to initialize a Gaussian field, reconstructs a feature field through EFD, then supports language-guided manipulation.The manipulation stages include object localization, grasp-pose generation, normal-guided pose selection, and scene updating.Source: §III · pp. 2–3, Figure 2 · p. 3

1) Gaussian Primitive Initialization:

  • Depth-camera pixels are reprojected into world coordinates to initialize Gaussian centers, with rotation and scale initialized randomly.Each Gaussian is defined by a mean and a 3D covariance matrix constructed from rotation and scaling.Source: 1) Gaussian Primitive Initialization · p. 3

2) Differentiable Rasterizer for 3D Gaussians:

  • The differentiable rasterizer projects Gaussians onto the image plane, sorts them by depth, and accumulates pixel values through tile-based rasterization.The pixel-color expression uses feature vectors and Gaussian-weighted opacity.Source: 2) Differentiable Rasterizer for 3D… · p. 3

B. Efficient Feature Distillation

GaussianGrasper reconstructs an open-vocabulary 3D feature field by distilling dense CLIP features into 3D Gaussian primitives. SAM instance masks provide shape-aware priors for feature extraction and contrastive distillation.

B. Efficient Feature Distillation

1) Instance-level Segmentation Prior and Open-vocabulary Features:

  • SAM mask proposals are filtered with non-maximum suppression, then CLIP features from valid mask-aligned regions form feature map F.The masks provide instance-level segmentation priors.Source: 1) Instance-level Segmentation Prior… · p. 4

2) Open-vocabulary Feature Distillation:

  • High-dimensional CLIP embeddings are compressed into low-dimensional latent features on Gaussian primitives before rasterization into feature map L.Directly storing CLIP features over 500 dimensions on millions of primitives would incur unacceptable memory and computation costs.Source: 2) Open-vocabulary Feature Distillation · p. 4
  • Contrastive learning samples pixel pairs within each SAM mask to align latent features, then a trainable decoder recovers sampled features to CLIP space for distillation.The total number of sampled pairs is fixed, with pair counts per mask proportional to mask area.Source: 2) Open-vocabulary Feature Distillation · p. 4

C. Language-guided Robotic Manipulation

The system localizes language-described objects in the reconstructed feature field, then uses rendered geometry for grasping and updates the scene after manipulation. Query relevance scores are thresholded to produce object regions for localization.

C. Language-guided Robotic Manipulation

  • After a language query locates the target, rendered depth and normals provide object geometry for grasp generation and normal-based filtering; observations from fewer viewpoints update the scene after manipulation.The manipulation pipeline uses a point-cloud-based grasping module.Source: C. Language-guided Robotic Manipulation · p. 4

1) Open-vocabulary Querying:

  • For each text query, the method computes relevance using its CLIP embedding and canonical phrase embeddings, including “object,” “things,” “stuff,” and “texture.”This follows the LERF approach for calculating query relevance.Source: 1) Open-vocabulary Querying · p. 4
  • Thresholding the relevance heatmap yields a queried-object mask, which is localized with a bounding box and convex hull.Points below a predetermined relevance threshold are filtered out.Source: 1) Open-vocabulary Querying · p. 4

2) Geometry Restruction:

The method renders depth and surface normals from multiple viewpoints to obtain dense object point clouds and geometry relevant to grasping. Depth is supervised from camera measurements, while Gaussian orientation supplies the rendered normals.

2) Geometry Restruction:

  • Depth and surface normals are rendered from multiple viewpoints to obtain dense point-cloud representations and grasp-relevant surface geometry.Source: 2) Geometry Restruction · p. 4
  • Rendered depth is supervised against the corresponding depth-camera map using an L1 loss over pixels with valid depth.Source: 2) Geometry Restruction · p. 5
  • Each Gaussian’s shortest-axis direction serves as its surface normal because normals are directional vectors requiring rotational equivariance.Rendered normals are geometric properties related to Gaussian orientation; normal maps are expressed in the robot base coordinate and supervised within valid-depth regions.Source: 2) Geometry Restruction · p. 5

3) Feasible Grasp Pose Generation:

The method generates collision-free grasp proposals from the queried object’s point cloud and selects a feasible proposal using rendered surface normals. The final pose is the highest-scoring proposal that passes the force-closure screening threshold.

3) Feasible Grasp Pose Generation:

  • AnyGrasp takes colorful point clouds and proposes collision-free parallel two-finger grasps, each represented by pose dimensions and a grasping score.Source: 3) Feasible Grasp Pose Generation · p. 5
  • The queried-object cloud is combined with a Gaussian-derived scene cloud before AnyGrasp generates proposals restricted to the localized bounding box.Rendered depth from each viewpoint is reprojected to form the object cloud; the scene cloud depicts other objects’ approximate shapes for collision avoidance.Source: 3) Feasible Grasp Pose Generation · p. 5
  • A proposal is feasible when the sum of its two contact-point angles between the grasping line and surface normals is at most θthre; the highest-scoring feasible pose is selected.Source: 3) Feasible Grasp Pose Generation · p. 5

4) Gaussian Field Updating:

GaussianGrasper updates its explicit Gaussian scene after manipulation by transforming the moved object's primitives, then fine-tuning with fewer views. The experiments evaluate the system in desktop scenes and collect RGB-D views to initialize its Gaussian field.

4) Gaussian Field Updating:

  • After an object is manipulated, its Gaussian primitives are transformed together using the robot end-effector transform, then the field is fine-tuned with fewer views and less time.Source: 4) Gaussian Field Updating · p. 5

IV. EXPERIMENT

  • The experiments assess EFD, geometry reconstruction, normal-guided grasping, and scene updates through grasp-update-grasp trials.Source: §IV · p. 5

1) Scenes, Objects and Devices:

  • The system is tested in 10 desktop scenes containing 44 objects, with 40 graspable, across 120 language-guided manipulation trials.The setup uses a UR5 arm with a ROBOTIQ gripper; reconstruction requires approximately 6GB of memory.Source: 1) Scenes, Objects and Devices · p. 5

2) Data Collection and Processing:

  • Sixteen RGB-D viewpoints initialize the scene, with roughly 300k downsampled points used for Gaussian primitives and SAM/CLIP processing producing segmentation and feature maps.Source: 2) Data Collection and Processing · p. 5
  • Fig. 3 compares language-query relevance maps: GaussianGrasper has clearer boundaries than LERF and more consistent multi-view features than SAM + CLIP.For “Roasted chicken wing,” SAM + CLIP also responds to the fork, while GaussianGrasper identifies the chicken wing.Source: Figure 3 · p. 6

B. Results of Efficient Feature Distillation

GaussianGrasper's feature field is evaluated for semantic segmentation, localization, and query speed, alongside qualitative comparisons and geometry reconstruction. The reported results include stronger segmentation and localization than other approaches, faster queries than LERF, and rendered geometry that remains useful where scanned ground truth is invalid.

B. Results of Efficient Feature Distillation

  • Qualitative relevance maps show more accurate features than SAM + CLIP and better segmentation boundaries than LERF, addressing multi-view feature inconsistency.Source: B. Results of Efficient Feature… · p. 6
  • GaussianGrasper significantly outperforms other approaches on segmentation and localization in the reported Table I comparison.Segmentation uses mIoU against manual annotations; localization counts a query as successful when its highest-relevance point lies within the target object.Source: B. Results of Efficient Feature… · p. 6, Table I · p. 6
  • The method achieves an approximate 180 × query-speedup over LERF, while directly distilling CLIP features into the Gaussian field requires over 70 GB of memory.Query time is measured per text query at 640 × 480 resolution.Source: B. Results of Efficient Feature… · p. 6

C. Results of Geometry Reconstruction

  • Rendered depth and surface normals are smoother than scanned ground truth and remain accurate in regions where the scan has invalid values.The example includes a specular silver eyeglass case whose depth the camera cannot capture in that view.Source: C. Results of Geometry Reconstruction · p. 6

D. Effectiveness of Normal-guided Grasp

The normal-guided grasp approach filters infeasible proposals using contact-point surface normals, while the language-guided manipulation results illustrate scene updating after an object is moved. The supplied evidence describes qualitative outcomes and the scope of an efficiency comparison, but gives no values for those tables.

D. Effectiveness of Normal-guided Grasp

  • Rendered depth and surface normals are smoother than scanned values and remain accurate where ground-truth values are invalid.Source: Figure 4 · p. 7
  • The normal-guided grasp example filters the top-ranked proposal because its grasping-line angles conflict with contact-point normals, then executes the feasible second-ranked proposal.Source: D. Effectiveness of Normal-guided Grasp · p. 7, Figure 5 · p. 7

E. Results of Language-guided Manipulation

  • The language-guided manipulation subsection presents scene updating through continuous picking and placing, and Table III is identified as an efficiency comparison between LERF and GaussianGrasper.Source: E. Results of Language-guided… · p. 7, Table III · p. 7
  • Fig. 6 shows an orange moved to a plate, with the previously occluded region restored and the updated orange retaining accurate geometry and semantic features.Source: Figure 6 · p. 7

1) Successful rate of manipulation:

GaussianGrasper outperformed the compared methods in language-guided grasp success-rate testing, and scene updates preserved reported RGB, geometry, and semantic quality. The paper also identifies limits involving unrecorded scene changes and transparent objects.

1) Successful rate of manipulation:

  • The method far exceeded LSeg + AnyGrasp and LERF + AnyGrasp in success rate across 120 trials on 40 objects.Success required stably picking the language-specified object and raising it more than 10 cm for over 3 seconds.Source: 1) Successful rate of manipulation · p. 7

2) Scene updating:

  • After an update using RGB-D observations from five viewpoints, the scene retained high-quality RGB, geometry, and semantic features for another manipulation.The authors also compare update efficiency with LERF using viewpoint count, memory, and reconstruction time, but the supplied passage gives no values.Source: 2) Scene updating · p. 7

V. LIMITATION

  • The reconstructed scene remains static and cannot account for unrecorded changes such as object shifts caused by collisions or vibrations.The method also fails to estimate depth and normals for transparent objects because ground truth is unavailable.Source: §V · pp. 7–8

VI. CONCLUSION

  • The paper presents GaussianGrasper as an RGB-D-based language-guided grasping approach combining feature-field localization, geometry estimation, normal-guided grasping, and scene updates.The authors report that real-world experiments demonstrate the method’s effectiveness.Source: §VI · p. 8

Related papers

Loading 2403.09637v1…