PaperlayerView exact revision on arXiv

Source-linked AI summary

GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation

Julien Merand, Boris Meden, Mathieu Grossard, Liming Chen

arXiv:2608.19759v1cs.ROcs.AI

TL;DR

Dexterous grasp planners often generalize poorly because they depend on limited, object-specific grasp data. GOAG learns gripper-intrinsic contact distributions without object geometry during training, then uses object features at inference to generate grasps. It reports competitive state-of-the-art performance, including an 86.93% average success rate on MultiDex, with faster processing when generating numerous grasps.

  • Problem

    Existing data-driven grasp planners commonly rely on restricted object sets or object-specific grasp databases, limiting generalization to novel shapes.

  • Method

    GOAG trains a CVAE-based gripper-centric model on feasible contact distributions generated from gripper geometry, kinematics, and taxonomy, introducing object features only at inference.

  • Results

    GOAG achieves competitive state-of-the-art performance across multiple benchmarks and validates generalization through successful real-robot deployment.

  • Takeaways & Limitations

    GOAG provides an object-agnostic grasp-planning formulation that avoids large-scale object-specific grasp databases while retaining competitive performance.

Abstract

from arXiv · show

Multifingered grasping is a crucial robotic skill, but current deep-learning grasp planners often struggle to generalize to new objects because they are trained on limited, object-specific datasets. We introduce a fundamentally different approach, grounded in the observation that the gripper and the object share identical surface geometry at their mutual contact points. We propose GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation, a novel deep generative model that learns a compact latent representation of a specific gripper's contact surface distribution, enabling the efficient sampling of valid grasp configurations without relying on object-specific training data. We show that by introducing object features only at inference time, our model can effectively retrieve admissible contact areas that are compatible with the gripper's capabilities. We validate our approach through extensive experiments on established grasp protocols in both simulated and real-world scenarios, demonstrating its effectiveness with different grippers from the literature. Our method delivers state-of-the-art results on the objects from the MultiDex dataset, achieving an average success rate of 86.93%. It offers significantly faster processing when generating numerous grasps, while matching the performance of leading approaches specifically trained on this dataset. Unlike these methods, our approach does not rely on object-specific training data, highlighting the advantages of object-agnostic learning. It effectively addresses the generalization challenges faced by traditional data-driven grasp planners. Code and videos are available on our project website https://cea-list.github.io/goagweb/ .

I. INTRODUCTION

GOAG addresses limited generalization in dexterous grasp planning by shifting from object-specific training to a gripper-centric, object-agnostic paradigm. It learns feasible contact distributions from gripper geometry and kinematics, then introduces object features only during inference.

  • Dexterous grasp planners must handle complex multi-fingered hardware in real time while generalizing beyond restricted object sets.Traditional analytical methods face time constraints, while data-driven methods often evaluate on limited object collections.
  • GOAG trains exclusively on gripper geometry and kinematics rather than object-specific grasp databases.The approach shifts from grasp-centric or object-centric learning to a gripper-centric perspective.
  • GOAG learns feasible contact-area distributions from taxonomy-labeled, randomly sampled kinematic configurations.This synthetic training procedure maps grasp taxonomy labels to valid gripper configurations and contact regions.
  • Object features are introduced only at inference, enabling zero-shot generalization to arbitrary object shapes through a CVAE and BPS encoding.The model retrieves object contact areas compatible with gripper kinematics using the same BPS representation during inference.
  • GOAG combines object-agnostic learning with experimental validation on established protocols and novel objects.The reported validation includes simulated and real-world environments and comparisons with existing methods.

II. RELATED WORK

Existing dexterous grasp planners use data-driven mappings, synthetic grasp databases, or direct pose prediction, but these approaches face dataset dependence and physical-validity challenges. GOAG instead samples feasible contact distributions from gripper geometry and kinematics.

  • Modern data-driven grasp planners are summarized as a major recent direction in dexterous grasp planning.
  • Large-scale grasp databases are costly and typically map specific grippers to specific object geometries.Their construction can rely on demonstrations, motion capture, teleoperation, thermal imaging, or synthetic physical validation.
  • Direct-prediction methods map object representations to complete grasp configurations using regression, generative, diffusion, or foundation models.
  • GOAG adapts grasp taxonomy to gripper geometry and samples contacts within admissible regions determined by gripper kinematics.The sampling procedure is independent of object geometry.
  • Directly generated grasp poses may violate physical constraints and therefore require expensive validation or learned filtering.

C. Learning Intermediate Representation

GOAG estimates object grasps through an intermediate contact representation defined by the gripper’s intrinsic capabilities. It uses gripper-centric contact regions and subsequently maps them to executable configurations.

  • GOAG estimates numerous grasps for a specific object while training only on gripper geometry and using object geometry during inference.
  • GOAG samples gripper configurations, generates contact points, encodes them with BPS, and predicts contact distributions and link assignments.The architecture uses a CVAE for contact distributions and PointNet++ for gripper-link association.
  • The method defines contact regions as intrinsic properties of the gripper rather than directly as object-surface targets.
  • The gripper handprint represents surface points on active grasping areas, including the palm and inward-facing link surfaces.

A. Shifting to a Gripper-Oriented Paradigm

GOAG recasts contact detection in the gripper’s canonical frame by transforming the object with the inverse gripper pose. This makes feasible contact patterns learnable from gripper geometry and kinematic constraints before querying object shapes.

  • Object-surface contacts are defined by an aligned-distance criterion between object points and the positioned gripper handprint.
  • Applying the inverse gripper pose transforms the object into the gripper’s canonical frame for gripper-oriented contact analysis.
  • GOAG defines contact points as a subset of the gripper surface rather than the object surface.
  • The approach learns intrinsic feasible-contact priors solely from gripper kinematics and grasp taxonomy, then queries object shapes at inference.

B. Training GOAG from C(H(Q))

GOAG constructs object-agnostic training data from valid gripper configurations and taxonomy-constrained contact points, then learns their feasible contact distribution and link assignments.

  • GOAG samples valid joint configurations and grasp-taxonomy regions to generate structurally valid contact maps on the gripper.The taxonomy constrains contact points to admissible regions, providing a kinematic prior for training.
  • The projected contact field assigns each workspace basis point a continuous proximity-and-alignment value relative to active contact points.Values approach 1 for close, aligned points and 0 for distant or misaligned points.
  • A CVAE learns the distribution of feasible contacts from BPS representations and contact labels using reconstruction and KL-divergence losses.The reconstruction loss emphasizes high-likelihood contact regions through an attention weighting.
  • A parallel PointNet++ links mapper associates sampled contact points with gripper phalanx links for downstream joint-configuration recovery.This assignment gives spatial contact points the kinematic meaning needed for optimization.

C. GOAG Inference on O([R, T]−1)

At inference, GOAG predicts object-compatible contact zones, filters unstable samples with force-closure estimation, and optimizes the gripper while enforcing geometric and hardware constraints.

  • GOAG transforms an unseen object into the gripper’s canonical workspace and decodes contact likelihoods from its BPS-aligned distance field.High-likelihood basis points above threshold τ form the discrete contact set used downstream.
  • The links mapper assigns predicted contacts to gripper phalanges before force-closure estimation and full kinematic optimization refine execution.The latent prior can produce geometrically plausible contacts without guaranteeing physical stability, motivating these later stages.
  • If force closure fails, GOAG discards the prediction and resamples a latent vector to generate a new contact distribution.This resampling avoids proceeding with an unstable predicted grasp.
  • Joint optimization minimizes contact distance while penalizing object penetration, self-penetration, and violations of joint limits.These terms respectively target link-contact alignment, object collision avoidance, self-collision avoidance, and hardware-limit compliance.

A. Implementation Details

The implementation uses dense synthetic gripper sampling, workspace discretization, and separate training schedules for the CVAE and links mapper; Table II evaluates three grippers with three metrics.

  • 10,000 kinematically valid gripper configurations are sampled to cover the reachable workspace and produce active surface point clouds.Active points are selected from contact pads and nearby inward-facing finger sides using a gripper-specific normal threshold.
  • Table II reports success rate, efficiency, and diversity for the Barrett, Allegro, and Shadow Hand grippers on the Multidex test set.The table is framed as an in-depth performance analysis across three classic dexterous grippers.
  • The workspace uses M = 8,192 points, with γ = 2.0 and τ = 0.8; the CVAE and links mapper train for 100 and 50 epochs, respectively.The CVAE uses latent dimension ψ = 128, β = 0.01, and attention factor α = 3.0.

B. Grasp Pose Constraints and Generation

GOAG can impose object pose constraints and generates diverse candidate poses by sampling around a dilated convex hull, while grasp success is physically tested under external disturbances.

  • Grasp Pose Constraints and Generation: GOAG supports imposing the object pose [R, T], avoiding grasp poses that are physically infeasible under task and environmental constraints.A single pose inference can cover many valid configurations for objects with translational or rotational symmetries.
  • Grasp Pose Constraints and Generation: For objects without substantial symmetry, GOAG samples gripper poses uniformly on a convex hull dilated by 110%.The palm points opposite the hull normal to produce diverse, well-distributed surface candidates.
  • Evaluation Metrics: Grasp success is measured in Isaac Gym by applying sequential ±xyz external forces for one second each and requiring less than 2 cm deviation.This physical simulation validates final grasps beyond theoretical force-closure estimates.
  • Evaluation Metrics: Figure 4 visualizes GOAG grasps on Multidex objects for Barrett, Allegro, and Shadow Hand grippers using green, pink, and purple, respectively.
  • Evaluation Metrics: Efficiency is the average time per grasp when generating 100 grasps, including inference and optimization but excluding Isaac Gym simulation.

D. Experimental Protocols

The evaluation compares GOAG with established baselines across multiple grippers and grasp datasets, emphasizing accuracy, efficiency, and object-agnostic generalization. Experiments also examine diverse grasp synthesis and real-world visualization.

  • Multidex evaluation: GOAG was evaluated against baselines on the Multidex test set using Barrett, Allegro, and Shadow hands.The protocol used 100 inference runs per method with batch size 10; the dataset contains 10 ContactDB and YCB objects.
  • Cross-dataset generalization: The five-benchmark evaluation covered 3438 objects from three simulated datasets, one real-world dataset, and one retargeted human-hand dataset.The evaluation protocol followed the previously described procedure.
  • Multidex evaluation: GOAG achieved higher average grasp success across all three grippers while remaining object-agnostic during training.Its vectorized optimization processes all grasp candidates through one energy function, improving efficiency when generating many grasps.
  • Grasp diversity: GOAG can generate multiple grasp types for one object pose by sampling its CVAE latent space.This provides a mechanism for diverse grasp synthesis.
  • Cross-dataset generalization: GOAG achieved the second-highest average success rate across five grasp benchmarks after one Shadow Hand training run.Competing methods were retrained for each dataset, whereas GOAG used object-agnostic training only on Shadow Hand kinematics.
  • Real-world visualization: Figure 5 presents real Allegro-hand grasps in the first row and corresponding virtual grasps in the second row for YCB objects.Objects are rotated around the z-axis to clarify the grasp poses.

F. Real-Robot Experiments

The real-robot experiment used an Allegro Left Hand on a 7-DoF arm to test transfer to physical YCB objects. The system successfully grasped 11 objects, with results illustrated in Figure 5.

  • Hardware and objects: An Allegro Left Hand mounted on a 7-DoF robot arm successfully grasped 11 YCB objects.The experiments demonstrate transfer of the method to real-world objects.
  • Validation: Figure 5 illustrates the successful grasps and shows the corresponding experimental setup.Videos of the experiments are provided in the supplementary material.
  • Validation: The real-robot deployment validates GOAG’s generalization beyond object-specific training data.The paper reports competitive performance across multiple benchmarks without training on their specific datasets, alongside successful real-robot deployment.
Loading 2608.19759v1…