PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Diverse and Physically Feasible Dexterous Grasps with Generative Model and Bilevel Optimization

Albert Wu, Michelle Guo, C. Karen Liu

arXiv:2207.00195v2cs.RO

TL;DR

Dexterous grasp planning must handle diverse possible finger placements while respecting nonconvex physical constraints. The paper combines a CVAE that predicts finger placements from object point clouds with bilevel optimization that refines them into feasible grasps. The method achieved 86.7% success over 120 grasp trials on 20 objects, with full-object point-cloud observation and a flat, uncluttered setting among its limitations.

  • Problem

    Dexterous grasp planning must accommodate multimodal finger placements and nonconvex physical constraints, while practical approaches often relax constraints without guaranteeing exact satisfaction.

    Source: §1 · p. 1

  • Method

    A CVAE predicts finger placements from object point clouds, and bilevel optimization refines them into grasps satisfying physical constraints.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    86.7% overall success over 120 grasp trials on 20 objects, exceeding the hardcoded baseline’s 53.3% overall success rate.

    Source: §4.4 · p. 7

  • Takeaways & Limitations

    The pipeline combines learned grasp generation with bilevel refinement and rejection sampling to remove poor predictions and produce diverse, physically feasible grasps.

    Source: §5 · p. 8

    The method requires a full object point cloud and assumes a flat surface with no other objects in the scene.

    Source: §5 · p. 8

Abstract

from arXiv · show

To fully utilize the versatility of a multi-fingered dexterous robotic hand for executing diverse object grasps, one must consider the rich physical constraints introduced by hand-object interaction and object geometry. We propose an integrative approach of combining a generative model and a bilevel optimization (BO) to plan diverse grasp configurations on novel objects. First, a conditional variational autoencoder trained on merely six YCB objects predicts the finger placement directly from the object point cloud. The prediction is then used to seed a nonconvex BO that solves for a grasp configuration under collision, reachability, wrench closure, and friction constraints. Our method achieved an 86.7% success over 120 real world grasping trials on 20 household objects, including unseen and challenging geometries. Through quantitative empirical evaluations, we confirm that grasp configurations produced by our pipeline are indeed guaranteed to satisfy kinematic and dynamic constraints. A video summary of our results is available at youtu.be/9DTrImbN99I.

1 Introduction

Diverse dexterous grasps can support flexible interaction across objects and higher-level tasks, but planning them must address both multimodal finger placements and nonconvex physical constraints. The introduction proposes combining learning with optimization to generate diverse, physically feasible grasps for unseen objects.

1 Introduction

  • Fully actuated multi-finger dexterous hands can produce diverse grasp configurations, unlike grippers whose fingertip workspaces are restricted to a subspace of 3D task space.Diverse configurations can support flexible object interaction for higher-level tasks.Source: §1 · p. 1
  • Dexterous grasp planning combines a multimodal solution space with nonconvex constraints on collision, contact, and force closure.Many finger placements may be valid, without a metric that identifies one optimal grasp.Source: §1 · p. 1
  • Previous approaches often relax physical constraints into scalar losses because direct dexterous grasp optimization is considered too nonconvex in practice.This trades exact constraint-satisfaction guarantees for practicality.Source: §1 · pp. 1–2
  • The proposed pipeline predicts initial contact points with a CVAE, then uses bilevel optimization to project them onto kinematically and dynamically feasible grasps.The method targets diverse grasps for unseen objects.Source: §1 · p. 2

2 Related Work

Related work contrasts learning-based grasp generation with analytical methods that encode physical constraints, and notes limitations in diversity, physical guarantees, and hardware validation. It also situates bilevel optimization as relatively new in motion planning and, to the authors’ knowledge, its first use for dexterous robotic manipulation.

2.1 Dexterous Grasping

  • Learning-based dexterous grasping includes generative and convolutional models, but many works omit physical laws or have restricted hardware evaluation.Some simulation-focused work targets photorealistic human grasps rather than strict real-world physical constraints.Source: §2.1 · p. 2
  • Analytical dexterous planners model constraints such as force closure and friction, but can depend on high-fidelity object models and lack grasp diversity.The paper limits its related-work discussion to uncluttered scenes and emphasizes dexterous grasping.Source: §2.1 · p. 2, §2 · p. 2
  • Grasp-refinement methods often make optimization tractable by summing constraint violations as scalar losses, without guaranteeing that the output satisfies the original constraints.This motivates the paper’s distinction between relaxed losses and direct constraint satisfaction.Source: §2.1 · pp. 2–3

2.2 Simple Grasping

  • Simple grasp planning can reduce parallel-jaw gripper planning to computing a 6-DoF pose, a formulation that seldom scales directly to dexterous grasp planning.The latter is described as significantly more complex.Source: §2.2 · p. 3

2.3 Bilevel Optimization (BO) for Planning

  • Bilevel optimization has been applied to several motion-planning problems, but the authors know of no prior application to dexterous robotic manipulation.They characterize BO’s application to motion planning as relatively new.Source: §2.3 · p. 3

3 Method

The method predicts three fingertip placements from an object point cloud, converts them to an initial grasp configuration, and refines that guess with bilevel optimization. It defines physical feasibility through dynamic and kinematic constraints and trains its placement predictor on synthesized feasible grasps.

3 Method

  • At inference, a CVAE predicts finger placements from an object point cloud, those placements seed an initial grasp configuration, and bilevel optimization outputs a physically feasible grasp.The pipeline assumes exactly three grasping fingers.Source: Figure 2 · p. 3, §3 · p. 3

3.1 Dynamic Constraints: Wrench Closure and Friction Cone

  • Dynamic feasibility requires wrench closure, with contact forces summing to zero in both force and moment, and contact forces lying within friction cones.The friction cone is approximated by polyhedral constraints; a quadratic program tests dynamic feasibility, with a positive normal-force lower bound to avoid the trivial zero-force solution.Source: §3.1 · pp. 3–4

3.2 Kinematic Constraints: Reachability and Collision

  • Kinematic feasibility requires each fingertip to reach its object-surface contact point while robot links avoid collisions with the object and environment.Both reachability and collision constraints are nonconvex.Source: §3.2 · p. 4

3.3 Learning to Predict Finger Placement from a Physically Feasible Grasp Dataset

  • The learning component predicts potential finger placements from an object point cloud, using a synthesized dataset of physically feasible grasps for the Allegro hand.The authors cite the lack of large-scale dexterous-hand datasets and challenges in retargeting human grasps to robot hands as motivation.Source: §3.3 · p. 4

3.3.1 Creating a Physically Feasible Dexterous Grasp Dataset

  • The dataset is built by sampling object rest poses on a flat surface, enumerating finger placements, and retaining placements for which a feasible grasp is found.It is generated in simulation for six YCB objects.Source: §3.3.1 · p. 4

3.3.2 Training a Conditional Variational Autoencoder (CVAE)

  • A PointNet++-based CVAE learns to predict feasible contact points from point clouds using L1 reconstruction and KL-divergence losses.At inference, a latent sample from a standard normal distribution and the point cloud are passed to the decoder to produce the placement prediction.Source: §3.3.2 · pp. 4–5

3.4 Computing Grasps with Physical Feasibility Guarantees using Bilevel Optimization (BO)

The method projects predicted finger contacts onto the object surface and uses them to initialize a bilevel optimization that seeks a physically feasible grasp. A lower-level force-closure problem handles dynamic feasibility, while a nonconvex optimizer refines the hand configuration and may reject poor initial predictions.

3.4 Computing Grasps with Physical Feasibility Guarantees using Bilevel Optimization (BO)

  • The predicted finger placements are projected onto the object surface, then an inverse-kinematics solution provides the initial guess for bilevel optimization.The IK solution matches each finger to its projected contact within a numerical tolerance and enforces a finger-object distance constraint.Source: §3.4 · p. 5

3.4.1 Formulating the Bilevel Optimization Grasp Refinement

  • Bilevel optimization moves dynamic-feasibility computation into a lower-level force QP; a valid solution satisfies force closure, surface-contact, and collision constraints.The paper states that a valid solution to Equation (6) defines a physically feasible grasp.Source: §3.4.1 · p. 5

3.4.2 Solving the Bilevel Optimization Grasp Refinement

  • The nonconvex optimizer uses constraint gradients and may return a locally optimal grasp near the IK seed; infeasibility can prompt rejection and resampling of the CVAE prediction.The gradient of the lower-level objective is obtained using differentiable optimization.Source: §3.4.2 · pp. 5–6

4 Experiments

Experiments evaluate the pipeline on hardware, compare it with baseline and ablated approaches, and measure grasp success and physical-constraint satisfaction. Results include diverse grasps on novel objects and a strong association between wrench closure and hardware success.

4.1 Hardware Setup

  • Hardware evaluation used a 16-DoF Allegro hand mounted on a 7-DoF Franka arm and tested 20 rigid household objects.The objects comprised 3 seen, 4 familiar, and 13 novel objects.Source: §4.1 · p. 6

4.4 Grasp Trial Results

  • 86.7% overall success across 120 trials on 20 objects exceeded the hardcoded baseline’s 53.3%.The results are summarized in Table 1.Source: §4.4 · p. 7, Table 1 · p. 7
  • The method grasped challenging novel objects, while the hardcoded baseline nearly always succeeded only on similarly sized objects permitting top-down grasps.The baseline failed when objects required a different grasp type or differed significantly in size.Source: §4.4 · p. 7
  • Different latent-variable samples generated distinct successful grasps on the upright sugar box from the same initial object rest pose.Figure 4 shows 10 distinct successful grasps.Source: §4.4 · p. 7, Figure 4 · p. 7
  • Median grasp-generation time was 14.4 seconds, with a median of two CVAE samples, two IK solves, and two BO runs.The repeats refer respectively to samples for z, IK solves for q′, and bilevel optimizations for q∗.Source: §4.4 · p. 7

4.5 Quantitative evaluation of physical constraint enforcement

  • The method’s maximum finger-object distance violation was 0.08 cm, while CVAE-only reached 2.032 cm.The 0.08 cm violation was described as negligible compared with other error sources, such as camera observation error.Source: §4.5 · p. 8, Table 2 · p. 8
  • Median force and torque ratios were each ≤0.01% for the method, indicating effective enforcement of wrench closure.CVAE+kinematics, which does not explicitly consider external wrench, also produced many grasps achieving wrench closure by coincidence.Source: §4.5 · p. 8
  • Hardware success strongly correlated with wrench closure, and every grasp BO reported infeasible failed on hardware.The comparison examined CVAE+kinematic trials on ramen and mustard bottle, as well as plans reported infeasible by BO.Source: §4.5 · p. 8, Table 3 · p. 8

5 Limitations and Conclusion

The pipeline combines CVAE grasp generation with bilevel optimization and rejection sampling, achieving an 86.7% success rate on 20 challenging real-world objects. Its use is bounded by full-point-cloud and scene assumptions, and it cannot specify grasp type.

5 Limitations and Conclusion

  • The method requires a full object point cloud, cannot select grasp type, and assumes a flat surface with no other objects; vision errors are a major failure source.The physical-constraint formulation lacks explicit robustness margins.Source: §5 · p. 8
  • 86.7% grasp success was achieved on 20 challenging real-world objects.The conclusion attributes output quality to combining learned models with physics-inspired metrics.Source: §5 · p. 8
  • Bilevel refinement and rejection sampling remove poor CVAE grasp predictions that may violate physical constraints.The pipeline combines learning-based grasp generation with bilevel optimization to produce diverse, physically feasible grasps.Source: §5 · p. 8

Supplementary Material for “Learning Diverse and Physically Feasible Dexterous

The supplied supplementary-material passages identify the paper, its authors and affiliations, and its arXiv version date.

Supplementary Material

1 Method

The method approximates friction constraints with a polyhedral cone, bounds fingertip-to-object distance, and constructs a feasible grasp dataset from simulated object poses and contacts.

1.1 Polyhedral Approximation of the Friction Cone

  • Friction constraints use a polyhedral approximation formed from two orthogonal tangent directions at each contact.The approximation constrains normal and tangential force components.Source: §1.1 · p. 12

1.2 Enforcing the Finger-Object Distance Constraint

  • Fingertip-object distance is constrained by signed distance, with bounds of −0.68 cm and −0.32 cm chosen for compliant Allegro fingertips.The bounds reflect the thickness of the compliant material.Source: §1.2 · p. 12

1.3 Creating a Physically Feasible Dexterous Grasping Dataset

  • The dataset uses six YCB objects dropped onto a flat surface to generate object rest poses and sampled point clouds with surface-contact points removed.For each rest pose, 256 candidate contact points are generated using Poisson disk sampling.Source: §1.3 · pp. 12–13
  • Dynamically feasible three-finger placements are checked across all six finger assignments for kinematic feasibility, then augmented by random translation and yaw during training.Feasible object-cloud, grasp-pose, and finger-placement tuples are added to the dataset.Source: §1.3 · p. 13

2 Software

The software uses automatic-differentiation tools for bilevel optimization, and the CVAE training process selects a model checkpoint based on test loss.

2.1 Implementation Details

  • The CVAE was implemented in PyTorch and trained on a 16-GPU NVIDIA Tesla A100 cloud instance for approximately 76 hours.Source: §2.1 · p. 13
  • The bilevel optimization combines automatic-differentiation tools from Drake, OptNet, and PyTorch; the source code is available on GitHub.Source: §2.1 · p. 13

2.2 CVAE Training Process

  • The model checkpoint after 1,550 epochs was selected based on test losses, and fingertip-placement reconstruction accuracy was sub-centimeter.Source: Figure 1 · p. 13, §2.2 · p. 13

3 Hardware

The hardware section covers robot control and scene reconstruction, then specifies CVAE trial sampling, object-specific grasp challenges, and a hardcoded baseline.

3.1 Robot Control

  • The Franka arm is controlled with Polymetis, and the Allegro hand uses an open-source driver.Source: §3.1 · p. 14

3.2 Scene observation

  • Four calibrated cameras provide depth images that are filtered into an object point cloud and converted to a convex-hull mesh.Tabletop points are removed by plane fitting, and points outside a predefined bounding box or identified as outliers are filtered.Source: §3.2 · p. 14

3.3 CVAE-based methods

  • Each CVAE trial uses a unique random seed and samples latent variables until the first simulation-successful grasp, with six IK orientation initializations per latent.The selected grasp is the one executed and evaluated on hardware.Source: §3.3 · p. 14

3.4 Challenges Stemming from Selected Object Shapes

  • Selected shapes pose distinct grasp challenges: the triangular sandwich box, irregular massage ball, cone-topped castle, slender objects, and tetrahedron.Side grasps benefit the castle and slender objects; the tetrahedron is difficult to force-close and requires large squeezing force.Source: §3.4 · p. 14

3.5 Hardcoded Baseline Design

  • The hardcoded baseline manually sets finger joints, places the wrist using point-cloud means and maxima plus tuned offsets, and aligns wrist orientation with the short PCA axis.The wrist translation uses mean x and y coordinates and the maximum z coordinate.Source: §3.5 · p. 14

4 Results

Results are reported as grasp statistics across test-object categories, physical-constraint evaluations, and per-stage timing statistics.

4.1 Grasp Trial Statistics

4.2 Quantitative Evaluation of Physical Constraints

  • Finger-object distances are evaluated against [−0.68, −0.32], while wrench-closure force and torque ratios are expected to be zero.The distance measure is signed distance to fitted object meshes; Table 5 reports force and torque ratios as medians and interquartile percentiles.Source: §4.2 · p. 15, Table 4 · p. 15, Table 5 · p. 15

4.3 Timing Statistics

Related papers

Loading 2207.00195v2…