PaperlayerView exact revision on arXiv

Source-linked AI summary

UGG: Unified Generative Grasping

Jiaxin Lu, Hao Kang, Haoxiang Li, Bo Liu, Yiding Yang, Qixing Huang, Gang Hua

arXiv:2311.16917v2cs.CVcs.RO

TL;DR

Dexterous grasping seeks diverse postures with high success, but regression-based and generation-based methods can leave a diversity–success gap. UGG unifies diffusion over object, hand, and contact representations and uses a physics discriminator trained with simulation data. The paper reports state-of-the-art results in grasp success and diversity, and demonstrates object generation from hand poses.

  • Problem

    Regression-based grasp prediction can lack diversity, while generative methods may lack discriminative information for high grasp success.

    Source: §1 · p. 2

  • Method

    UGG jointly models object point clouds, hand parameters, and contact anchors with diffusion, complemented by a physics discriminator trained using simulation data.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    UGG outperforms state-of-the-art results on both grasp success rate and diversity.

    Source: §7 · p. 14

  • Takeaways & Limitations

    The unified design also supports generating objects from hand poses and generating hand and object together.

    Source: §7 · p. 14

    The paper addresses dexterous grasping narrowly as producing hand postures that grasp a given object, with success and diversity defined according to the chosen evaluation criteria.

    Source: §3.1 · pp. 4–5

Abstract

from arXiv · show

Dexterous grasping aims to produce diverse grasping postures with a high grasping success rate. Regression-based methods that directly predict grasping parameters given the object may achieve a high success rate but often lack diversity. Generation-based methods that generate grasping postures conditioned on the object can often produce diverse grasping, but they are insufficient for high grasping success due to lack of discriminative information. To mitigate, we introduce a unified diffusion-based dexterous grasp generation model, dubbed the name UGG, which operates within the object point cloud and hand parameter spaces. Our all-transformer architecture unifies the information from the object, the hand, and the contacts, introducing a novel representation of contact points for improved contact modeling. The flexibility and quality of our model enable the integration of a lightweight discriminator, benefiting from simulated discriminative data, which pushes for a high success rate while preserving high diversity. Beyond grasp generation, our model can also generate objects based on hand information, offering valuable insights into object design and studying how the generative model perceives objects. Our model achieves state-of-the-art dexterous grasping on the large-scale DexGraspNet dataset while facilitating human-centric object design, marking a significant advancement in dexterous grasping research. Our project page is https://jiaxin-lu.github.io/ugg/.

1 Introduction

UGG addresses the tension between grasp diversity and success by combining diffusion-based generation with a physics discriminator. Its unified model also supports object generation from hand information, with potential applications to human-centric object design.

1 Introduction

  • Regression-based methods can lack diversity, while generation-based methods may lack the discriminative information needed for high grasp success.The introduction identifies these as complementary limitations of the two approaches.Source: §1 · p. 2
  • UGG combines diffusion-based grasp generation with a physics discriminator to promote diversity and push for a high grasp success rate.The model operates in latent spaces for 3D object point clouds and hand-joint parameters, and diffuses hand positioning and Contact Anchor.Source: §1 · pp. 2–3
  • UGG can generate objects from hand poses, and qualitative experiments indicate valid solutions for human-centric object design.The generated shapes also offer perspectives on how the model perceives objects.Source: §1 · p. 3
  • A contact-anchor representation replaces the conventional contact map and aligns contact information with point clouds for multimodal generation.The unified transformer architecture brings object, hand, and contact information into one diffusion model.Source: §1 · p. 3

2 Related Work

Prior grasping methods balance diversity and quality, while contact information is often used indirectly. UGG draws on diffusion-based 3D generation and differs from related grasping models by jointly generating objects and hands while modeling contact context.

2 Related Work

  • Data-driven grasp methods often face a diversity–quality trade-off, and contact information is commonly used as an indirect bridge to improve grasp quality.Earlier analytical and simulation-based methods were also described as having limited generalizability or requiring heavy computing.Source: §2 · p. 3
  • Diffusion models have been applied to 3D point clouds, meshes, signed distance functions, and neural fields; UGG adopts LION VAEs for object representation.The paper states that LION's hierarchical latent representation facilitates diffusion-based generation of hand posture and object.Source: §2 · pp. 3–4
  • Unlike SceneDiffuser, UGG models grasp-specific contact context and can simultaneously generate an object and a dexterous hand.The paper also contrasts UGG with gripper-pose generation [72], which it says is not geared for dexterous grasping.Source: §2 · p. 4

3 Problem and Our Approach

The paper defines dexterous grasping narrowly as producing diverse hand postures that successfully grasp a point-cloud object. Its approach jointly models objects, hands, and contacts with diffusion, then uses a simulation-trained classifier to assess candidate grasp success.

3.1 Problem Statement

  • Dexterous grasping is treated narrowly as producing hand postures that successfully grasp a given object, represented as a point cloud.The hand posture comprises joint angles, root rotation, and translation; the adopted ShadowHand model has 22 degrees of freedom.Source: §3.1 · p. 4
  • Success follows Wang et al.: a grasp must withstand at least one of six gravity directions with maximum penetration below 5 mm; diversity is measured by joint-angle entropy.The paper notes that success and diversity definitions can depend on the application.Source: §3.1 · p. 5

3.2 Overview of Our Approach

  • The approach has two parts: a unified diffusion model generating candidate grasps and a lightweight classifier trained on simulation data to predict success.The model treats hands, objects, and their contacts as interacting data manifolds.Source: §3.2 · p. 5
  • Contact Anchor represents contact as a set of 3D points, while one transformer models distributions over object, hand, and contact information.The model uses asynchronous denoising schedules to formulate conditional, marginal, and joint distributions; changing the sampling schedule enables grasp, object, or joint generation.Source: §3.2 · pp. 5–6
  • The physics discriminator uses simulator-derived success labels to select among diverse generated candidates and improve grasp success.The paper describes the classifier as providing discriminative guidance unavailable directly from the generated candidates.Source: §3.2 · p. 6

4 Formulation

The formulation unifies object, hand, and contact representations in a latent diffusion framework for grasp generation. It combines a joint generative model with a Physics Discriminator intended to balance grasp success and diversity.

4 Formulation

  • The framework combines object and hand embeddings, Contact Anchor modeling, unified diffusion, and a Physics Discriminator for diversified grasp generation with high success rate.The discriminator is formulated to improve success without compromising diversity.Source: §4 · p. 6

4.1 Unified Shape Encoding

  • Object point clouds use hierarchical global and local embeddings, while hand representations combine a shape code with rotation and translation.The object embedding is reconstructed as a point cloud; the hand shape code is encoded from joint angles independently of rigid transformations.Source: §4.1 · pp. 6–7

4.2 Unified Contact Modeling

  • Separate object and hand encoders may miss grasp-critical details, and prior contact-map modeling did not achieve joint generation with hand parameters.Subtle object changes can cause grasp failure when the latent representation does not capture the relevant detail.Source: §4.2 · p. 7
  • Contact Anchors represent contact as points sampled from the object cloud within a distance threshold of the hand mesh, avoiding extensive mesh-intersection calculations.The point-set representation is designed to align with shape and integrate into the unified diffusion framework.Source: §4.2 · p. 8

4.3 Unified Generation

  • The U-ViT transformer tokenizes object, hand, and contact representations for a joint noise-prediction network.The object uses global and downsampled local tokens, each hand component has a token, and each contact point has a token.Source: §4.3 · p. 8
  • Asynchronized diffusion timesteps let one model represent conditional generation among objects, hands, and contacts, including grasp generation conditioned on an object.The unified objective predicts noise across the three representations and their timesteps.Source: §4.3 · pp. 8–9

4.4 Physics Discriminator

  • The Physics Discriminator identifies successful generated postures because simulator-based grasp evaluation is not differentiable and existing datasets lack invalid grasps.The generative model supplies adversarial samples, which the simulator labels for discriminator training.Source: §4.4 · p. 9

5 Learning and Inference

Training proceeds from separate object and hand VAEs to unified diffusion, then discriminator optimization. At inference, generated grasp candidates are ranked by the discriminator and refined with test-time adaptation.

5 Learning and Inference

  • The Physics Discriminator uses a cross-entropy loss on object and generated-hand inputs.Its output is compared with the target labels using the two-class softmax expression.Source: §5 · p. 9
  • Training has three stages: individual object and hand VAEs, unified diffusion, then Physics Discriminator optimization on training-set generated grasps.At inference, the model generates M hands, selects the top N by discriminator classification score, and applies test-time adaptation.Source: §5 · p. 9
  • Test-time adaptation minimizes penetration, self-penetration, joint-angle, and contact losses using ADAM for 100 gradient-descent steps.The contact loss sums each Contact Anchor point’s distance to the hand surface.Source: §5 · p. 10

6 Experiment and Evaluation

UGG is evaluated on DexGraspNet for grasp quality and diversity, with additional qualitative tests of contact anchors, object generation, joint generation, and component ablations. The reported results cover seen and novel categories as well as models trained on different dataset subsets.

6.1 Experiment Setup

  • DexGraspNet evaluation uses Isaac Gym and penetration depth, with 4229 training objects and 1126 test objects, including 236 from novel categories.The study also uses 20-object and 10-bottle subsets for results and ablations because full-dataset training is time-consuming.Source: §6.1 · p. 10

6.2 Results of Dexterous Grasping

  • UGG achieves the highest success rate (72.7%), Q1 (0.063), and lowest penetration (0.14), while outperforming DDG, GraspTTA, and UniDexGrasp in mean entropy (7.17) and standard deviation (0.07).The comparison is reported for DexGraspNet grasp generation.Source: §6.2 · p. 10, Table 1 · p. 10
  • UGG generates three distinct grasps per object for seen and novel categories, with examples showing varied mug-handle and headphone grasps.The seen-category examples are outside the training set, and novel-category objects were not encountered during training.Source: §6.2 · p. 11, Figure 4 · p. 11
  • Generated contact anchors identify grasp points on unseen objects; anchors may appear close when the hand-object contact area is small under the strict training threshold.The authors report that this closeness does not negatively affect optimization.Source: §6.2 · p. 11, Figure 5 · p. 11

6.3 Other Generative Results

  • Models trained on the full dataset generate valid objects that often resemble small cubes, while models trained on smaller subsets generate varied objects adapted to the hand pose.The results are presented for full-dataset, 20-object, and 10-bottle training settings.Source: §6.3 · p. 12, Figure 6 · p. 12
  • Joint generation produces diverse object-and-hand poses from two subset-trained models and can provide additional grasping data pairs from a minimal initial dataset.The full-dataset model is omitted because it consistently generates cube-like objects.Source: §6.3 · p. 13, Figure 7 · p. 13

6.4 Ablation Study

  • The physics discriminator shows the expected threshold and pose-count behavior; more than half of generated poses score above 0.9, and over 81% of those are successful.The discriminator ablation evaluates 1000 generated poses.Source: §6.4 · p. 13, Figure 8 · p. 13
  • Adding both hand-joint encoding and Contact Anchor outperforms two baselines in success rate and diversity; hand encoding increases diversity without significantly affecting success rate.The authors attribute the diversity increase to hand-joint encoding acting like data augmentation.Source: §6.4 · p. 14

7 Conclusions

UGG unifies object, hand, and contact generation in a diffusion framework, with a physics discriminator supporting grasp success and diversity. The model also enables object generation from hand information and reports state-of-the-art results on success rate and diversity.

7 Conclusions

  • UGG reports state-of-the-art results in both grasp success rate and diversity, while enabling generation of objects from hand information.Its unified design also supports generating hand and contact information from objects, or generating objects, hands, and contacts jointly.Source: §7 · p. 14

A.1 Introduction of Diffusion Model

  • The diffusion process adds Gaussian noise through a fixed Markov chain, then learns a denoising network to predict that noise.The training objective minimizes the noise-prediction error over samples and timesteps.Source: §A.1 · p. 21
  • Asynchronous object and hand timesteps let the unified diffusion model represent both conditional distributions and their joint distribution.Setting one modality's timestep to T approximates unconditional generation of the other; setting it to 0 gives conditional generation, while equal timesteps model the joint distribution.Source: §A.1 · pp. 21–22

A.2 Generation Algorithms

  • The generation algorithms use a DDPM sampler, while DDIM can also be applied and may yield different performance.The described procedures cover grasping or object generation and joint generation.Source: §A.2 · p. 22

B Implementation

The implementation encodes object shapes and hand parameters for unified generation, then applies test-time optimization and a learned discriminator to refine and rank grasps. Its framework is presented as adaptable to other object modalities and hand models.

B.1 Structure of UGG

  • The UGG architecture and its training and testing pipeline are illustrated in Figure 10.The appendix directs readers to Figure 2 of the main paper for a broader view of the pipeline.Source: §B.1 · p. 22, Figure 10 · p. 24

B.2 Unified Shape Encoding

  • Object shapes use a LION VAE trained on ShapeNet, while hand joint parameters are encoded by a dedicated hand VAE.The hand parameters are normalized before encoding, and object and joint generation results use Shape As Points; the framework allows alternative VAEs for different object modalities and hand models.Source: §B.2 · pp. 22–24

B.3 Test-time Optimization

  • Test-time optimization combines penetration, self-penetration, joint-angle, and contact losses; the pipeline uses object point clouds rather than signed distance functions.The authors contrast this point-cloud representation with DDG's reliance on signed distance functions during training.Source: §B.3 · pp. 25–26

B.4 Physics Discriminator

  • A learned discriminator scores object-hand pairs, and the most confident of the generated grasps are selected for subsequent optimization.It processes object point-cloud tokens and embedded hand parameters through attention layers before predicting confidence.Source: §B.4 · p. 26

B.5 Model Parameters

  • Table 3 lists the model, training, and testing parameters.The supplied caption does not report parameter values.Source: Table 3 · p. 25, §B.5 · p. 26

C.1 Dataset Setup

The evaluation uses DexGraspNet with its provided train/test splits and includes separate human-object interaction datasets for evaluation.

C.1 Dataset Setup

  • DexGraspNet contains 5,355 objects, each with over 200 grasps across five object scales, and the experiments follow its provided train/test splits.Novel object categories are manually labeled from object-name codes, and test objects are evenly distributed across scales.Source: §C.1 · pp. 26–27
  • HO3D evaluation uses all 10 YCB objects, which are absent from DexGraspNet but may overlap with its object categories.The evaluation follows the procedure used in.Source: §C.1 · p. 27
  • GRAB evaluation follows prior work and uses six test objects from a dataset of 51 YCB objects grasped by 10 subjects.These objects are absent from DexGraspNet but may overlap with its categories.Source: §C.1 · p. 27

C.2 Metrics

The evaluation measures grasp quality and diversity across DexGraspNet, HO3D, and GRAB, while noting that the hand models affect diversity metrics. Baseline results are drawn from prior work, and UniDexGrasp is evaluated on grasp synthesis without its reinforcement-learning stage.

C.2 Metrics

  • Quality metrics include simulated success rate, wrench-based Q1, penetration depth and volume, and simulation displacement; diversity uses joint-angle entropy on DexGraspNet and cluster-assignment entropy on HO3D and GRAB.The success-rate criterion requires enduring at least one of six gravity directions with maximal penetration below 0.5 cm.Source: §C.2 · pp. 27–28
  • MANO-based HO3D and GRAB diversity metrics use keypoints, whose parameter count is more than twice ShadowHand's, giving MANO an advantage over the joint-angle-based metric.The authors report that hand-model effects on the quality metrics are minimal.Source: §C.2 · p. 28

C.3 Baseline Methods

  • HO3D and GRAB baseline results come from ContactGen, while DDG and GraspTTA results come from DexGraspNet; UniDexGrasp is evaluated without its reinforcement-learning stage.UniDexGrasp's table-based grasping objective differs from the other methods' holding objective, which the authors note affects interpretation of penetration and simulation success rates.Source: §C.3 · p. 28

C.4 Training and Inference Time

UGG training requires substantial GPU time, while generating 200 grasps and optimizing them take seconds on one GPU.

C.4 Training and Inference Time

  • Experiments use a machine with 8 NVIDIA Tesla A100s, and the listed running times are based on an NVIDIA Tesla A100 GPU.Source: §C.4 · p. 28
  • LION VAE training takes ≈560 GPU hours, hand VAE training ≈1.3 GPU hours, and Physics Discriminator training ≈10 GPU hours.Source: §C.4 · pp. 28–29
  • ≈4000 GPU hours train UGG on the full DexGraspNet for 3000 epochs; the 20-object and 10-bottle subsets require ≈192 and ≈96 GPU hours, respectively.The reported training times are for the full dataset and the two specified subsets.Source: §C.4 · p. 29
  • ≈66 seconds generate 200 grasps on one GPU, and 100 optimization steps take approximately 12 seconds.The 200-grasp inference timing is also described as equivalent to generating 200 objects and 200 grasp-object pairs.Source: §C.4 · p. 29

D.1 Grasp Generation on Human-object Interaction Datasets

Supplementary evaluations report UGG’s performance on HO3D and GRAB without finetuning, compare model modules on DexGraspNet, and examine scale effects and qualitative grasp differences. The paper also identifies hand-conditioned object generation as a possibility for human-centric design.

D.1 Grasp Generation on Human-object Interaction Datasets

  • Without finetuning, UGG achieves lower penetration volume and simulation displacement, and higher entropy than prior methods on HO3D and GRAB.Most objects in these datasets are absent from DexGraspNet training, and the datasets use a different hand model.Source: §D.1 · p. 29

D.2 Detailed Break-down on Model Performance

  • UGG performs best among the compared generative models, and adding the Physics Discriminator consistently improves performance across variants.The comparison sequentially adds generation, test-time adaptation, and the Physics Discriminator; contact-informed GraspTTA and UGG benefit more from test-time adaptation.Source: §D.2 · pp. 29–30, Table 5 · p. 29

D.3 Grasp Generation of Different Scales

  • The success rate for 0.15-scale objects is half that for 0.06-scale objects, alongside a dataset distribution that underrepresents larger scales.The 0.15-scale portion is half the 0.08-scale portion; the authors note that the full-dataset model tends to generate small, cube-like objects.Source: §D.3 · pp. 30–31, Figure 13 · p. 30

D.4 Qualitative Comparison

  • UGG’s grasps show more precise visual contact control; UniDexGrasp grasps sit farther from objects, while some GraspTTA examples penetrate them.The passage attributes GraspTTA penetration to limitations of its initial CVAE generation.Source: §D.4 · p. 31, Figure 14 · p. 31

E Potential Impact

  • The authors identify generating objects from hand gestures as a possibility for human-centric object design.Source: E Potential Impact · p. 31

Related papers

Loading 2311.16917v2…