OmniDexVLG: Learning Dexterous Grasp Generation from Vision Language Model-Guided Grasp Semantics, Taxonomy and Functional Affordance

I. INTRODUCTION

• Existing approaches emphasize functional grasping but often omit explicit grasp-taxonomy and contact-semantic control, while datasets can be pinch-dominated and semantically coarse.The introduction also identifies a lack of unified reasoning across taxonomy, contact semantics, and functional affordance.Source: §I · pp. 1–2
• OmniDexDataGen integrates functional affordance cues, contact-pattern priors, and finger-configuration semantics to produce physically stable, semantically rich, structurally diverse grasp samples.Its dataset-generation contribution targets greater diversity in grasp types, contact paradigms, and functional affordances.Source: §I · p. 2
• OmniDexReasoner models affordance, contact semantics, and grasp taxonomy using large multimodal models with multi-agent collaboration, retrieval-augmented generation, and chain-of-thought reasoning.The authors present these mechanisms as improving comprehension of complex grasping behaviors.Source: §I · p. 3
• OmniDexGraspNet combines language instructions and visual point clouds to generate grasps guided by functional intent, grasp taxonomy, and contact configuration.The proposed model is described as producing diverse grasps with semantic and structural variability.Source: §I · pp. 2–3

A. Semantic-Aware Dexterous Robotic Grasp Generation

• Semantic-aware grasp generators add language guidance or contact and grasp-type encodings, yet a unified framework spanning grasp type, contact structure, functional intent, and finger configuration remains lacking.Two-finger grippers often use object-level affordances, whereas dexterous hands require finer semantic modeling.Source: A. Semantic-Aware Dexterous Robotic… · p. 3

B. Dexterous Grasp Taxonomy and Reasoning

• Multimodal models support grasp-type prediction, but similar grasp categories can trigger semantic hallucinations, leaving physically grounded, task-aware taxonomy reasoning an open gap.The surveyed work notes that structured grasp knowledge must be integrated with multimodal embeddings for more controllable and diverse generation.Source: B. Dexterous Grasp Taxonomy and… · p. 3

C. Dexterous Hand Pose Data Generation

• Optimization-based datasets favor stable grasps, but DexGraspNet is described as pinch-biased, while other diverse-grasp methods rely on manually predefined category heuristics.Existing multimodal datasets also retain sparse, coarse annotations, often limited to object parts or high-level functions.Source: C. Dexterous Hand Pose Data Generation · pp. 3–4

A. Problem Statement and Method Overview

• The method overview combines semantic dataset generation, large-multimodal-model reasoning about object–grasp relationships, and 3D vision-language generation of dexterous grasp poses.The stated data gap is a scarcity of rich, multidimensional semantic annotations.Source: A. Problem Statement and Method Overview · p. 4

terous Grasp Dataset Generation Enriched by Grasp Taxonomic Diversity

• OmniDexDataGen samples taxonomy-based configurations and affordance-aware contacts, applies taxonomy-aware differential force-closure sampling, then optimizes and physically evaluates the grasps.The pipeline targets functional, contact-semantic, and grasp-taxonomy diversity.Source: terous Grasp Dataset Generation… · pp. 4–5

1) Grasp Taxonomy-Aware Configuration Sampling:

• Grasp configuration sampling represents active fingers, contacted links and regions, force direction, and grasp center, conditioned on a specified grasp type.The configuration is denoted G = {Sf, Sl, Sc, v, c}.Source: 1) Grasp Taxonomy-Aware Configuration… · p. 5

2) Functional Affordance-Aware and Grasp Type-Aware Differential Force-Closure Grasp Sampling:

• Tax-DFCSampler generates initial grasp poses and AC-Sampler selects affordance-aware contact candidates, which are force-closure filtered before pose alignment and collision screening.Random permutations increase pose diversity, and candidates with collision counts below a predefined threshold are retained.Source: 2) Functional Affordance-Aware and… · p. 5

3) Grasp Optimization and Simulation Validation:

• Optimization updates only activated fingers and the wrist, keeping non-activated fingers fixed while combining semantic interaction, DFC, and penetration/joint-limit losses.Contact alignment uses object-surface targets and SDF to encourage valid contact while minimizing interpenetration; DFC promotes force closure, and joint updates respect anatomical limits.Source: 3) Grasp Optimization and Simulation… · pp. 5–6
• The sampling algorithm builds initial poses from valid DFC contacts, randomizes contacts and idle fingers, and rejects poses exceeding a collision threshold.It estimates a grasp center and applies the grasp configuration's force direction when constructing each pose.Source: 3) Grasp Optimization and Simulation… · p. 6
• Refined candidates undergo physics simulation with joint impedance control, and only grasps completing without slippage or failure enter the final dataset.The control torque uses current and target joint positions, joint velocity, stiffness, damping, and external compensation terms.Source: 3) Grasp Optimization and Simulation… · p. 6

C. LMM-Grounded Dexterous Grasping Multi-Dimensional Semantics Understanding Method

• OmniDexReasoner combines multi-agent collaboration, CoT, and RAG across three modules to create expressive semantic annotations and priors for downstream multimodal models.The modules address hand–object interaction understanding, grasp taxonomy reasoning, and multidimensional semantic description generation.Source: C. LMM-Grounded Dexterous Grasping… · p. 6

1) Hand–Object Interaction Understanding Model:

• The interaction model maps hand, object, contact, and multi-view scene inputs to affordance classifications, descriptions, contact fingers and links, finger flexion, palm-contact status, and a contact semantic map.Hand inputs include available fingers, links, joint configurations, and maximum graspable size; object inputs include name and affordance annotations.Source: 1) Hand–Object Interaction… · p. 6
• The Contact Semantic Map labels object-surface points with the contacting hand link and finger, based on their spatial distance.The map supports identifying involved fingers and links and whether the palm contacts the object; palm contact can indicate power versus precision grasp, while force requires visual estimation.Source: 1) Hand–Object Interaction… · pp. 6–7
• Functional affordance is classified by voting over the affordance labels associated with contacted object points.The model also incorporates multi-view scene images, which provide visual information about physical contact and finger pose.Source: 1) Hand–Object Interaction… · p. 7

2) Grasp Taxonomy Understanding with Multimodal Reasoning:

• Grasp type depends on contact, force direction, finger configuration, affordance, and task context, while LMMs can hallucinate when interpreting physical interactions.The module reasons over hand configuration, object properties, contact, affordance cues, and multi-view scene images.Source: 2) Grasp Taxonomy Understanding with… · pp. 7–8
• Taxonomy classification proceeds hierarchically, first assigning Power, Precision, or Intermediate, then a specific subtype such as pinch, tripod, or hook.The same object may support different grasp types depending on human preferences or task goals.Source: 2) Grasp Taxonomy Understanding with… · pp. 7–8
• RAG supplies grasp-type descriptions and discrimination rules, while CoT structures inference from scene to affordance, contact type, and taxonomy prediction.The retrieved database includes semantic knowledge about fingers, links, contacts, affordances, and use cases.Source: 2) Grasp Taxonomy Understanding with… · p. 8

3) Dexterous Grasp Multi-Dimensional Semantic Description Generation Model:

• The Dexterous Grasp Descriptor Generator jointly reasons over functional affordance, hand–object contact semantics, and grasp taxonomy to produce contextually appropriate grasp descriptions.The descriptions connect physical interaction to language and provide a language interface for aligning robotic actions with human intent.Source: 3) Dexterous Grasp Multi-Dimensional… · p. 8

D. Vision Language Dexterous Hand Grasping Pose Action Generation Model

• The language-conditioned VLG model generates grasp poses from partial object point clouds and language instructions while modeling affordance, contact semantics, and grasp taxonomy.Its conditioning embeds language and point-cloud inputs into a shared semantic space.Source: D. Vision Language Dexterous Hand… · p. 8
• A diffusion transformer generates grasp poses by gradually denoising a Gaussian-initialized latent conditioned on semantic features.The noise prediction network is conditioned on timestep and semantic features.Source: D. Vision Language Dexterous Hand… · pp. 8–9, Figure 4 · p. 9
• The pose generator is trained on the annotated dexterous-grasp dataset produced by OmniDexDataGen and OmniDexReasoner after point-cloud projector training.Training targets high-quality poses responsive to multiple semantic conditions and language instructions.Source: D. Vision Language Dexterous Hand… · p. 9

A. Experimental Setup

• The real-world setup uses a 7-axis Diana arm and LEAP Hand; simulated evaluations also use LEAP and DLR-HIT II hands.Real-world grasp trajectories are planned with constraint-based optimization and executed under impedance control.Source: Figure 5 · p. 9, A. Experimental Setup · p. 9

1) Qualitative Experiment:

• The full method shows stronger affordance control, contact-configuration diversity, finger-joint variation, and taxonomy coverage across the LEAP and DLR-HIT II examples.Fig. 6 compares DexGraspNet [1], sampler variants, and the full method; Fig. 7 shows DLR-HIT II results.Source: Figure 6 · p. 10, Figure 7 · p. 10, 1) Qualitative Experiment · p. 10
• The qualitative comparison includes DexGraspNet [1], OGG with either Tax-DFCSampler or AC-Sampler, and the full combination.The evaluation covers grasp dataset generation and semantic-conditioned action generation.Source: 1) Qualitative Experiment · p. 10
• Without affordance-aware sampling, Tax-DFCSampler alone can target semantically mismatched regions, such as a refrigerator body instead of its handle.AC-Sampler supports adaptive grasp generation across objects of varying sizes.Source: 1) Qualitative Experiment · p. 10
• Removing Tax-DFCSampler limits contact-region and finger-configuration variation, biasing grasps toward multi-finger pinches.This is the OGG + AC-Sampler variant in Fig. 6 (3).Source: 1) Qualitative Experiment · p. 10

• Pose-level diversity uses standard deviations of hand position, orientation, and joint angles, plus KL divergence for categorical distributions.Categorical KL compares generated distributions with a uniform reference; lower divergence indicates greater diversity.Source: 2) Evaluation Metrics for Quantitative… · p. 11
• Affordance-level KL divergence evaluates diversity in grasp functionality by comparing generated categories with a uniform reference distribution.Source: 2) Evaluation Metrics for Quantitative… · p. 11
• Average Hausdorff distance between contact semantic maps measures contact-surface coverage, with higher values indicating more widely distributed contact points.The score averages pairwise distances across unique grasp pairs.Source: 2) Evaluation Metrics for Quantitative… · p. 11

3) Quantitative Experiments and Results:

• The full AC-Sampler and Tax-DFCSampler combination performs best across grasp diversity metrics, with highest pose standard deviations, lowest KL divergence, better contact coverage, and richer affordance variation.Source: 3) Quantitative Experiments and Results · p. 11

4) Analysis and Discussion:

• Affordance-driven contact sampling improves alignment with intended object regions, while taxonomy-aware DFC sampling generates types with distinct finger combinations, link involvement, and contacts.The method is also reported to achieve superior grasp-type, affordance, and contact-semantic diversity.Source: 4) Analysis and Discussion · p. 11

5) Limitation of OmniDexGen.:

• Similar visual and structural features across grasp types make straightforward classification difficult for off-the-shelf classifiers, motivating OmniDexReasoner for semantic categorization.Source: 5) Limitation of OmniDexGen · p. 11

1) Qualitative Experiments:

• OmniDexReasoner produces structurally coherent, semantically consistent descriptions across contact semantics, grasp taxonomy, and functional affordance, capturing interdependence among these dimensions.The qualitative examples are shown in Fig. 8.Source: 1) Qualitative Experiments · pp. 11–12, Figure 8 · p. 12

2) Quantitative Experiments:

• Reasoning accuracy is evaluated on 100 manually annotated grasp poses labeled for contact configuration, coarse and fine grasp types, and functional affordance.The evaluation computes multidimensional classification accuracies; configuration and model comparisons include Qwen and GPT variants.Source: 2) Quantitative Experiments · pp. 12–13

3) Ablation Study Results and Analysis:

• The multi-agent variant reaches ACCcoarse 0.89, ACCcontact 0.68, and ACCfunc 0.63, outperforming its single-agent counterpart.Source: 3) Ablation Study Results and Analysis · p. 13
• Removing visual inputs degrades ACChuman, removing finger-contact cues impacts ACCfine, and removing contact or affordance cues harms their corresponding classification measures.The reported effects are modality-specific, with affordance removal sharply reducing ACCfunc.Source: 3) Ablation Study Results and Analysis · p. 13
• CoT and RAG each improve metrics, their combination yields the best results, and GPT4o outperforms Qwen72B and GPT-4o-mini across evaluation dimensions.Source: 3) Ablation Study Results and Analysis · p. 13

1) Simulation Comparison Experiments:

• Simulation comparisons use 100 grasping trials and assess stable holding plus contact, taxonomy, and affordance consistency; GraspGPT fails to identify suitable candidates under complex instructions.Source: 1) Simulation Comparison Experiments · p. 13

2) Real-World Experiments:

• In real-world execution, evolving finger-object contact reduces contact and taxonomy accuracy and can change the grasp type intended by the semantic prompt.Source: 2) Real-World Experiments · p. 13
• Grasp success rate increases progressively with the number of fingers.The reported comparison covers different finger configurations.Source: 2) Real-World Experiments · p. 13, Table IV · p. 13

V. CONCLUSION AND FUTURE WORKS

• The unified framework explicitly models grasp taxonomy, contact semantics, and functional affordance to generate semantically consistent, structurally diverse grasps from natural-language instructions.Source: §V · p. 13
• OmniDexDataGen combines taxonomy-driven configuration sampling, affordance-aware contact sampling, taxonomy-aware DFC sampling, and physics-based optimization to produce semantically varied grasp samples.Source: §V · p. 13

3) Failure Examples:

• Fig. 12 illustrates failure examples, which the authors attribute primarily to suboptimal force closure or object slippage during execution.Source: 3) Failure Examples · p. 13, Figure 12 · p. 15

V. CONCLUSION AND FUTURE WORKS

• OmniDexReasoner uses multimodal-model reasoning, retrieval-augmented generation, and chain-of-thought mechanisms to infer grasp intentions and generate semantic annotations.Source: §V · p. 15
• Ablations report that affordance-aware sampling improves validity and affordance diversity, taxonomy-aware DFC sampling enriches grasp diversity, and RAG and CoT improve taxonomy-inference accuracy and stability.Source: §V · p. 15
• OmniDexVLG outperforms existing approaches in grasp diversity, contact distribution quality, semantic consistency, and generalization across functional tasks.Source: §V · p. 15
• Future work plans to extend the framework to task-chained language instructions and continuous manipulation trajectories.Source: §V · p. 15