OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping

• Functional grasping asks whether an object is held to preserve an intended action, beyond whether it can be held stably.The task requires grounding free-form instructions in multi-view observations and object geometry while generating a feasible high-DoF grasp.Source: §1 · p. 2
• The paper treats functional affordance and dexterous pose as complementary supervision views of one task-conditioned perception-action distribution, rather than independent causal stages.Affordance supervision makes the latent interpretable at the object surface, while action supervision makes it generative and physically grounded.Source: §1 · p. 2
• OpenDexVerse pairs OpenDex-Scale’s broad semantic-geometric coverage with OpenDex-Align’s high-quality functional alignment through teleoperation and category-level transfer.The recipe is designed to build broad semantic-geometric priors before aligning the grasp distribution to reliable functional behavior.Source: §1 · pp. 2–3
• OPENDEXGRASP jointly learns open-vocabulary functional grounding and dexterous grasp pose generation within a shared perception-action latent space.It fuses vision-language, geometry, and action tokens before grasp generation, with point-level affordance supervision on the same representation.Source: §1 · pp. 2–3
• The C2A Recipe is evaluated in simulation and on a real dexterous hand, with reported improvements in functional alignment, physical plausibility, inference efficiency, and real-world task execution success.Source: §1 · p. 3

• Geometry-driven methods improve physical feasibility and grasp diversity, but their objectives usually prioritize stable holding over functional use.Language-conditioned approaches [14] [15] add task descriptions or intent labels, though many have closed task spaces or weak coupling between semantics and hand-object contact.Source: §2 · p. 3
• Open-vocabulary affordance pipelines provide interpretable grounding, but errors can propagate from part prediction to action generation and test-time retrieval or optimization can be expensive.OPENDEXGRASP retains point-level affordance supervision [17] while shaping a shared action representation instead of requiring affordance as an inference stage.Source: §2 · p. 3
• Functional grasping is judged by downstream manipulation, and OPENDEXGRASP combines structured 3D geometry with open-vocabulary context while evaluating simulated physical metrics and real-world functional success.Demonstration-based [21] [22] and reinforcement-learning [27] approaches have also advanced closed-loop functional execution.Source: §2 · p. 3

• Each example pairs a free-form instruction, multi-view RGB observations, an object point cloud, a dexterous grasp pose, and an optional point-level functional affordance label.The affordance label is auxiliary; the target is direct pose generation from language, images, and geometry.Source: §3 · p. 3
• The grasp pose comprises global hand translation, a continuous 6D wrist rotation representation, and the hand joint configuration.Source: §3 · p. 3
• OpenDexVerse’s C2A pipeline pairs automatically synthesized, annotated grasps for coverage with teleoperated and transferred demonstrations for alignment.The training-data table describes OpenDex-Scale as semantic-geometric coverage and OpenDex-Align as embodied functional alignment.Source: Figure 2 · p. 4, Table 1 · p. 4
• Affordance and action are modeled as observations of a shared latent state, with affordance grounding and grasp generation supervising that state rather than forming a test-time cascade.Source: §3 · p. 4

4 OpenDexVerse Construction

• OpenDexVerse pairs broad semantic-geometric coverage from OpenDex-Scale with high-quality embodied alignment from OpenDex-Align in a shared learning pipeline.The recipe first expands language-conditioned grasp support, then aligns it to embodied functional behavior; Figure 3 shows examples from both subsets.Source: §4 · p. 4, Figure 3 · p. 5
• Omni6DPose supplies scanned object instances with category-level coordinates, supporting realistic shape variation and correspondence-based transfer.Categories and instances are selected for scale, geometry, and functional suitability for multi-finger interaction.Source: §4 · pp. 4–5
• OpenDex-Scale generates physically plausible grasp candidates for selected instances, renders three object-centered views, and uses vision-language annotation for functional use and task descriptions.The resulting data intentionally prioritizes breadth over perfect curation across semantic-geometric configurations.Source: §4 · p. 5
• OpenDex-Align collects teleoperated grasps from three size-aware templates per category and transfers them to nearby, compatible-scale instances using category-level correspondences.These demonstrations ground contact placement, hand posture, and task intent in embodied grasps.Source: §4 · p. 5

5.1 Overview

• The model extracts functional context from multi-view images and language, conditions on point-cloud geometry, and uses flow matching to generate dexterous grasp poses directly.Point-level affordance supervision grounds the action distribution on functional regions without making affordance prediction a test-time bottleneck.Source: §5.1 · p. 5, Figure 4 · p. 6

5.2 Open-Vocabulary Semantic-Geometric Latent

• Multi-view images and instructions yield vision-language hidden states that retain task-word, object-appearance, and view-dependent part correlations, while point-cloud features encode object shape for contact.A global geometry token conditions generation, and intermediate point features support affordance decoding.Source: §5.2 · pp. 5–6

5.3 Flow-Matching Dexterous Action Generation

• A transformer flow-matching action model interpolates noise and target actions, fuses geometry, query, and action tokens, and predicts the target velocity conditioned on vision-language states.This directly models task-conditioned actions rather than relying on post-hoc candidate filtering.Source: §5.3 · p. 6

5.4 Auxiliary Affordance Grounding

• An auxiliary affordance head predicts point-level functional maps and adds focal-plus-Dice grounding loss to the action-generation objective.The combined objective weights affordance grounding by λ_aff = 0.3.Source: §5.4 · p. 6

6.1 Experimental Setup

• The evaluation uses OpenDex-Scale for broad annotated grasp coverage and OpenDex-Align for teleoperated functional supervision and category-level transfer, comparing functional and non-functional grasps on seen and unseen splits.The adapted DexGraspNet 2.0⋆ baseline adds CLIP features and uses the same data and splits; reported metrics cover interpenetration, displacement, success, diversity, and perceptual quality.Source: §6.1 · pp. 6–7, §6 · p. 6

6.2 Main Results

• OPENDEXGRASP achieves lower SIV, PD, and SD than DexGraspNet 2.0⋆, with a smaller physical-plausibility gap on unseen objects.Table 2 reports simulation results on seen and unseen splits; the cited results passage describes the metric comparisons but gives no values.Source: §6.2 · p. 7, Table 2 · p. 7
• OPENDEXGRASP produces a wider range of grasp styles while maintaining stronger physical feasibility; Perceptual Score assesses naturalness and task appropriateness through automated and human judgments.The supplied results passage does not give numerical scores.Source: §6.2 · p. 7

6.3 Ablation Study

• Reducing OpenDex-Scale pretraining most clearly degrades physical robustness and simulation success, underscoring the role of broad geometric and semantic coverage.The ablation passage attributes this coverage to large-scale automatically synthesized and annotated data.Source: §6.3 · p. 8
• Removing OpenDex-Align weakens physical plausibility and perceptual quality; slightly greater style diversity comes with less natural and less functionally aligned grasps.The passage suggests teleoperated demonstrations refine contact choices and hand articulation.Source: §6.3 · p. 8
• Without affordance grounding, grasps remain physically plausible, but simulation success and perceptual quality decrease.The ablation passage says point-level supervision helps associate task semantics with functional object regions rather than only stable contact.Source: §6.3 · p. 8
• Replacing the pretrained Qwen VLM with CLIP encoders degrades physical, semantic, and success metrics, indicating the importance of pretrained VLM features for grounding instructions and object-part cues.The comparison is the w/o pretrained VLM ablation.Source: §6.3 · p. 8

6.4 Real-World Experiments

• Real-hand tests on everyday objects from seen and unseen categories show OPENDEXGRASP outperforming DexGraspNet 2.0⋆ in success rate and GPT-5 and human-rated perceptual quality.Table 4 reports success rate and perceptual scores for five seen and unseen test objects separately.Source: §6.4 · p. 8, Table 4 · p. 8

7 Conclusion and Limitation

• OPENDEXGRASP jointly grounds functional affordances and generates hand poses from free-form instructions, multi-view observations, and object geometry.The framework combines broad semantic-geometric coverage with high-quality embodied alignment through OpenDexVerse and the Coverage-to-Alignment recipe.Source: §7 · p. 9
• Simulation and real-robot experiments show improved task alignment, physical plausibility, generalization, and execution success.Source: §7 · p. 9
• Part-grounding errors may affect grasp generation because the method depends on its vision-language representation and visual coverage; it also uses open-loop actions without tactile feedback or closed-loop correction.OpenDex-Align remains modest relative to household-tool diversity.Source: §7 · p. 9

8 Limitation

• Part-grounding failures can affect generated grasps, and the open-loop model lacks tactile feedback and closed-loop correction.OpenDex-Align also remains small relative to the diversity of household tools and long-horizon tasks.Source: §8 · p. 9

A.1 Implementation Details

• The action head uses a flow-matching diffusion transformer conditioned on selected Qwen hidden states, with alternating cross-attention and causal self-attention blocks.Its action-side sequence contains a point-cloud token, future-query tokens, and a noisy action token.Source: §A.1 · p. 12

A.2 C2A Details

• C2A pretrains on OpenDex-Scale for broad coverage, then fine-tunes the same latent model on OpenDex-Align to refine contact selection and hand articulation.The alignment stage changes supervision from automatic feasibility toward embodied functional preference without adding a separate inference procedure.Source: §A.2 · pp. 12–13

B.1 Object Asset Selection

• The object pool filters Omni6DPose’s 153 categories and 4,302 instances to 105 graspable categories, then divides them into 185 category-size pairs.For teleoperation, the authors select 95 categories and 159 pairs, excluding objects too large or awkward for human handling.Source: §B.1 · p. 14

B.2 Relationship Between Columns in Table 1

• OpenDex-Align contains 27,425 grasp poses: 1,590 teleoperated poses (10 for each of 159 category-size pairs) and 25,835 transferred poses across additional instances within those pairs.Source: §B.2 · pp. 14–15

B.3 Text Description Generation

• The annotation pipeline uses a VLM to label each grasp’s functionality and generate five grasp descriptions and five task instructions.Labels draw on six RGB views and hand-object contact data; descriptions characterize the hold, while instructions state an achievable task without regrasping.Source: §B.3 · p. 16

B.4 Data Visualization

• OpenDex-Scale visualizes automatically synthesized grasps across object categories, sizes, poses, and contact modes.The examples illustrate broad semantic-geometric coverage without human supervision.Source: §B.4 · pp. 17–18
• OpenDex-Align examples show natural hand articulation and functional contact placement from teleoperation and dense-correspondence transfer.The subset is smaller than OpenDex-Scale but provides embodied functional supervision.Source: §B.4 · p. 18

C.1 Train/Test Split

• The test set separates functional from non-functional tasks and further divides each by seen versus unseen object categories, using three instances per category.Functional categories require task-oriented grasping; non-functional categories require general pick-and-place manipulation.Source: §C.1 · p. 18, Table 6 · p. 18

C.2 Metric Calculation

• Evaluation covers penetration depth, solid intersection volume, simulated success and displacement, style diversity, and GPT-5-assisted and human perceptual ratings.The success protocol uses a fixed hand pose, 0.1 kg object mass, friction 1.0, gravity, a 2-second rollout, and a 0.2 m displacement threshold; displacement is computed only for successful grasps.Source: §C.2 · pp. 18, 21

C.3 Baseline Details

• The adapted DexGraspNet 2.0⋆ baseline adds CLIP language conditioning to point-wise geometry features while retaining the original seed-point and grasp-generation pipeline.It is trained and evaluated on the same splits as OpenDexGrasp.Source: §C.3 · pp. 21–22

C.4 Real Robot Setup

• Real-robot tests use a Sharpa Wave Hand on a Franka Panda, with a RealSense D435 for object pose estimation before planned execution.Ten valid poses per object are each executed once after manual rejection of table-colliding or occluded poses and a fixed 0.05-rad squeezing refinement; high-risk objects are excluded.Source: §C.4 · p. 22, Figure 8 · p. 22

D.1 Ablation Study: w/o Pose Generation Module

• OpenDexGrasp’s direct pose generation outperforms affordance-then-optimization across grasp-quality metrics and inference time.A single forward pass generates a pose in a fraction of a second, while the optimization baseline requires hundreds of iterative steps per grasp.Source: Table 7 · p. 23, §D.1 · p. 23

D.2 Affordance Prediction Results

• Different instructions shift predicted high-affordance regions, and generated finger placement and wrist orientation align with those regions; optimization often yields penetration or functional misalignment.Both approaches use the same predicted affordance, isolating the contrast between direct decoding and test-time optimization.Source: §D.2 · p. 24, Figure 9 · p. 24

D.3 Real Robot Manipulation Visualization

• Figure 10 presents keyframes from ten real-world experiments showing task-conditioned reaching, grasping, and lifting.The supplied passage identifies the depicted actions but reports no execution outcome.Source: Figure 10 · p. 25, §D.3 · p. 25

E Failure Case Analysis

• Thin-geometry objects with ambiguous part semantics can cause finger penetration or fragile contacts, even when a grasp looks visually close to valid.The model may default to stable contacts on broad surfaces, rims, or key regions.Source: E Failure Case Analysis · p. 26
• Fine-grained instructions may reach the right semantic neighborhood but miss exact fingertip contacts, partly because OpenDex-Align has limited such demonstrations.This is particularly challenging when an instruction requires precise contact on a small actuation region.Source: E Failure Case Analysis · p. 26
• The remaining failures center on high-precision contact reasoning, with stronger collision handling and denser part-level supervision proposed as remedies.Thin objects motivate geometry-aware collision handling; localized interactions motivate more demonstrations or higher-resolution part/contact constraints.Source: E Failure Case Analysis · p. 26