OmniDex: Scaling Dexterous Hand Grasping to Diverse Cluttered Scenes

1 Introduction

• Real-world dexterous grasp data are expensive to collect, while generative models must map perceptual inputs to high-DoF grasps with multiple valid poses and millimeter-sensitive contacts.Existing approaches may use intermediate representations or costly post-optimization and test-time fine-tuning.Source: §1 · p. 2
• Prior cluttered-scene work faces constrained benchmark scale, geometry-focused perception, and limited diversity in supporting bases and layouts.Existing benchmarks cited here contain 0.4K–8K scenes; depth-based pseudo point clouds can hinder semantic target localization.Source: §1 · p. 2
• The benchmark uses seed-and-filter generation to provide over 2.6 million realistic scenes and 0.4B scene-specific validated grasp ground truths.It includes diverse supporting bases and multimodal observations.Source: §1 · p. 2
• OmniDex couples Soft Winner-Takes-All learning with human-inspired physical constraints and physics-driven ranking to address grasp multimodality and last-millimeter precision.The ranking mechanism is used during inference.Source: §1 · pp. 2–3
• OmniDex achieves state-of-the-art performance on its benchmark and generalizes across scene subsets, supporting bases, input views, and unseen objects.The reported results describe robustness in realistic cluttered scenes.Source: §1 · p. 3

2.1 Dexterous Grasping Datasets and Benchmarks

• Single-object simulation datasets reached billion-scale, but did not model clutter's spatial dynamics; newer cluttered-scene benchmarks [9] [14] remained limited by costly per-scene optimization.OmniDex's seed-and-filter strategy is presented as a way to bypass that optimization while producing 2.6 million scenes and 0.4B validated ground truths.Source: §2.1 · p. 3
• OmniDex pairs seed-and-filter data generation with a Soft Winner-Takes-All objective and human-inspired physical constraints during training, plus physical ranking during inference.The approach is framed as addressing data scale, multimodal grasp distributions, precision, and optimization latency.Source: §2.1 · p. 3, §2.2 · p. 3

2.2 Dexterous Grasp Generation Methods

• Generative grasp methods face multimodality and last-millimeter errors, while heuristic contact maps [8] [10] or affordance fields [9] [20] make pipelines non-end-to-end.Test-time optimization [21] can compromise real-time efficiency; prior internalized physical priors [18] are described as insufficient for dense clutters.Source: §2.2 · p. 3

3.1 Overview of Data Generation

• The seed-and-filter pipeline reuses free-space seed grasps, filters transformed candidates in settled clutter simulations, and renders aligned visual observations for supervision and evaluation.Accepted grasps contact the target while avoiding support surfaces and clearly disruptive neighboring-object interactions; the task targets direct retrieval rather than rearrangement.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 3D Assets

• The asset library includes over 6,000 curated graspable objects and supporting bases comprising tables, open-top boxes, and shelves.Objects are screened for grasping suitability and represented by spatially aligned physics-ready and high-fidelity visual models.Source: §3.2 · p. 4

3.3 Scene-Specific Ground-Truth Grasp Pose Generation

• Scene-specific labels are formed by applying selected seed grasps to settled object poses, then sanity-checking scenes with visual models.The scene configuration combines object poses with dexterous grasp poses and their wrist translation, wrist orientation, and joint angles.Source: §3.3 · pp. 4–5

3.4 Diversity and Protocol

• The benchmark varies objects, supports, and clutter layouts, with one ego-centric and four auxiliary views by default and separate test tracks for layout and object novelty.Dstd varies target objects with a fixed support; Dmix varies objects and supports, while Dbox, Dgrid, and Dshelf model confined packing, identical-object grids, and shelf arrangements.Source: §3.4 · pp. 5–6

3.5 Metrics

• Grasp evaluation reports Success Rate variants for scene gravity, all six canonical directions, or at least one canonical direction, plus Collision-Free Rate for support-surface and non-target contacts.SRscene is the primary stability metric; CFR measures protocol-consistent scene interaction under direct retrieval.Source: §3.5 · p. 6

4.1 Overview

• The single-view model predicts a target grasp pose from RGB, depth, an instruction mask, camera parameters, and table height.The method also describes a multi-view extension in Appendix C.2.Source: §4.1 · p. 6

4.2 Multimodal Scene-Aware Condition Embedding

• RGB, depth, the instruction mask, and scene parameters are fused into a unified tokenized condition embedding for scene-aware grasp generation.Image features provide semantics, depth features provide geometry, and camera parameters and table height anchor the representation in a global frame.Source: §4.2 · p. 7

4.3 Physically-Constrained Denoising for Multimodal Grasp Generation

• Soft-WTA training selects the most compatible among multiple valid grasp modes, while physics constraints address stable contact and millimeter-scale errors.The physics-constrained objective is defined on a differentiable dexterous hand simulator.Source: §4.3 · p. 7

4.4 Physics-Driven Ranking and Selection

• At inference, the model generates multiple poses and ranks them by physical criteria, filtering invalid modes to select an optimal grasp.The score evaluates force-closure stability, rewards additional fingertip contacts, and penalizes interpenetration, collisions, or missing thumb-opposition.Source: §4.4 · pp. 7–8

5.1 Implementation Details

• The framework uses the Shadow Dexterous Hand with 24 DoF; benchmark subsets use an 8/2 train-test split and separate seen-object and unseen-object tracks.The seen-object test track uses unseen clustered scenes.Source: §5.1 · p. 8

5.2 Main Results

• 55.80 SRscene and 84.12 CFR on Dstd ego-view surpass DexGraspVLA by 14.81 and 2.68 points; unseen-object SRscene is 59.74.All Table 3 baselines were retrained on the same OmniDex training split.Source: Table 3 · p. 8, §5.2 · p. 8

5.3 Ablation Studies

• CFR rises from 81.54% (20% subset) to 83.66% (100% subset), with performance scaling strictly with data volume.The authors report that scaling enhances reasoning about inter-object physical interactions in cluttered layouts.Source: §5.3 · p. 9, Table 4 · p. 9
• The input ablation identifies depth as foundational for shape cognition, RGB as supplying semantics and target localization, and camera parameters as supporting geometric perception.Multi-view inputs yield performance gains, while table height anchors depth-derived geometry to grasp ground truths.Source: §5.3 · p. 9, Table 5 · p. 9
• Component ablations find that Soft-WTA improves SRscene over standard denoising, physical constraints address last-millimeter contact, and ranking raises SRscene from 30% to 40%.Test-time optimization yields limited gains, which the authors attribute to instability and depth-backprojection occlusions.Source: §5.3 · p. 9, Table 6 · p. 9

6 Conclusion

• A scalable seed-and-filter strategy produces over 2.6 million scenes and 0.4 billion validated grasp ground truths with rich multimodal observations.The benchmark includes diverse supporting bases.Source: §6 · p. 9
• The generation framework couples Soft Winner-Takes-All learning with human-inspired physical constraints to address grasp multimodality and last-millimeter precision.Source: §6 · p. 9
• The authors report state-of-the-art performance on the proposed benchmark and significant outperformance over existing baselines on standard cluttered scenes.Source: §6 · p. 9

A.1 Cluttered Scene Geometry Database Generation

• Supporting bases are gravity-aligned with a standardized local origin, and all objects use density 500 kg/m³ with linear and angular damping of 1.0 and 0.3.Source: §A.1 · p. 12
• Dstd and Dmix use 2–7 objects, Dbox uses 2–10, Dgrid uses 4–10 identical objects, and Dshelf arranges objects by annotated layer heights.The subsets use different drop heights, spacing, or placement areas to produce their respective layouts.Source: §A.1 · p. 12
• Scenes settle until object velocities fall below 0.03 m/s and 0.10 rad/s or 500 steps, then undergo geometric validation against specified tolerances.The supplied thresholds include maximum sinking or floating of 3 mm and inter-object penetration of 2 mm.Source: §A.1 · p. 12
• Contact filtering tolerates object–support penetration up to 2 cm before rejection, retaining shallower overlaps to reduce false negatives from imperfect collision geometry.Source: §A.1 · p. 12

A.2 Cluttered Scene Visual Database Generation

• Five fixed cameras provide visual coverage, with tighter placement for box scenes and layer-aligned, horizon-level views for shelves.For other subsets, the primary camera is above the origin and auxiliary cameras sit near table corners.Source: §A.2 · pp. 12–13

B Benchmark Split and Density Details

• The Dstd and Dmix subsets span density ratios of 0.1, 0.2, and 0.3, with about 0.6M scenes per tier; the roughly 90K-scene ablation mini-split samples across tiers.The unseen-object split is relatively simpler on average because more challenging assets were assigned to the seen split, so the tracks are not difficulty-balanced.Source: B Benchmark Split and Density Details · p. 13

C.1 Single-View Condition Embedding

• For single-view input, DINOv2 RGB features are combined with depth-derived XYZ features, instruction-mask features, camera parameters, and table height.The features are fused into a scene representation with an appended parameter token.Source: §C.1 · p. 13

C.2 Multi-View Input Fusion

• Multi-view fusion learns RGB view weights, reprojects depth into the anchor camera using camera parameters, and adds tokens encoding camera relationships.Source: §C.2 · pp. 13–14

C.3 Physically-Constrained Loss

• The physical loss uses simulated hand contact points and target-object points to penalize contact distance, emphasizing thumb opposition and the K closest opposing-finger points.A separate closure term encourages opposing approach directions using the thumb and opposing-finger vectors.Source: §C.3 · p. 14
• An SDF loss penalizes hand-mesh vertices inside the target object, discouraging penetration of its interior volume.Source: §C.3 · pp. 14–15

D Supplementary Implementation Details

• Training uses 25 epochs, M = 64 hypotheses, λWTA = 1, and λphys = 100, with physical loss disabled for 10 epochs and ramped to its target by epoch 20.The physical loss weights are calibrated as Lthumb : Lopp : LSDF : Lclosure = 5 : 1 : 0.2 : 2.Source: D Supplementary Implementation Details · p. 15
• The benchmark uses third-party assets from OmniObject3D, ShapeNetCore v2, Google Scanned Objects, ABO, and Isaac Lab under their stated licenses and terms.Source: D Supplementary Implementation Details · p. 15

E.1 Inference Ablation

• Increasing the parallel hypothesis pool M consistently improves grasp success with negligible inference-latency sacrifice, while TTP has prohibitive overhead and optimization instability.The comparison is presented in Table D1.Source: §E.1 · p. 15, Table D1 · p. 15

E.2 Failure Analysis

• Among SRscene failures, 95.71% are stability failures after contact and 4.29% insufficient contact; among CFR failures, 61.26% involve support collisions and 38.74% non-target collisions.The breakdowns are summarized in Table D2.Source: §E.2 · pp. 15–16, Table D2 · p. 16

E.3 IK Feasibility Audit

• Across five Franka arm-base configurations, IK feasibility for grasps passing SRscene ranges from 83.75% to 87.98%.The audit results are detailed in Table D3.Source: §E.3 · p. 16, Table D3 · p. 16

F.1 OmniDex Benchmark Visual Results

• The benchmark galleries show multi-view images, depth maps, instruction masks, and physically validated grasp ground truths for representative scenes.Figures E1–E5 illustrate the cluttered layouts and aligned modalities.Source: §F.1 · p. 16

F.2 OmniDex Model Visual Results

• OmniDex grasps remain stable and semantically consistent across layouts, including tight spacing and geometrically similar instances.These visualizations support the reported generalization across scene structures and grasping difficulties.Source: §F.2 · p. 16

F.3 Real-Robot Demonstration

• Figures E1–E5 provide qualitative visualizations of the Dstd, Dmix, Dbox, Dgrid, and Dshelf subsets.Source: Figure E1 · p. 17, Figure E2 · p. 17, Figure E3 · p. 18
• In a small real-robot study, OmniDex performed representative cluttered-scene grasping trials on an apple, mangosteen, and cup.The study used a Tianji Marvin M6S arm and BrainCo Revo2 hand, with camera placement approximately matching simulation; the model was retrained on a hardware-matched mini-split and SAM generated target masks.Source: §F.3 · pp. 16, 20, Figure E7 · p. 20

• A non-negligible sim-to-real gap may remain because sensor noise, calibration error, contact mismatch, and actuation uncertainty are not modeled.Source: G Limitation · p. 20
• Benchmark construction and the main evaluation are conducted in simulation, despite realistic approximations including diverse bases, multi-view RGB-D, and physics-based validation.Source: G Limitation · p. 20
• Generating the full benchmark and training the model involve substantial GPU costs, motivating further resource reduction.Source: G Limitation · p. 20