Source-linked AI summary
Dex1B: Learning with 1B Demonstrations for Dexterous Manipulation
Jianglong Ye, Keyi Wang, Chengjing Yuan, Ruihan Yang, Yiquan Li, Jiyue Zhu, Yuzhe Qin, Xueyan Zou, Xiaolong Wang
TL;DR
Dexterous manipulation lacks demonstrations that capture its diverse and complex distributions, while generative models face feasibility and diversity challenges. Dex1B combines optimization with generative modeling to build a one-billion-demonstration dataset, and reported results include a 22% grasp-synthesis improvement over previous best methods.
Problem
Dexterous manipulation lacks data capturing its diverse and complex distributions, and generative models can have lower success rates and fail to expand dataset diversity.
Source: §I · p. 2Method
Dex1B iteratively seeds generation with optimization demonstrations, trains a generative model, refines proposals through optimization, and debiases the resulting data.
Source: §I · p. 2Results
22% performance improvement over previous best methods on grasp synthesis tasks is reported using Dex1B and DexSimple.
Source: §I · p. 2Takeaways & Limitations
Benchmarks for lifting and articulation highlight the effectiveness of Dex1B and DexSimple and their utility for dexterous-hand manipulation research.
Source: §VII · p. 9Key-frame action generation operates open-loop in real-world deployment, making the method prone to the sim-to-real gap and control or observation errors.
Source: §VII · p. 9
Abstract
from arXiv · showhide
Generating large-scale demonstrations for dexterous hand manipulation remains challenging, and several approaches have been proposed in recent years to address this. Among them, generative models have emerged as a promising paradigm, enabling the efficient creation of diverse and physically plausible demonstrations. In this paper, we introduce Dex1B, a large-scale, diverse, and high-quality demonstration dataset produced with generative models. The dataset contains one billion demonstrations for two fundamental tasks: grasping and articulation. To construct it, we propose a generative model that integrates geometric constraints to improve feasibility and applies additional conditions to enhance diversity. We validate the model on both established and newly introduced simulation benchmarks, where it significantly outperforms prior state-of-the-art methods. Furthermore, we demonstrate its effectiveness and robustness through real-world robot experiments. Our project page is at https://jianglongye.com/dex1b
I. INTRODUCTION
Dex1B addresses dexterous-manipulation data scarcity and generative models’ feasibility and diversity challenges with an optimization–generation pipeline. It produces a one-billion-demonstration dataset and introduces DexSimple, which improves grasp-synthesis performance over previous best methods.
I. INTRODUCTION
- The paper frames data scarcity as a barrier to diverse dexterous manipulation and notes the cost and imprecision of human annotation, optimization’s speed and initialization limits, and RL’s lack of diversity.Source: §I · p. 2
- Geometric constraints address generative models’ lower success rates, while additional conditions and sampling of rarer values encourage greater diversity.Source: §I · p. 2
- The pipeline starts with a high-quality optimization seed, trains a generative model, then refines proposals through optimization and debiases the data.Source: §I · p. 2
- DexSimple extends prior work [21] with conditional generation and enhanced loss functions, and is reported to achieve state-of-the-art performance across dexterous manipulation tasks.The approach is described as a baseline suited to the iterative pipeline and policy deployment.Source: §I · p. 2
- Dex1B combines optimization and generative models in an iterative pipeline for grasping and articulation demonstrations.It contains one billion demonstrations; compared with DexGraspNet’s similarly scaled object set, it offers 700× more demonstrations.Source: §I · p. 2
- 22% performance improvement over previous best methods on grasp synthesis tasks is reported using Dex1B and DexSimple.Source: §I · p. 2
II. RELATED WORK
Prior work spans control-based and learning-based dexterous manipulation, alongside synthetic and real-world datasets. The paper distinguishes Dex1B by its scale, diversity strategy, and joint treatment of grasping and articulation.
II. RELATED WORK
- Control-based methods support precise, agile, and safe manipulation, but generally lack generalization across diverse environments and use cases.Source: §II · p. 2
- Learning-based methods include direct pose generation, intermediate representations, and contact maps, but are sensitive to data quality and scope.Prior work also uses RL to transfer robust grasping to the real world using point-cloud or RGB inputs [29] [42].Source: §II · pp. 2–3
- Existing datasets range from synthetic grasp collections to real-world human-hand recordings, which are restricted to human hand structures and common daily poses.DexGraspNet2.0 [54] learns diffusion over optimized grasp demonstrations, whereas this work integrates generation into the data pipeline and covers grasping and articulation.Source: §II · p. 3
III. DEX1B BENCHMARK
Dex1B benchmarks grasping and articulation with one billion demonstrations across more than 6,000 objects and three dexterous hands. Its iterative generation pipeline combines optimization, generative scaling, debiasing, and planned motion to produce complete demonstrations.
III. DEX1B BENCHMARK
- The benchmark covers grasping and articulation across over 6,000 objects, three hands, and one billion complete action-sequence demonstrations.The hands are the Shadow, Inspire, and Ability Hands; sequences run from initial reaching through object manipulation.Source: §III · p. 3
- Demonstrations combine synthesized key hand poses at critical object interactions with motion-planned reaching, lifting, and opening sequences.The benchmark evaluation uses ManiSkill.Source: §III · p. 3
- The pipeline builds a small optimization-generated seed set, trains a generative model to scale it, then debiases and iteratively refines the dataset.The final dataset is Dex1B; the seed contains around 5 million poses.Source: §III · p. 3
- Seed-pose optimization uses a sphere-based hand approximation and task-specific energy terms, with force closure for grasping and task wrench space for articulation.The optimization generates around 1,000 grasps per minute on a single GPU; articulation replaces force closure because achieving it is difficult and unnecessary for that task.Source: §III · pp. 3–4
- Geometric constraints improve generative feasibility, followed by post-optimization to prevent penetration and ensure fingers closely cover objects.The post-optimization stage excludes task-specific terms because it makes only slight finger-position adjustments.Source: §III · p. 4
- Sampling is approximately 100 times faster, with 100 vs. 6000 iterations in post-optimization versus pure optimization.The refined poses are evaluated in the simulator.Source: §III · p. 4
- To expand diversity, the model conditions on object points and samples actions inversely to their observed frequency, while sampling more actions for more challenging objects.The process can be repeated by retraining on the expanded dataset.Source: §III · p. 4
- Reaching trajectories optimize intermediate hand poses for smoothness and collision avoidance, using sphere-based SDF errors against nearby scene meshes.The process starts from a manually defined initial action and goal action, with linear interpolation before optimization.Source: §III · p. 5
IV. DEXSIMPLE MODEL
DexSimple revisits the CVAE for dexterous manipulation, combining an SDF-based geometric constraint with additional conditioning for diverse generation. Its point-sphere SDF loss is reported to outperform state-of-the-art methods and to train more stably than the mesh-sphere loss used during optimization.
IV. DEXSIMPLE MODEL
- DexSimple uses a CVAE conditioned on object features, with PointNet encoding the input point cloud.The point-cloud feature is concatenated to both the VAE encoder and decoder inputs.Source: §IV · p. 5
- Training combines reconstruction with an SDF-based constraint, using sampled object points and sphere-based hand geometry.The sampled-point representation avoids the per-iteration BVH construction used during optimization.Source: §IV · p. 5
- With the SDF loss, DexSimple outperforms state-of-the-art methods by a large margin.The point-sphere SDF loss is reported as more stable than the mesh-sphere SDF loss used in optimization.Source: §IV · pp. 5–6
- The base model’s overall loss balances its component terms with separate weights.The text identifies λR, λKL, and λSDF as weights for the loss terms.Source: §IV · p. 6
- For data generation, DexSimple conditions on a local object feature associated with the closest point along the hand pose’s heading direction.Additional vectors can be concatenated to incorporate conditions.Source: §IV · pp. 5–6
V. EXPERIMENTS
The experiments evaluate DexSimple on grasp synthesis, compare datasets on lifting and articulation, and examine dataset scale and design choices.
V. EXPERIMENTS
- The evaluation first tests DexSimple for grasp synthesis.The paper also compares synthesized Dex1B demonstrations with human-annotated datasets for lifting and articulation.Source: §V · p. 6
- The experiments compare methods trained on Dex1B with methods trained on human-annotated datasets for lifting and articulation.The evaluation also downscales Dex1B to assess methods trained on smaller versions.Source: §V · p. 6
- The evaluation additionally studies downscaled Dex1B and conducts ablations of the proposed design choices.Source: §V · p. 6
A. Grasping Synthesis Evaluation
On DexGraspNet, DexSimple leads the reported grasp-synthesis quality metrics and improves diversity over baseline methods. Filtering further raises its success rate, while its unfiltered, unoptimized success rate is slightly below DDG’s.
A. Grasping Synthesis Evaluation
- 92.6% success rate is reported when UGG’s discriminator filters DexSimple grasps.Source: A. Grasping Synthesis Evaluation · p. 6
- 86.0% success rate, 0.125 Q1 score, and 0.13 penetration are DexSimple with post-optimization’s best reported quality results; its entropy mean is 8.56.The comparison is on the DexGraspNet benchmark; the table caption states that DexSimple significantly outperforms all baselines.Source: Table II · p. 6, A. Grasping Synthesis Evaluation · p. 6
- Without post-optimization or filtering, DexSimple’s success rate is slightly below DDG’s, but its diversity is higher: 8.53 versus 5.68.The paper attributes the expected trade-off to DexSimple being generative while DDG is regression-based.Source: A. Grasping Synthesis Evaluation · p. 6
B. Dataset Analysis
Dex1B spans grasping and articulation demonstrations, with data curated and rebalanced through iterative generation. Its joint values are more evenly distributed than those in DexYCB and ARCTIC, and methods trained on Dex1B outperform counterparts trained on those datasets across tasks, baselines, and splits.
B. Dataset Analysis
- Dex1B covers grasping and articulation, using 5,751 grasping assets and more than 650 articulation assets under the stated table-top task settings.Grasping demonstrations reach, grasp, and lift an object to 0.4 m; articulation demonstrations open an object by 0.5 in joint angle while maintaining contact.Source: B. Dataset Analysis · p. 6
- Curation iteratively grows an optimized 5-million-demonstration seed through 50-million and 500-million proposal datasets to 950 million successful trajectories.The pipeline refines proposals, filters unsuccessful trajectories, and rebalances the data before retraining.Source: B. Dataset Analysis · pp. 6–7
- Models trained on Dex1B consistently outperform those trained on DexYCB or ARCTIC across tasks, baselines, and splits.The reported comparison includes lifting and articulation benchmarks.Source: Table III · p. 7, B. Dataset Analysis · p. 8
- Retargeted and noise-augmented human demonstrations from DexYCB and ARCTIC achieve their task goals in 62% and 64% of collected trajectories, respectively.DexYCB contains 20 objects; ARCTIC contains 10 articulation objects and 301 trajectories.Source: B. Dataset Analysis · p. 7
- Dex1B’s joint-value distribution is more evenly spread around the mean, unlike DexYCB and ARCTIC, whose values often concentrate near limits.The paper attributes this to debiasing and an optimization regularizer discouraging proximity to joint limits.Source: B. Dataset Analysis · p. 7, Figure 6 · p. 7
- Articulation is more resilient than lifting to data reduction, retaining reasonable performance with less data while lifting needs more for stable performance.The scaling figure reports consistent improvement with more training data for both tasks.Source: Figure 8 · p. 8
C. Scaling the Dataset
DexSimple success rates improve with more training data in both lifting and articulation. Lifting degrades more sharply as data decreases, while articulation remains more resilient.
C. Scaling the Dataset
- Both lifting and articulation success rates improve with more training data, with degradation increasing as data is reduced.The authors report a more pronounced degradation for lifting than articulation.Source: C. Scaling the Dataset · p. 8
- Articulation maintains reasonable performance with less data, whereas lifting requires a more extensive dataset for stable performance.The authors relate lifting’s greater sensitivity to its reliance on stable grasping and precise object geometry; articulation emphasizes trajectory execution and adapts to unseen objects through more generalized motion.Source: C. Scaling the Dataset · p. 8
D. Ablation Study
The geometric losses address distinct aspects of grasp quality: Lsdf prevents object penetration, while LD encourages stable surface contact. Both contribute to grasp quality, with little impact on diversity metrics.
D. Ablation Study
- Removing Lsdf drops grasp success rate from 63.7 to 0.7, while the loss helps prevent object penetration during grasp synthesis.The authors describe the sphere-representation SDF loss as providing fine-grained geometric guidance.Source: D. Ablation Study · p. 8
- LD encourages stable hand-object contact and improves success rate and Q1, although it slightly increases penetration value.The authors characterize its contribution as enhancing grasp stability.Source: D. Ablation Study · p. 8
- The diversity metrics H mean and H std are minimally affected by either geometric loss, which the authors associate primarily with grasp quality.The two losses address penetration prevention and stable contact, respectively.Source: D. Ablation Study · p. 8
VI. REAL-WORLD EXPERIMENTS
In real-world grasping trials on unseen objects, the proposed method outperformed DexSampler. The experiments used point-cloud inputs and motion-planned execution across xArm with an Ability Hand and H1 with an Inspire Hand.
VI. REAL-WORLD EXPERIMENTS
- The real-world platforms were xArm with an Ability Hand and H1 with an Inspire Hand.The xArm camera used a third-person view and the H1 camera an egocentric view; camera poses were calibrated using hand-eye calibration.Source: §VI · pp. 8–9
- 96% mean success for Ours versus 58% for DexSampler in real-world grasping trials on 10 unseen objects.Each method was evaluated for five trials per object; the authors report better sim-to-real transfer for their method.Source: §VI · p. 9
- The grasping pipeline takes a partial point cloud, samples 128 poses, selects a valid IK solution, and applies motion planning.The successful grasping trajectories are visualized in Fig. 9.Source: §VI · p. 9, Figure 9 · p. 9
- DexDiffuser was retrained for the xArm and Ability Hand platform, using its DexSampler module without the DexEvaluator.Both methods used partial point clouds and motion planning to execute grasp poses.Source: §VI · p. 9
VII. CONCLUSION AND LIMITATIONS
Dex1B is a synthetic dexterous-manipulation dataset of one billion demonstrations built through iterative optimization and generative modeling. DexSimple with geometric loss improves on prior state-of-the-art methods on DexGraspNet, while the approach has open-loop, simulation-runtime, and scene-scope limitations.
VII. CONCLUSION AND LIMITATIONS
- Dex1B contains 1 billion demonstrations generated through an iterative pipeline that seeds a generative model with optimization data, then refines, verifies, and debiases its proposals.The pipeline is presented as an efficient way to generate manipulation demonstrations.Source: §VII · p. 9
- A 22-point improvement over previous state-of-the-art methods on DexGraspNet is reported for DexSimple enhanced with geometric loss.The paper also reports that lifting and articulation benchmarks highlight the effectiveness of Dex1B and DexSimple.Source: §VII · p. 9
- Real-world deployment is open-loop and prone to sim-to-real and control or observation errors; data filtering still relies on time-consuming simulation, and the method mainly considers single-object scenes.The authors identify reducing simulation runtime and using a stronger vision backbone for multi-object scenarios as future directions.Source: §VII · p. 9
I. GRASPING SYNTHESIS EVALUATION DETAILS
The section specifies grasp-synthesis evaluation and benchmark task setups, then details optimization for grasping and motion planning and the DexSimple model and training losses.
I. GRASPING SYNTHESIS EVALUATION DETAILS
- DexGraspNet evaluation uses 5,355 objects, split into 4,229 training and 1,126 test objects, and reports grasp quality and diversity metrics.Quality includes success rate, Q1-score, and penetration; diversity uses joint-angle entropy, with H mean and H std reported across joints. Entropy is estimated using 10,000 bins, following the UGG implementation.Source: §I · p. 12, §II · p. 12
II. BENCHMARK DETAILS
- Dex1B evaluates lifting and articulation in simulation: lifting grasps and raises an object to 0.4 m with at least two-finger final contact, while articulation opens an interactable link by 0.5 radians with that contact.The lifting split has 3,491 training and 933 test objects; the articulation split has 43 training and 21 test objects.Source: §II · p. 12
III. ITERATIVE DATA ENGINE DETAILS
- The iterative data engine begins with a controllable optimization algorithm for hand-pose and trajectory optimization.Both grasp synthesis and motion planning are formulated as optimization problems.Source: §III · p. 12
A. Grasping Synthesis as Optimizaiton
- Grasp optimization combines force-closure, signed-distance, contact-distance, self-penetration, and joint-limit energy terms.The penetration term uses sphere-hand geometry; the stated weights are 100.0 for SDF, 100.0 for contact distance, 10.0 for self-penetration, and 1.0 for joint limits.Source: A. Grasping Synthesis as Optimizaiton · pp. 12–13
B. Motion Planning as Optimizaiton
- Motion planning optimizes SDF, self-penetration, joint-limit, and smoothness energies across multiple frames.For trajectory rotation, smoothness and rotation-difference terms optimize revolute joint angles to address discontinuities when converting rotation representations to intrinsic Euler angles.Source: B. Motion Planning as Optimizaiton · p. 13
C. Task-specific Trajectory Optimization
- Lifting and articulation demonstrations use pre-grasping motion planning, grasp synthesis, and task-specific post-grasping trajectories.The post-grasping stage raises hand height for lifting and rotates the hand along the articulated structure's axis for articulation.Source: C. Task-specific Trajectory Optimization · p. 13
IV. DEXSIMPLE MODEL DETAILS
- DexSimple adds differentiable SDF, distance, and smoothness losses to generative models, and uses a CVAE architecture with a PointNet encoder.During large-scale training, point-sphere SDF queries replace mesh-sphere queries because warp-lang lacks BVH cache support.Source: §IV · p. 13