PaperlayerView exact revision on arXiv

Source-linked AI summary

RealDex: Towards Human-like Grasping for Robotic Dexterous Hand

Yumeng Liu, Yaxun Yang, Youzhuo Wang, Xiaofei Wu, Jiamin Wang, Yichen Yao, Sören Schwertfeger, Sibei Yang, Wenping Wang, Jingyi Yu, Xuming He, Yuexin Ma

arXiv:2402.13853v2cs.ROcs.AIcs.CV

TL;DR

Dexterous grasping methods often lack real-world data that captures human behavioral patterns, while synthetic data and engineered rewards do not fully reproduce human-like interaction. RealDex supplies real teleoperated motions and multimodal observations, and its MLLM-assisted generation framework achieves strong performance across datasets and on a real robot. The authors nevertheless identify collision handling and environmental conditioning as remaining limitations.

  • Problem

    Existing grasping methods and synthetic datasets do not adequately capture human behavioral patterns or bridge training with real-world dexterous-hand applications.

    Source: §1 · pp. 1–2

  • Method

    RealDex combines teleoperated human-like grasping data with a two-stage point-cloud-based grasp-pose and motion-generation method using an MLLM Selection Module.

    Source: §1 · p. 2

  • Results

    Experiments on RealDex and other open datasets, together with real-robot testing, show that the method produces human-like practicable grasping motions and outperforms alternatives.

    Source: §1 · p. 2, §5.2 · p. 7, §5.4 · p. 7

  • Takeaways & Limitations

    RealDex supports training and evaluating dexterous hands for human-like perception, cognition, and manipulation in real-world settings.

    Source: §1 · p. 2, §6 · p. 7

    Pose generation can produce hand-object intersections requiring test-time optimization, and motion generation is guided only by the target pose rather than object and environmental conditions.

    Source: D Limitation · p. 12

Abstract

from arXiv · show

In this paper, we introduce RealDex, a pioneering dataset capturing authentic dexterous hand grasping motions infused with human behavioral patterns, enriched by multi-view and multimodal visual data. Utilizing a teleoperation system, we seamlessly synchronize human-robot hand poses in real time. This collection of human-like motions is crucial for training dexterous hands to mimic human movements more naturally and precisely. RealDex holds immense promise in advancing humanoid robot for automated perception, cognition, and manipulation in real-world scenarios. Moreover, we introduce a cutting-edge dexterous grasping motion generation framework, which aligns with human experience and enhances real-world applicability through effectively utilizing Multimodal Large Language Models. Extensive experiments have demonstrated the superior performance of our method on RealDex and other open datasets. The complete dataset and code will be made available upon the publication of this work.

1 Introduction

Dexterous grasping requires human-like behavior that existing planning, reinforcement-learning, and synthetic-data approaches do not fully capture. RealDex addresses this gap with real human-driven motion data and an MLLM-assisted generation method that performs strongly across datasets.

Motivation

  • Human-like grasping remains difficult because RL rewards and synthetic data do not adequately model human behavioral habits or real-world conditions.Existing approaches can produce physically valid grasps but may fail to reproduce choices such as grasping a cup by its handle.Source: §1 · pp. 1–2

RealDex Dataset

  • RealDex records human-driven dexterous grasping through real-time teleoperation, covering 52 objects, 2.6k motion sequences, and approximately 955k multimodal visual frames.The dataset combines synchronized human-robot poses with multi-view images and point clouds.Source: §1 · p. 2

Method

  • The proposed two-stage method generates grasp poses and synthesizes motions from real object point clouds, using an MLLM Selection Module to choose natural and physically plausible poses.The method is designed to exploit human-like motion patterns learned from RealDex and generalize to unseen objects.Source: §1 · p. 2

Results

  • Quantitative and qualitative experiments on RealDex and other open datasets show that the method obviously outperforms alternatives for human-like, practicable dexterous grasping motions.The evaluation also includes tests on a real robot hand.Source: §1 · p. 2

2 Related Work

Prior dexterous-grasping datasets and generation methods rely heavily on synthesized static poses, while human-object datasets use incompatible hand representations. These limitations motivate real dynamic motion data and supervised learning from accurate human-like robotic trajectories.

2.1 Dataset for Dexterous Hand Grasping

  • Existing dexterous-grasping datasets are primarily optimization-generated and static, limiting pose diversity, real-world transfer, and modeling of human-like behavior.Planner-based methods restrict grasps to eigengrasp spaces, while simulation-based datasets remain disconnected from real robotic hand data.Source: §2.1 · p. 2
  • Human-object interaction datasets provide relevant behavioral data, but differences between human and robotic hand representations hinder their direct use for dexterous manipulation.This representation mismatch separates human grasping observations from robot-hand control requirements.Source: §2.1 · p. 2

2.2 Dexterous Grasping Generation

  • Analytical and reinforcement-learning methods struggle with high-DoF hands, complex objects, exact geometric inputs, and reward functions that fail to model human motion patterns.Raw image or point-cloud inputs improve real-world applicability, but the human-like behavior problem remains.Source: §2.2 · pp. 2–3
  • Because RealDex supplies accurate ground-truth human-like dexterous motions, supervised methods are presented as a better fit for guiding grasping-generation learning.The related-work discussion connects RealDex’s data properties to human-motion-generation approaches.Source: §2.2 · p. 3

3 RealDex Dataset

RealDex is a real-world dataset of human-driven dexterous grasping that combines dynamic robot poses with multi-view visual data. Its calibrated capture system and three distinguishing properties target the realism, transfer, and behavioral limitations of prior datasets.

3 RealDex Dataset

  • RealDex contains 52 varied objects, 2.6k grasping sequences, and approximately 955k visual frames, with object-disjoint training, validation, and test splits.The splits contain 2114 sequences for 40 objects, 245 for 6 objects, and 271 for 6 objects.Source: §3 · p. 3

3.1 Hardware System Setup

  • The capture platform combines four RGB-D cameras, a UR10e arm, a 24-DoF Shadow Hand, and glove-based teleoperation for real-time human-like control.The cameras operate at 15Hz, while the hand has over 100 sensors operating at up to 1kHz.Source: §3.1 · p. 3

3.2 Calibration and Synchronization

  • Calibration and synchronization align camera views, robot geometry, and timestamps using AprilTag initialization, ICP refinement, hand-eye calibration, and ROS-based temporal matching.Camera sampling is staggered to reduce infrared interference, and robot-camera alignment is refined against point clouds.Source: §3.2 · pp. 3–4

3.3 Characteristics

  • Unlike prior datasets, RealDex captures dynamic human-like grasping motions directly through human teleoperation rather than static poses produced by physical optimization.This design adds behavioral patterns that prior synthesized datasets lack.Source: §3.3 · p. 4
  • RealDex records real robotic dexterous poses for every frame, reducing the gap between trained models and practical robot execution.Prior synthesized poses can create a performance gap when transferred to real applications.Source: §3.3 · p. 4
  • Its multi-view and multimodal visual data support vision-based dexterous-grasping research under varied input configurations.The dataset provides genuine visual data that previous datasets largely lack.Source: §3.1 · p. 3, §3.3 · p. 4

4 Methodology

The framework generates human-aligned dexterous grasping motions from object point clouds by combining candidate pose generation, MLLM-based selection, and autoregressive motion synthesis.

4 Methodology

  • The framework uses cVAE-generated grasp candidates, MLLM selection of natural and physically plausible poses, and MotionNet to synthesize executable motion sequences.Candidate poses are optimized for contact feasibility, filtered for stability, and selected according to naturalness, physical plausibility, human-likeness, and preference.Source: §4 · p. 5, §4.2 · p. 5, Figure 4 · p. 5

4.1 Problem Formulation

  • The problem is to generate preferred grasp poses and corresponding hand-motion sequences from an observed object point cloud.A dexterous hand pose combines 22 joint angles with a global 6D hand pose, forming a 28-dimensional representation.Source: §4.1 · p. 5

4.2 Grasping Pose Generation

  • The cVAE samples diverse candidate poses from a latent grasping space, aligns them with predicted contacts, filters unstable grasps, and retains K target poses from N candidates.The MLLM then chooses poses that better reflect human grasping preferences and priors.Source: §4.2 · p. 5

4.3 Pose-guided Hand Motion Synthesis

  • MotionNet autoregressively predicts future hand motion from recent poses, joint features, hand-object features, and target guidance toward the selected grasp.It uses self-attention to model joint relationships, a gating mechanism to encode motion phase, and pose changes to capture temporal dynamics.Source: §4.3 · p. 6

5 Experiments

Experiments compare RealDex-based grasping against adapted baselines and UniDexGrasp, evaluate pose and motion quality quantitatively, and demonstrate real-robot execution.

5.1 Comparison for Dexterous Grasping

  • Our method outperforms adapted SAGA and GOAL baselines in user-rated grasping motion generation on GRAB and RealDex.The study uses 40 users and scores humanoid characteristics, grasping stability, and hand-object interaction quality.Source: §5.1 · p. 6

5.2 Evaluation for Grasping Pose Generation

  • Across GRAB, DexGraspNet, and RealDex, our pose generation achieves the highest simulated stability and user acceptance, with MLLM selection improving performance.Ten grasp poses are generated per test object, and stability is measured using simulation displacement alongside intersection and penetration metrics.Source: Table 3 · p. 6, §5.2 · p. 7

5.3 Evaluation for Motion Synthesis

  • Motion synthesis remains superior with the ground-truth goal pose supplied, reflecting the method’s modeling of spatial relationships among joints.MPJPE, AVE, hand-mesh vertex offset, and minimum distance to the object quantify motion-sequence quality against ground truth.Source: §5.3 · p. 7, Table 4 · p. 7

5.4 Test on Real Robot

  • Real-robot execution converts qualified simulated trajectories into stamped Shadow Hand trajectories, producing natural, stable grasping over approximately 20 seconds.The hand and object maintain a relatively static relationship throughout execution, and the process is visualized in Figures 5 and 6.Source: §5.4 · p. 7, Figure 5 · p. 7, Figure 6 · p. 7

6 Conclusion

The conclusion presents RealDex as a real-world dataset of human-behavior-embedded dexterous grasping motions and introduces an MLLM-based motion-generation method.

6 Conclusion

  • RealDex provides genuine dexterous grasping motions with human behaviors and rich multi-view, multimodal visual data for training human-like robotic movements.The paper positions the dataset for humanoid-robot perception, cognition, and manipulation in real-world settings.Source: §6 · p. 7
  • The proposed MLLM-based method shows strong performance and practical value through extensive experiments and real-robot testing.Source: §6 · p. 7

A Data Process

The data-processing pipeline denoises RGB-D point clouds, merges viewpoints, and labels object poses with ICP followed by human inspection and adjustment.

A.1 Point Cloud Denoising

  • Point-cloud denoising removes statistical outliers using each point’s average distance to its 20 nearest neighbors before merging viewpoints into a unified cloud.Points deviating by more than two standard deviations from the mean are excluded to reduce RGB-D noise.Source: §A.1 · p. 10

A.2 Object Pose Labeling

  • Object 6D poses are initialized manually, propagated through subsequent frames with ICP, and finally inspected and fine-tuned by a human annotator.Most sequences require only one annotation pass.Source: §A.2 · p. 10

A.3 Visualization of Dataset

The dataset visualization shows annotated object and hand motion, grasping meshes, and sampled frames from dexterous-hand motion sequences.

A.3 Visualization of Dataset

  • Figure 7 visualizes annotated object motion together with dexterous hand motion in the dataset.Source: §A.3 · p. 10, Figure 7 · p. 10
  • Figure 8 presents eight sampled frames of the robotic hand mesh with its arm during a RealDex grasping motion.Source: §A.3 · p. 10

B.1 Training

The framework trains grasp pose generation and motion synthesis with ground-truth supervision, then uses MLLM scoring to select target grasps for sequential motion generation.

B.1 Training

  • Ground-truth annotations supervise separate training stages for grasp pose generation and motion synthesis.Source: §B.1 · p. 10
  • Pose generation reconstructs hand poses and contact maps from object and hand features using a cVAE, supervised by four weighted losses.The losses include distribution regularization, pose reconstruction, contact-map binary cross entropy, and hand-mesh Chamfer distance.Source: §B.1 · p. 10

B.2 MLLM Selection

B.3 Inference

  • At inference, candidate grasps from unseen object point clouds are refined, MLLM-selected, and converted into motions from the mean pose until time or distance termination.Source: §B.3 · p. 11

C More Results and Discussions

The results visualize pose generation, target-guided motion synthesis, and MLLM-based grasp selection as complementary stages of the framework.

C.1 Pose generation

  • Figure 11 shows automatically computed hand configurations for objects with different shapes.Source: §C.1 · p. 11

C.2 Motion synthesis

  • The motion synthesis module generates progressive hand-motion steps from a mean initial pose toward a target pose.The mean pose sets all joint angles to zero, while hand translation is initialized from the dataset’s average location.Source: §C.2 · p. 11, Figure 10 · p. 11

C.3 MLLM selection

  • The MLLM selection module scores rendered hand-and-object grasp images and provides detailed explanations for its choices.Source: §C.3 · pp. 11–12

D Limitation

The authors identify collision artifacts in generated poses and limited environmental conditioning during motion generation as remaining weaknesses.

D Limitation

  • Generated hand poses can intersect objects and require test-time optimization, while motion generation uses only the target pose rather than object and environmental conditions.The authors suggest collision penalty loss during training as one possible improvement.Source: D Limitation · p. 12
  • Figure 11 visualizes generated grasps but does not establish a quantitative limitation or comparison.Source: Figure 11 · p. 13

Related papers

Loading 2402.13853v2…