PaperlayerView exact revision on arXiv

Source-linked AI summary

CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements

Hongxiang Zhao, Mutian Xu, Zeyu Jin, Yiming Hao, Shuguang Cui, Xiaoguang Han

arXiv:2609.07498v1cs.ROcs.CV

TL;DR

Existing human-to-gripper transfer methods are largely limited to simple or planar manipulation, motivating a data-driven approach for complex spatial movements. CosmoH2G builds a large paired hand–gripper dataset and uses staged keyframe-to-sequence generation with translation optimization. The resulting system enables stable and precise transfer in simulation and real-robot experiments, while remaining open-loop and lacking real-time error correction and explicit collision avoidance.

  • Problem

    Existing hand-to-gripper methods are largely confined to simple or planar tasks and fail to facilitate complex spatial movements involving rotations, flips, and intricate trajectories.

  • Method

    CosmoH2G collects complexity-focused paired hand–gripper demonstrations and transfers them with a two-stage framework that predicts endpoint keyframes before continuous actions while optimizing translation.

  • Results

    The dataset contains 6,189 episodes across 1,254 objects, and experiments demonstrate precise and stable transfer of complex spatial manipulations.

  • Takeaways & Limitations

    The dataset and framework provide a foundation for data-driven cross-embodiment learning and complex robotic manipulation.

  • Takeaways & Limitations

    The current framework operates open-loop, lacking real-time error correction and explicit collision avoidance.

Abstract

from arXiv · show

Transferring human hand demonstrations to robotic grippers has recently emerged as a cost-effective solution for robot learning. However, existing methods are largely confined to simple, planar tasks and fail to handle complex spatial movements (e.g., intricate trajectories involving rotations or flips) that are essential for robot manipulation. Motivated by this gap, we adopt an implicit, data-driven approach guided by fine-grained hand-pose motions. To this end, we introduce a scalable acquisition pipeline to collect hand-gripper paired demonstrations, governed by a rigorous protocol that prioritizes motion complexity and leverages a handheld gripper for seamless action mimicry. This yields a large-scale paired dataset comprising 6,189 episodes across 1,254 unique objects, exhibiting significantly higher spatial complexity than existing benchmarks. However, learning such complex mappings remains challenging. We observe that naive end-to-end generation of full gripper pose sequences is insufficient, as minor trajectory deviations compound rapidly under intricate dynamics. To address this, we propose a two-stage framework: Stage I predicts sparse gripper keyframes (initial and terminal) to simplify the mapping objective, while Stage II generates the full continuous action sequence conditioned on these keyframes. Furthermore, to mitigate cumulative drift, we keep the gripper's orientation being learned while post-optimizing its translation based on the grasping heuristic and kinematic consistency. In both simulation and real-robot experiments, our framework enables stable and precise hand-to-gripper transfer of complex spatial manipulations, significantly outperforming traditional baselines. Project page: https://cosmoh2g.github.io.

1 Introduction

CosmoH2G targets the gap between human demonstrations and robotic grippers in complex spatial manipulation, where existing rule-based and trajectory-based methods are limited. It introduces paired data collection and a two-stage data-driven transfer framework, with improved performance reported in simulation and real-robot experiments.

  • Existing hand-to-gripper methods largely handle simple or planar tasks but struggle with trajectories involving substantial rotation, flipping, and 3D traversal.
  • Rule-based retargeting often fails under complex movements because hand-crafted mappings cannot bridge large human-hand and gripper embodiment differences.
  • CosmoH2G shifts from hand-crafted rules to implicit, data-driven mapping guided by fine-grained human hand-pose motions.
  • Its scalable pipeline collects paired demonstrations using complexity-focused protocols, varied grasp gestures, a handheld gripper, and RGB-D-based 3D action extraction.
  • The two-stage framework predicts starting and terminal gripper poses before generating continuous actions, while separating learned orientation from optimized translation to reduce drift.
  • Simulation and real-robot experiments report consistent improvements over optimization-based and learning-based methods on complex spatial movements.

2 Related Works

Prior work uses teleoperation, object-centric trajectories, or hand-crafted mappings to transfer human demonstrations, but these approaches remain constrained for complex spatial manipulation.

  • Object-centric trajectory methods remain limited because 2D trajectories miss orientation changes and 3D flow tracking degrades under complex-motion drift.
  • Paired-dataset approaches exist, but cumbersome teleoperation can restrict their demonstrations to simple tasks in clean environments.
  • Teleoperation produces robot demonstrations directly but is labor-intensive and difficult to execute for intricate tasks.
  • Earlier hand-to-gripper methods often rely on fixed hand-crafted mappings that cannot accommodate diverse human grasping patterns.

3 Dataset

The CosmoH2G dataset combines scalable RGB-D processing with complexity-focused paired demonstrations, yielding broad object and motion diversity with closely aligned hand–gripper trajectories.

  • Dataset construction: The dataset uses handheld grippers, fixed-view paired RGB-D videos, and extracted 3D motion data for cross-embodiment learning.
  • Dataset construction: Its demonstrations include diverse natural grasp types and complex in-hand orientation transformations beyond simple translation.
  • 3D motion extraction: The extraction pipeline reconstructs objects and hands, registers scenes, and tracks the handheld gripper to obtain 6-DoF pose sequences.
  • Dataset pairing: Pairs are retained only when their 3D trajectory similarity exceeds 0.9, producing a mean similarity of 0.957 and median similarity of 0.958.
  • Dataset statistics: 6,189 episodes across 1,254 unique objects provide substantially greater orientation variation than prior datasets dominated by translational actions.

4 Method

CosmoH2G learns hand-to-gripper action transfer from paired 3D data through staged keyframe and sequence generation. It generates endpoint poses first, predicts orientation sequences conditionally, and derives and optimizes translation for physically consistent motion.

  • Framework overview: The method learns a purely data-driven transfer function from paired demonstrations without predefined alignments.
  • Stage I: Stage I independently predicts starting and terminal 6-DOF gripper poses from hand points, object points, and contact maps.
  • Stage I: Stage I represents gripper actions with three keypoints, allowing conditioning and generated actions to share a common 3D-point space.
  • Stage II: Stage II predicts the gripper orientation sequence from hand-point sequences, object points, contact maps, and Stage-I endpoint generations.
  • Framework overview: The framework combines generated orientation with optimized translation to mitigate cumulative drift in full action sequences.
  • Position optimization: Gripper positions are initialized from the hand trajectory using contact-region alignment, then refined for grasp stability, smoothness, and kinematic feasibility.

5 Evaluation

The evaluation tests CosmoH2G on unseen objects and hand motions in simulation and real-robot settings, comparing it with optimization- and learning-based trajectory methods. Results show stronger performance for complex spatial manipulation, while ablations support the two-stage design, model conditions, keypoint action representation, and post-optimization.

  • Evaluation Setup: The test set contains 186 cases across 40 objects unseen during training, evaluated in simulation and on a Galaxea R1 Lite real robot.The experiments focus on pick-and-place tasks with unseen hand motions.
  • Evaluation Metrics: Evaluation measures grasp orientation accuracy, trajectory similarity, success rate, and target orientation placement accuracy.Angular deviation evaluates grasp and target orientation, while trajectory similarity and success rate assess motion adherence and task completion.
  • Quantitative Comparison: CosmoH2G outperforms optimization-based and learning-based trajectory baselines in both simulation and real-robot experiments.The baselines include MimicFunc, 3DFlowAction, Im2Flow2Act, and Track2Act.
  • Quantitative Comparison: Real-robot performance is slightly lower because large rotations can cause object slipping, while premature movement from the table can cause friction-related drops.These physical effects produce orientation shifts, drops, or unsuccessful lifts.
  • Ablation Results: All evaluated single-stage architectures suffer from insufficient accuracy, supporting the necessity of the two-stage formulation for complex transfer.The ablation evaluates Diffusion Transformer, Diffusion Policy, and ACT against the two-stage framework.
  • Ablation Results: Removing the contact map causes a slight performance drop, while adding the terminal action is essential for achieving the highest precision.The contact map provides geometric cues, and the terminal action complements the starting action’s kinematic initialization prior.
  • Ablation Results: Three gripper keypoints improve transfer because they explicitly encode spatial contact regions and gripper orientation.This representation is compared with homogeneous transformation matrices and quaternion-translation pairs.
  • Ablation Results: End-to-end pose generation suffers from hand-gripper alignment errors, while purely geometric computation produces inaccurate orientations and difficult high-dimensional optimization.These findings motivate learning orientation while refining translation through post-optimization.

6 Conclusions

CosmoH2G combines a complexity-focused paired hand-gripper dataset with two-stage learning for complex spatial manipulation. The resulting system enables precise and stable hand-to-gripper transfer across such movements.

  • Contributions: The dataset contains 6,189 paired episodes across 1,254 unique objects and emphasizes greater manipulation complexity than existing benchmarks.Its acquisition pipeline uses a rigorous protocol that prioritizes motion complexity.
  • Method: The two-stage framework predicts keyframe poses before generating continuous action sequences to simplify hand-to-gripper transfer.This design separates sparse pose prediction from full-sequence generation.
  • Results: Experiments demonstrate precise and stable transfer of complex spatial manipulations.The conclusion reports this outcome for the proposed approach without specifying a separate metric value.

7 Limitations and Future Work

The framework handles spatial transfer but remains open-loop, and future work targets scaling paired data, improving generalizability, and adding feedback and collision-aware planning.

  • Future Work: Unified one-stage direct transfer may become increasingly viable as paired data scales, while the current framework remains a two-stage design.The authors plan to expand the dataset and refine the architecture for better generalizability.
  • Limitations: The current framework operates open-loop, lacking real-time error correction and explicit collision avoidance.The authors identify closed-loop feedback and collision-aware motion planning as key future directions.

A Implementation Details

The implementation computes contact-aware gripper positions, optimizes boundary poses to align with hand contact regions while avoiding penetration, and applies IK refinement for physical feasibility.

  • Contact Map: The pipeline builds an object-surface contact map by converting nearest hand-point distances into thresholded soft contact likelihoods.The likelihood equals 1 at direct contact and decays to 0 beyond a predefined interaction radius.
  • Position Initialization: Hand and gripper contact-region centers are aligned to initialize the gripper position sequence from the reconstructed hand motion.The hand center is the centroid of hand points near the object; the gripper center is the midpoint between left and right finger keypoints.
  • Grasping Optimization: Boundary-frame gripper positions are optimized to approach contacted object points, limit corrections, and prevent gripper penetration into the object.The objective combines contact alignment, L2 position regularization, and a penetration penalty.
  • IK Refinement: Each optimized 6-DoF pose is refined with inverse kinematics to reduce end-effector residuals and joint-limit violations while remaining near initialization.The IK objective regularizes the pose toward the previous-stage estimate.

A.5 Real-robot Setup.

The real-robot evaluation uses a dual-arm Galaxea platform with binocular vision and standardized simulation and baseline comparisons for complex manipulation.

  • Sensing: A binocular camera captures the workspace and human demonstrations from a fixed viewpoint, while object pose is estimated during evaluation.The setup uses a position anchor and FoundationPose++ for object orientation estimation.
  • Hardware Setup: The real-robot setup uses a Galaxea R1 Lite with two 7-DoF A1X arms and parallel-jaw G1 grippers.The tabletop defines the world frame, with the robot base centered at its origin and the arms symmetrically arranged.
  • Simulation Physics: Simulation uses a common physical configuration with approximately 1000 kg/m3 uniform density and a 0.15 contact-friction coefficient.Object centroids are anchored at mesh geometric centers.
  • Baselines: The study compares against MimicFunc and 3DFlowAction optimization baselines and retrains Track2Act with 3D hand-point sequences.Track2Act remains below CosmoH2G despite receiving enriched 3D information.

C Details for the Computed Variant

The Computed variant derives gripper motion from object and contact geometry, initializes boundary poses with grasp proposals, and optimizes smooth, kinematically feasible sequences.

  • Object-Pose Induction: The variant uses the object as an intermediate geometric anchor because direct hand-pose orientation targets are unreliable across different embodiments.The induced object 6D trajectory becomes the target for gripper-pose optimization.
  • Object-Pose Induction: A stable-grasp prior propagates the initial hand-object relative pose to induce the object trajectory from the reconstructed hand mesh.The hand contact pose at each frame is combined with the initial relative transform.
  • Boundary Initialization: Boundary poses are initialized from GraspNet candidates scored by contact consistency, orientation alignment, and grasp quality.The highest-scoring candidate is selected at the starting and terminal frames.
  • Full-Pose Optimization: Intermediate orientations use SLERP, while full-pose optimization enforces angular smoothness, object-trajectory tracking, and IK feasibility.The tracking term aligns the object pose induced by the gripper with the target object trajectory.
  • Failure Cases: The Computed variant fails on complex motions such as flipping because rough SLERP initialization and joint position-orientation optimization create difficult, ambiguous searches.These issues can produce discontinuous or unnatural motions, motivating direct learning of orientation trajectories.

D Evaluation Metrics

The evaluation measures trajectory similarity, position-level placement success, and orientation precision, with additional checks for viewpoint, demonstrator, and paired-data consistency.

  • Trajectory Similarity: Trajectory similarity compares ground-truth and executed 3D position paths using length-normalized FastDTW and a tolerance-based score.The tolerance is set to 20% of table width, and deviations beyond it receive zero similarity.
  • Success Rate: Placement success counts trials whose final position is within 10% of desktop width from the target.The success rate is the proportion of successful trials among all placement trials.
  • Orientation Accuracy: Target Orientation Placement Accuracy measures the mean angular error in degrees between final and desired object orientations.Lower angular error indicates higher orientation precision, with real-robot orientations estimated using FoundationPose++.
  • Generalization: Performance remains stable across simulation camera viewpoints, while generalization degrades for children’s hands outside the training scale distribution.The viewpoint result reflects 3D inputs, whereas the demonstrator result identifies hand scale as a boundary.
  • Data Quality: Retained hand-UMI pairs have similarity scores concentrated above the 0.9 retention threshold.The reported mean is 0.957 and the median is 0.958.

E Generalization Analysis

CosmoH2G generalizes across adult hand scales and evaluated camera viewpoints, while performance drops for children’s hands outside the training distribution.

  • Demonstrators: 83.33% and 91.67% SR across adult male and female hands, respectively, remain comparable to 83.87% on the original test set.Children’s hands reduce SR to 33.33% because their scales fall outside the training distribution.
  • Demonstrators: 33.33% SR on children’s hands marks the boundary of hand-scale generalization beyond the training distribution.
  • Camera Viewpoints: 75% to 90% SR across four unseen camera viewpoints indicates stable performance from first-person, egocentric top-down, third-person, and exocentric top-down views.The evaluated viewpoints cover 20 unseen cases.
Loading 2609.07498v1…