PaperlayerView exact revision on arXiv

Source-linked AI summary

ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning

Kailin Li, Puhao Li, Tengyu Liu, Yuyang Li, Siyuan Huang

arXiv:2503.21860v1cs.ROcs.CV

TL;DR

Complex bimanual manipulation remains difficult to transfer precisely to dexterous robotic hands because of morphological differences, error accumulation, and high-dimensional actions. ManipTrans uses two-stage trajectory imitation and residual fine-tuning, outperforming prior methods while producing the 3.3K-episode DexManipNet dataset.

  • Problem

    Complex bimanual manipulation transfer remains limited by hand morphology differences, accumulated motion errors, and high-dimensional action spaces, leaving tasks such as pen capping and bottle unscrewing largely unexplored.

  • Method

    ManipTrans pre-trains a generalist hand-motion imitator and then fine-tunes a residual module to satisfy object-interaction constraints in simulation.

  • Results

    MANIPTRANS outperforms all baselines across multiple metrics on single- and bimanual tasks, while DEXMANIPNET contains 3.3K robotic manipulation episodes.

  • Takeaways & Limitations

    ManipTrans provides an extensible resource and transfer framework for dexterous-hand policy training across diverse manipulation tasks.

  • Takeaways & Limitations

    The study primarily uses a simulated 12-DoF Inspire Hand, whereas its real-world counterpart has 6 DoF.

Abstract

from arXiv · show

Human hands play a central role in interacting, motivating increasing research in dexterous robotic manipulation. Data-driven embodied AI algorithms demand precise, large-scale, human-like manipulation sequences, which are challenging to obtain with conventional reinforcement learning or real-world teleoperation. To address this, we introduce ManipTrans, a novel two-stage method for efficiently transferring human bimanual skills to dexterous robotic hands in simulation. ManipTrans first pre-trains a generalist trajectory imitator to mimic hand motion, then fine-tunes a specific residual module under interaction constraints, enabling efficient learning and accurate execution of complex bimanual tasks. Experiments show that ManipTrans surpasses state-of-the-art methods in success rate, fidelity, and efficiency. Leveraging ManipTrans, we transfer multiple hand-object datasets to robotic hands, creating DexManipNet, a large-scale dataset featuring previously unexplored tasks like pen capping and bottle unscrewing. DexManipNet comprises 3.3K episodes of robotic manipulation and is easily extensible, facilitating further policy training for dexterous hands and enabling real-world deployments.

1. Introduction

ManipTrans transfers human bimanual manipulation to dexterous robotic hands in simulation through two stages: trajectory imitation followed by interaction-constrained action fine-tuning. It addresses the scalability, embodiment, precision, and efficiency limitations of reinforcement learning, teleoperation, direct retargeting, and complex bimanual policy learning while supporting the construction of DexManipNet.

  • Motivation and challenges: Human-like dexterous bimanual manipulation is important for embodied AI, but acquiring precise, large-scale, human-like sequences remains challenging.Traditional reinforcement learning requires task-specific reward functions, while teleoperation is labor-intensive, costly, and embodiment-specific.
  • Motivation and challenges: Imitation-based transfer can produce naturalistic hand-object interactions by leveraging motion-capture datasets and hand pose estimation.However, morphological differences make direct pose retargeting suboptimal, MoCap errors can accumulate, and bimanual actions create a high-dimensional learning problem.
  • Method: MANIPTRANS uses a two-stage framework that first pre-trains trajectory imitation for hand motion, then fine-tunes actions to satisfy interaction constraints.This design separates dynamic hand mimicry from physics-based object interaction and reduces action-space complexity.
  • Method: MANIPTRANS corrects arbitrary noisy hand MoCap data into physically plausible motion without predefined task stages or task-specific reward engineering.The method is validated across complex single- and bimanual manipulations, including articulated object handling.
  • Results and contributions: MANIPTRANS outperforms baseline methods in motion precision and transfer success rate while surpassing prior SOTA approaches in transfer efficiency.Cross-embodiment experiments across hands with different DoFs and morphologies show consistent performance with minimal additional effort.
  • Results and contributions: The framework enables precise transfer of human bimanual manipulation in simulation and supports construction of DEXMANIPNET, a large-scale dataset of novel bimanual tasks.The dataset is described as high-quality, precise, and compliant, and is intended to cover a wide array of manipulation tasks.

2. Related Works

Related work transfers human manipulation skills through imitation, object-trajectory-guided reinforcement learning, and quasi-physical simulation, but existing approaches face task-complexity and computational limitations. Dexterous hand datasets often neglect physical constraints, while residual learning is used to improve the efficiency and stability of reinforcement-learning control.

  • Dexterous Manipulation via Human Demonstration: Imitation learning transfers human manipulation skills to robots, with recent methods guiding reinforcement-learning policies using object trajectories.QuasiSim directly transfers reference hand motions to robotic hands through parameterized quasi-physical simulators, but these methods remain limited to simpler tasks and computationally intensive.
  • Dexterous Hand Datasets: MANO-based hand-object datasets commonly prioritize pose alignment with 2D images while neglecting physical constraints needed for robotic training.Teleoperation methods collect human-to-robot hand matching data, addressing a different route for acquiring robotic manipulation demonstrations.
  • Residual Learning: Residual policy learning incrementally refines action control to improve reinforcement-learning efficiency and stability in dexterous hand manipulation.Existing studies tailor residual strategies to specific tasks, including integrating user input or learning corrective actions from human demonstrations.

3. Method

ManipTrans transfers human bimanual hand–object trajectories to dexterous robotic hands through a two-stage framework: hand-only imitation followed by residual refinement under interaction constraints. The method models bimanual manipulation in simulation and expands residual learning with object and contact information for task-compliant execution.

  • Bimanual formulation: The bimanual formulation has left and right dexterous hands cooperatively manipulating two objects while replicating human hand behavior.The framework is illustrated with pen capping, where one hand holds the cap and the other grips the pen body.
  • Two-stage framework: ManipTrans first pre-trains a general hand trajectory imitation model, then fine-tunes a residual module to refine coarse actions for task compliance.The imitation model learns from hand motion, while the residual module adapts actions to task-specific physical constraints.
  • Hand imitation: The hand imitation reward combines wrist tracking, finger imitation, and smoothness terms to reproduce human motion while reducing morphological mismatch and jerky movements.Finger weights emphasize fingertips, particularly those of the thumb, index, and middle fingers; smoothness penalizes joint power from velocities and torques.
  • Hand imitation: Hand imitation is decoupled from object interaction, allowing training with hand-only datasets augmented through interpolation and mirroring.Reference state initialization and early termination are used for efficiency, with termination when hand keypoints deviate beyond a threshold.
  • Residual refinement: The residual module expands the state with object geometry, pose, velocities, center of mass, gravity, BPS shape features, and simulated contact forces.It samples a residual correction conditioned on the imitation action, adds it element-wise, clips the result to joint limits, and initializes residuals near zero to accelerate convergence.

4. Experiments

Experiments evaluate MANIPTRANS on quantitative and qualitative datasets using tracking-error and success-rate metrics, compare it with RL-combined and optimization-based baselines, and assess cross-embodiment transfer and real-world deployment. Results show superior precision and success, consistent performance across diverse hands, and challenging bimanual manipulation in hardware.

  • Datasets and Metrics: The evaluation uses OakInk-V2 for quantitative testing, filtered MoCap sequences for transfer assessment, and GRAB for qualitative evaluation.The selected MoCap set contains approximately 80 episodes after filtering sequences to 4–20 seconds at 60 fps and excluding deformable or oversized objects.
  • Datasets and Metrics: Success requires Er, Et, Ej, and Eft below 30°, 3 cm, 8 cm, and 6 cm, respectively, with bimanual failure if either hand misses a threshold.Er and Et measure rotation and translation errors, while Ej and Eft measure mean per-joint and per-fingertip position errors.
  • Comparison with Baselines: MANIPTRANS outperforms all RL-combined baselines across multiple metrics, achieving higher precision in both single- and bimanual tasks.The comparison evaluates RL-only exploration, retarget-only transfer, and Retarget + Residual methods; Retarget-Only is nearly infeasible and RL-Only sacrifices efficiency and precision.
  • Generalization and Qualitative Results: MANIPTRANS produces natural, precise motion on seldom-explored tasks and remains applicable across dexterous-hand embodiments.It is evaluated on Shadow, articulated MANO, Inspire, and Allegro hands with K = 22, 22, 12, and 16 DoFs, without changing network hyperparameters or reward weights.
  • Real-World Deployment: Real-world deployment demonstrates bimanual manipulation in which one hand stabilizes an object while the other performs precise motions such as opening a tiny toothpaste cap.The setup uses two 7-DoF Realman arms and upgraded Inspire Hands with tactile sensors, with fitting-based fingertip alignment and inverse-kinematics arm control.

5. Conclusion and Discussion

ManipTrans efficiently transfers human manipulation skills to dexterous robotic hands by decoupling hand-motion imitation from object interaction through residual learning. It achieves high-fidelity motion, efficient training, and strong motion precision, computational efficiency, and cross-embodiment adaptability.

  • ManipTrans is a two-stage framework for efficiently transferring human manipulation skills to dexterous robotic hands.The framework decouples hand-motion imitation from object interaction through residual learning.
  • Residual learning decouples hand-motion imitation from object interaction, overcoming morphological differences and complex task challenges.This design supports high-fidelity motions and efficient training.
  • Experiments show that ManipTrans surpasses state-of-the-art methods in motion precision and computational efficiency while exhibiting cross-embodiment adaptability.

via Residual Learning … A.2. Challenging Hand Embodiments

The appendix extends MANIPTRANS to articulated-object manipulation and challenging hand embodiments, showing successful imitation under accurate simulation modeling and after morphology-specific fingertip adaptations. It also documents supplementary evaluations, settings, dataset statistics, and policy-training details.

  • via Residual Learning: The appendix covers MANIPTRANS extensibility, noisy-condition robustness, time cost, method settings, DexManipNet statistics, and rearrangement-policy training details.These topics are organized across Appendices A–F.
  • A.1. Articulated Object Manipulation: MANIPTRANS is applied to the ARCTIC dataset, which contains approximately 10 articulated objects with precise bimanual manipulation trajectories.The evaluation targets single-object manipulation tasks involving articulated objects.
  • A.1. Articulated Object Manipulation: For articulated objects, MANIPTRANS represents object trajectories with transformations, velocities, articulated-part angles, and angular velocities.The articulated-object reward adds angle and angular-velocity differences while leaving the remaining pipeline unchanged.
  • A.1. Articulated Object Manipulation: Qualitative ARCTIC results show MANIPTRANS imitates human demonstrations and rotates articulated objects to desired target angles.The method’s extensibility depends on accurately modeling the articulated object’s physical properties in simulation.
  • A.2. Challenging Hand Embodiments: The Allegro Hand presents a challenging embodiment with K = 16 degrees of freedom, only four fingers, and approximately twice the size of a human hand.These morphological discrepancies substantially complicate transferring human demonstrations.
  • A.1. Articulated Object Manipulation: Figure 8 shows the two hands collaboratively close a book and place it stably on a table during articulated-object manipulation.The qualitative result accompanies the ARCTIC extension.
  • A.2. Challenging Hand Embodiments: Figure 9 shows that the transferred Allegro Hand motion remains stable and natural despite its four-finger structure and significantly larger size.This demonstrates successful application of MANIPTRANS to the challenging hand model.
  • A.2. Challenging Hand Embodiments: MANIPTRANS adapts to the Allegro Hand by mapping the pinky and ring fingers to one fingertip and relaxing ϵfinger to 8 cm.These changes accommodate the hand’s morphology and larger dimensions.

A.3. Discussion on the Extension

The extension experiments show that ManipTrans accommodates morphological differences across robotic hand embodiments and supports diverse single-hand and bimanual manipulation tasks. Although DexManipNet uses the Inspire Hand with FAVOR and OakInk-V2 data, the pipeline can transfer all referenced MoCap datasets to other robotic hands.

  • Extension experiments: ManipTrans accommodates morphological differences across hand embodiments in extension experiments.Successful transfers are included in DexManipNet, while dataset verification tests ManipTrans on subsets to assess generalizability.
  • Extension experiments: The supported task range includes single-hand manipulation, bimanual articulated-object manipulation, and bimanual two-object manipulation.These task categories are reported as part of the extension results.
  • Dataset setup: FAVOR and OakInk-V2 provide large, diverse task types, while the Inspire Hand offers high dexterity, stability, cost-effectiveness, and extensive prior use.This combination motivated the setup used to collect DexManipNet.
  • Dataset extensibility: ManipTrans can transfer all aforementioned MoCap datasets to other robotic hands despite DexManipNet’s collection setup using the Inspire Hand.The pipeline is described as fully adaptable across robotic hand embodiments.

B. Robustness Evaluation

The robustness evaluation tests whether MANIPTRANS can transfer noisy real-world data into stable robotic motions in simulation. MANIPTRANS maintains acceptable performance with noise levels up to 1.5 cm, supporting potential real-world scaling from hand pose estimation and web video data.

  • Evaluation motivation: MoCap data and model-based pose estimation results often contain noise, motivating tests of stable robotic motion transfer in simulation.The evaluation examines whether noisy real-world data can be reliably transferred by MANIPTRANS.
  • Evaluation motivation: MANIPTRANS uses general-purpose transfer without task-specific reward functions, so noisy object trajectories may destabilize rollouts.The passage contrasts this design with a task-specific twisting reward proposed for lip-twisting.
  • Results: 1.5 cm: MANIPTRANS maintains acceptable performance when the noise level reaches up to 1.5 cm.The results are presented in Table 4, which adds Gaussian noise N(0, σ2) to target hand-joint poses.
  • Implications: The robustness findings highlight MANIPTRANS’s potential for real-world scaling, particularly with hand pose estimation and web video data.Web video data may implicitly contain a vast array of dexterous manipulation skills.

C. Time Cost Analysis · D. Details of MANIPTRANS Settings · D.1. Correspondence Between Human Hand and Dexterous Hand

The paper evaluates MANIPTRANS’s training-time efficiency against QuasiSim and alternative training settings, while detailing hand-keypoint correspondence and MANO simplification for dexterous transfer.

  • C. Time Cost Analysis: QuasiSim divides transfer into three stages, each requiring approximately 10-20 hours for a 60frame trajectory.MANIPTRANS likewise uses a pre-trained hand imitation module and a residual refinement module tailored to physical dynamics.
  • D. Details of MANIPTRANS Settings: MANIPTRANS combines pre-trained hand imitation with residual refinement tailored to physical dynamics in its multi-stage framework.These modules support the method’s transfer process described in the time-cost analysis.
  • C. Time Cost Analysis: The comparison evaluates RL-Only, Retarget + Residual Learning, and MANIPTRANS on QuasiSim’s “rotating a mouse” demo data.The three settings’ success-rate curves are shown in Fig. 10.
  • C. Time Cost Analysis: Retarget + Residual achieves performance comparable to MANIPTRANS on “rotating a mouse” but requires slightly more time to converge.This result comes from the success-rate curves in Fig. 10.
  • C. Time Cost Analysis: 20 minutes is sufficient for RL-Only to produce acceptable motions, despite its suboptimal performance relative to the other methods.The result indicates that the reward design accelerates training and facilitates faster convergence.
  • D.1. Correspondence Between Human Hand and Dexterous Hand: Human fingertip keypoints use the midpoint of three tip anchors, whereas dexterous-hand fingertips use maximum-curvature points along each finger pad’s central axis.The dexterous-hand points are selected because they are most likely to contact objects.
  • D.1. Correspondence Between Human Hand and Dexterous Hand: 45-DoF MANO is reduced to a 22-DoF articulated model by constraining certain DoFs and fixing hand collision meshes.This reduces the exploration space faced by RL-based policies.

D.2. Details of Training Parameters · D.3. Details of Simulation Parameters

The appendix specifies MANIPTRANS’s reward-function settings and simulation physical properties. Finger-reward weights reflect contact frequency, while simulation parameters account for object density, measured masses, domain randomization, and the absence of elastic fingertip skin.

  • D.2. Details of Training Parameters: Finger-reward weights prioritize the thumb, index, and middle fingers because they interact with objects more frequently during grasping and manipulation.Weights are assigned according to contact frequency, and missing fingers or joints are assigned zero-valued parameters.
  • D.2. Details of Training Parameters: Missing fingers or joints are handled by setting their corresponding reward parameters to zero.The implementation gives this example for dexterous hands with different morphologies, such as the Inspire Hand.
  • D.2. Details of Training Parameters: The finger-reward hyperparameters balance each finger’s importance through a weight wf and decay rate λf.Each table cell contains separate values for the proximal, intermediate, distal, and tip joints.
  • D.2. Details of Training Parameters: 1 is used for both contact-reward parameters, wc and λc.The setting applies to the contact reward rt in Eq. (2) of the main paper.
  • D.3. Details of Simulation Parameters: 200 kg/m3 is assigned as the constant density for remaining objects whose masses were not directly measured.The value approximates the average density of low-fillrate 3D-printed models; some OakInk-V2 object masses were measured with the dataset authors.
  • D.3. Details of Simulation Parameters: Domain randomization is used during training alongside configured physical constants in the Isaac Gym environment.The passage emphasizes that physical-property configuration significantly influences transfer success rate.
  • D.3. Details of Simulation Parameters: Elastic human fingertip skin deforms slightly during grasping, improving surface contact and generating suitable friction, unlike dexterous robotic hands.The appendix motivates simulation assumptions that compensate for this missing behavior in robotic hands.
  • D.3. Details of Simulation Parameters: Slight fingertip penetration is permitted by prior kinematics-based grasp-generation methods to improve interaction stability.This practice is presented as compensation for robotic hands’ lack of elastic fingertip-skin deformation.

E. DEXMANIPNET Statistics

DexManipNet is presented as a large-scale bimanual manipulation dataset with directly transferred human trajectories and 3.3K trajectories spanning 61 tasks. Its simulation pipeline stabilizes object geometry and smooths policy rollouts with minimal tracking impact.

  • No prior work had collected a large-scale bimanual manipulation dataset whose trajectories were directly transferred from real human demonstrations without teleportation.
  • 3.3K diverse manipulation trajectories span 61 distinct tasks in DexManipNet.The task list is detailed in Table 6.
  • Object meshes are fixed to a watertight state, with object sizes sometimes adjusted to improve object-object interactions during simulation.The pipeline uses ManifoldPlus for watertight meshes and may adjust sizes for interactions such as between a bottle cap and body.
  • A first-order low-pass filter with α = 0.4 reduces rollout jitter while minimally affecting tracking accuracy.Sample policy-generated simulation trajectories are provided on the project website.

F. Details of Rearrangement Policy Learning

The rearrangement policies are benchmarked on moving a bottle to unseen goal positions, emphasizing stable grasping and smooth placement. Four imitation-learning methods use state-and-goal observations and PD-controlled joint-angle and wrist-pose actions, with success defined by a 10 cm distance threshold.

  • Benchmark Setup: Four data-driven imitation-learning methods benchmark moving a bottle to a goal position: IBC, BET, and Diffusion Policy with UNet- and Transformer-based architectures.The supplied passage states four methods but names IBC, BET, and Diffusion Policy; policies train for 500 epochs with Adam at 1 × 10−4.
  • Training Configuration: Policies are trained for 500 epochs using Adam with a learning rate of 1 × 10−4, while other hyperparameters remain at default settings.This training configuration is shared across the evaluated policies.
  • Observation and Action Spaces: Observations include object state, hand-wrist state, joint angles, and goal poses for both the object and hand wrist.The observation definitions include object and wrist positions with velocities, hand joint angles, and object and wrist goal poses.
  • Observation and Action Spaces: Actions specify target hand joint angles and wrist poses through a PD controller rather than 6-DoF force control.The passage contrasts this wrist-pose control with the 6-DoF force used in MANIPTRANS.
  • Evaluation: Rollouts evaluate previously unseen goal positions in IsaacGym and succeed when the object remains within 10 cm of the goal.Qualitative results appear in Fig. 11, while quantitative results are summarized in Table 2 of the main paper.
Loading 2503.21860v1…