H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation

Introduction

• Robot demonstrations require costly teleoperation or complex collection infrastructure, while robot datasets remain limited and vary in quality.Diverse robot morphologies and action spaces also make unified cross-embodiment training challenging.Source: Introduction · p. 2
• Large-scale egocentric videos with 3D hand-pose annotations offer behavioral priors about object affordances, manipulation strategies, and task decomposition.EgoDex is cited as an example with 829 hours of manipulation videos.Source: Introduction · p. 2
• H-RDT leverages egocentric human manipulation data to enhance robotic policy learning through a diffusion transformer with modular human-to-robot transfer components.The paper presents this as a framework for cross-embodiment knowledge transfer.Source: Introduction · p. 2
• The paper’s contributions include empirical validation across simulation and real-world settings and insights into human priors for few-shot robot learning.The contribution summary also highlights consistent improvements over state-of-the-art methods.Source: Introduction · p. 2

Learning-based Robotic Manipulation

• Specialized action policies model direct visuomotor control, while VLA models combine visual, language, and action processing and target instruction following and cross-embodiment generalization.The paper situates its approach as building on RDT while adding flow matching and human-to-robot transfer.Source: Learning-based Robotic Manipulation · p. 2

Learning from Egocentric Human Manipulation

• Egocentric datasets have often lacked precise 3D hand poses, while EgoDex provides 829 hours and 338k episodes with per-frame hand poses and language descriptions.These annotations address a stated gap for dexterous manipulation learning.Source: Learning from Egocentric Human… · pp. 2–3
• Earlier human-video approaches used 2k demonstrations for EgoMimic and 27k for HAT, and targeted a single humanoid embodiment.H-RDT uses 338k trajectories and separates large-scale human pre-training from robot-specific fine-tuning, with modular adapters for diverse morphologies.Source: Learning from Egocentric Human… · p. 3

Problem Formulation

• H-RDT formulates manipulation as generating future action sequences from visual observations, robot state, and a language instruction.Each action sequence covers a prediction horizon and can encode robot control commands such as joint positions or end-effector poses.Source: Problem Formulation · p. 3

Overview

• Robot-specific demonstrations are scarce, and human-to-robot transfer must address embodiment differences and adaptation to multiple robot platforms.H-RDT proposes a shared action representation and two-stage training to support cross-embodiment deployment.Source: Problem Formulation · p. 3, Overview · p. 3

Human Action Representation Design

• H-RDT represents human actions as 48-dimensional vectors comprising bilateral wrist poses and fingertip positions.The representation is described as a superset of action spaces for many robots controlled by end-effector poses, emphasizing transferable manipulation features.Source: Human Action Representation Design · p. 3

Two-Stage Training Paradigm

• The two-stage paradigm pre-trains on EgoDex human hand actions, then transfers vision, language, and transformer weights while reinitializing robot-specific state and action modules for fine-tuning.Pre-training uses 338K+ trajectories across 194 tasks; fine-tuning adapts the action modules to the target robot’s action space.Source: Two-Stage Training Paradigm · p. 4, Figure 2 · p. 4

H-RDT Architecture

• H-RDT uses flow matching to learn a vector field from noise toward target actions, with inference sampling actions by integrating that field using an ODE solver.The policy conditions on multi-view RGB observations, proprioception, and a language instruction.Source: Flow Matching for Action Generation · p. 4, Two-Stage Training Paradigm · p. 4
• A modular transformer combines vision and language encoders, an action encoder, a backbone, and an action decoder for robot-specific outputs.Image and language features use separate cross-attention; the decoder uses timestep embeddings and outputs actions in the target robot’s action space.Source: Network Architecture · p. 5

Experimental Setup / Baselines and Comparison Methods

• Evaluations span Easy and domain-randomized Hard simulation, three real-world platforms, and comparisons with RDT, π0, and H-RDT w/o human.The setup also assesses single- and multi-task performance, cross-embodiment generalization, robustness, and few-shot learning.Source: Experimental Setup · p. 5, Baselines and Comparison Methods · p. 5, Real-world Validation · p. 5

Real-world Validation

• On Aloha-Agilex-2.0, towel-folding complete success is 52% versus 40% for RDT, while cup-to-coaster complete success is 64% versus 28% for RDT.Training without human data achieves 0% on towel folding and 20% on cup-to-coaster placement.Source: Aloha-Agilex-2.0 Experiments · pp. 5–6, Table 1 · p. 5, Table 2 · p. 6
• 58% average success across two real-world bimanual tasks compares with 34% for RDT and 10% for training without human data.The evaluation includes deformable-object manipulation and spatial reasoning tasks.Source: Aloha-Agilex-2.0 Experiments · p. 6
• 41.6% average success in the 113-task, 1–5-demonstration ARX5 setting compares with 16.0% for RDT, 31.2% for π0, and 17.6% for H-RDT w/o human.The paper describes this as a real-world multi-task few-shot evaluation.Source: Dual-arm ARX5 Few-shot Experiments · p. 6, Table 3 · p. 7

Simulation Results on RoboTwin 2.0

• H-RDT records average success rates of 68.7% in Easy mode and 25.6% in Hard mode, exceeding training from scratch by 8.4% in each mode.The single-task evaluation covers 13 tasks trained on 50 demonstrations; Hard mode includes environmental randomization.Source: Single-Task Performance · p. 6
• 87.2% average success in multi-task RoboTwin 2.0 exceeds RDT (28.8%), π0 (48.4%), and H-RDT w/o human (67.2%).The reported improvement over training from scratch is 20.0% absolute.Source: Multi-Task Performance · p. 6, Table 5 · p. 7
• H-RDT achieves 87.2% on Aloha-Agilex-1.0 and 62.9% on Franka-Panda, outperforming baselines on both simulated embodiments.The authors attribute consistent improvements across morphologies to the modular action encoder design.Source: Cross-Embodiment Generalization · p. 6, Figure 3 · p. 7

Analysis and Discussion

• The authors report consistent improvements across settings, with benefits especially pronounced in few-shot real-world scenarios and performance maintained under environmental variation.They describe evaluation across diverse embodiments and tasks, including deformable-object manipulation and spatial reasoning.Source: Analysis and Discussion · p. 7

Conclusion

• H-RDT leverages egocentric human manipulation videos with 3D hand-pose annotations to learn behavioral priors for adaptation to diverse robotic tasks.Source: Conclusion · p. 7
• H-RDT combines a modular transformer with specialized action encoders and decoders, flow matching, and human-data pre-training followed by robot-specific fine-tuning.Source: Conclusion · p. 7
• Evaluations demonstrate consistent improvements over state-of-the-art methods.The paper presents these results as validation that human manipulation priors provide powerful inductive biases for sample-efficient robotic learning.Source: Conclusion · p. 7

• The single-task experiments cover 13 manipulation tasks on the RoboTwin 2.0 benchmark.Source: §A.1 · p. 10
• Table 8 presents success rates for all baseline methods in Easy and Hard evaluation modes.Source: §A.1 · p. 10
• Training concatenates three 240×320 views into a 360×320 input, which may degrade performance relative to higher-resolution settings.Source: §A.1 · p. 10

A.2 Franka-Panda Detailed Results

• Detailed Franka-Panda results cover the 10-task subset used for evaluation in the main paper.Source: §A.2 · p. 10
• Table 6 is identified as reporting detailed Franka-Panda multi-task results in percent on the 10-task subset.Source: Table 6 · p. 10

B.1 Model Architecture

• Table 7 provides key hyperparameter settings for the H-RDT model architecture.Source: §B.1 · p. 10

B.2 Training Configuration and Data Processing

• Pre-training uses the 338K+ trajectory EgoDex dataset with 48-dimensional human hand actions, while fine-tuning adapts to target robot action spaces.Both stages use AdamW with learning rate 1e-4, weight decay 0.01, and gradient clipping at norm 1.0; images use 196 vision patches and language instructions are capped at 1024 tokens.Source: §B.2 · p. 10

B.3 Details on Flow Matching

• Training uses uniform timestep sampling with τ ∈[0, 0.999].Source: §B.3 · p. 10
• Table 7 is identified as containing H-RDT architecture hyperparameters and key configuration settings.Source: Table 7 · p. 10
• Inference starts from Gaussian noise and integrates the learned vector field with a deterministic ODE solver using five function evaluations at 30Hz control frequency.The step size is 0.2.Source: §B.3 · p. 10

B.4 Real-World Task Definitions

• Figure 4 provides visual illustrations of the real-world manipulation task in the experiments.Source: §B.4 · p. 10

C. Training Configurations

• The training-configuration section provides detailed settings for all experimental settings described in the main paper.Source: C. Training Configurations · p. 10

C.1.1 Single-Task Configuration

• The single-task data comprises 13 tasks with 50 clean trajectories per task.Source: §C.1.1 · p. 10
• Experiments use the Aloha-Agilex-1.0 platform and train for 10k steps on 4 H100 GPUs with batch size 16 per GPU.Source: §C.1.1 · p. 10
• Evaluation covers Easy mode with clean scenes and Hard mode with domain randomization.Source: §C.1.1 · p. 10

C.1.2 Multi-Task Configuration

• The multi-task configuration uses 45 tasks with 50 domain-randomized trajectories each, evaluated in Hard mode with three camera views.Training uses Aloha-Agilex-1.0 and Franka-Panda, with 30k steps for Aloha and 10k for Franka.Source: §C.1.2 · p. 10
• Table 8 reports that H-RDT achieves the highest average success rate across Easy and Hard modes on the RoboTwin 2.0 single-task benchmark.Source: Table 8 · p. 11

C.2.1 Dual-arm ARX5

• The dual-arm ARX5 few-shot pick-place setup uses 1–5 trajectories per task, with 607 trajectories across 113 tasks.Training runs for 100k steps on 4 H100 GPUs.Source: §C.2.1 · p. 11

C.2.2 Dual-arm UR5+UMI Platform

• The dual-arm UR5+UMI setup uses 100 trajectories for dual-hand takeout bag placement and trains for 20k steps on 8 H100 GPUs.Source: §C.2.2 · p. 11

C.2.3 Aloha-Agilex-2.0 Platform

• The Aloha-Agilex-2.0 dual-arm Piper setup trains on 50 trajectories each for pouring water and folding clothes.Training runs for 50k steps on 8 H20 GPUs.Source: §C.2.3 · p. 11
• Figure 4 depicts the task definition for the real-world experiments.Source: Figure 4 · p. 12