Source-linked AI summary
Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots
Sijin Chen, Kaixuan Jiang, Haixin Shi, Yanhui Wang, Weiheng Zhong, Haosheng Li, Bo Jiang, Yuxiao Liu, Xihui Liu
TL;DR
Transferring human manipulation skills to bi-manual robots is difficult because noisy human hand poses and differing contact patterns make rotation-inclusive actions poorly matched. The paper uses wrist-translation bridging actions with interleaved action sequences and attention masking, achieving more effective human-to-robot transfer and benefiting from large-scale human pretraining.
Problem
Human-to-robot skill transfer lacks action representations that accommodate noisy hand-pose estimates and differing contact patterns between human fingers and parallel grippers.
Method
The paper represents actions through shared wrist translations and uses interleaved action sequences with attention masking to handle missing components across embodiments.
Results
The translation-based bridging action transfers human manipulation knowledge to robots more effectively than noisy 6DoF human actions and benefits from large-scale human pretraining.
Takeaways & Limitations
Shared wrist translations provide a compatible action space for transferring human manipulation knowledge to bi-manual robots within the evaluated tasks.
Takeaways & Limitations
Using only wrist translations limits transfer on tasks requiring fine rotational adjustments during contact-rich manipulation.
Abstract
from arXiv · showhide
We study whether we can learn novel manipulation skills from human actions to a bi-manual robot with parallel grippers. Human action data is cheap, abundant, and diverse, making it one of the most promising resources for scaling up robot learning. Yet transferring skills from humans to robots remains hard: most prior work treats humans as just another bi-manual 6DoF embodiment, where hand-pose estimates are noisy and the contact patterns of human fingers differ fundamentally from those of a parallel gripper. We argue that learning rotation-inclusive action signals from human data is therefore sub-optimal, and instead propose a bridging action representation: the relative wrist translation within the initial head-camera frame, an action space shared by humans and robots. To handle the potential absence of certain action components in different embodiments, we build a $π_0$-like vision-language-action model with interleaved action tokens and attention masking. On a suite of novel bi-manual manipulation tasks, our bridging action transfers human manipulation knowledge to robots far more effectively than noisy 6DoF human actions and scales with the amount of human data.
1 Introduction
The paper studies transferring manipulation skills from inexpensive, diverse human data to bi-manual robots and proposes wrist translation as a shared bridging action. A unified interleaved action model handles embodiment-specific missing components and supports scalable human-data pre-training.
- Motivation: Human manipulation data offers a scalable, economical source for expanding robot learning across environmental and skill diversity.Its robot-free collection can be conducted at much lower cost and almost anywhere.
- Challenges: Human-to-robot transfer is difficult because estimated human actions are noisy and human-finger contact patterns differ substantially from parallel-gripper contacts.These differences make direct use of rotation-inclusive human actions sub-optimal for the target robot embodiment.
- Unified model: A π0-like vision-language-action model uses interleaved action representations and attention masks to handle missing action components across heterogeneous embodiments.The interleaved design also enables large-scale human action pre-training.
- Training strategy: Three-stage training combines human wrist-translation pre-training, co-training with robot and task-specific human actions, and optional few-shot real-robot post-training.The model can complete certain tasks without task-specific robot demonstrations, while additional few-shot robot data can improve overall performance.
- Bridging representation: The proposed bridging action uses relative wrist translation, a representation shared by humans and robots and more robust than noisy 6DoF human actions.The paper reports that this representation facilitates superior manipulation skill transfer from human to robot.
2 Related Works
Related work leverages large-scale human and video data to learn representations and physical knowledge for robot learning. Cross-embodiment learning seeks shared knowledge across robot configurations, but differing action representations remain a major challenge.
- Learning from human: Researchers use large egocentric datasets and human videos to learn generative visual representations, affordances, keypoints, and latent actions.These approaches aim to harness physical knowledge from large-scale human data.
- Cross embodiment learning: Cross-embodiment learning learns shared knowledge across robot configurations within unified models for controlling diverse robot collections.Learning from explicit human actions is also framed as a form of cross-embodiment learning.
- Cross embodiment learning: Different action representations complicate cross-embodiment learning, motivating unified spaces based on concatenation or padding missing dimensions.The passage identifies representation differences as a significant challenge and lists these as existing strategies.
- Data pyramid and co-training: Robot learning systems use cross-embodiment data, human data, vision-language corpora, and videos as cost-effective alternatives to collecting real robot data.Data pyramids and co-training are discussed as approaches for building robust and generalizable systems from diverse data sources.
3 Hardware and Platform
The study uses ByteMini, a bi-manual mobile manipulation platform with two 7-DoF arms, parallel grippers, and three RGB-D cameras, alongside human demonstrations collected with PICO 4 Ultra Enterprise. During data collection, robot operators vary the platform configuration, while human operators imitate robot grippers and keep their hands visible to the top camera.
- Robot platform: ByteMini has two 7-DoF arms, parallel grippers, and three head- and wrist-mounted RGB-D cameras.During rollout, the base remains static while the arms use 6DoF end-effector poses and discrete gripper signals.
- Robot platform: Tele-operators randomly adjust the robot’s height, position, and rotation during data collection to provide configuration diversity.The robot base is kept static during rollout.
- Human data platform: Human action data is collected with PICO 4 Ultra Enterprise, with operators imitating robot-gripper hand postures and keeping their hands within the top camera’s field of view.These collection instructions target transferable robotic manipulation behaviors from human data.
- Architecture overview: The model is trained on mixed human and robot action data using a π0-like vision-language-action policy with interleaved action sequences.The architecture addresses missing action components and supports manipulation behavior transfer.
4 Method
The method transfers human manipulation skills to a bimanual robot through a shared wrist-translation representation rather than direct human 6DoF wrist actions. A π0-like vision-language-action model uses interleaved action tokens, attention masking, and per-embodiment supervision to accommodate missing action components and ground human knowledge in executable robot actions.
- Bridging action representation: The framework transfers manipulation skills from human actions to a bimanual robot using a shared bridging action representation.The representation is based on wrist translations that can be extracted from both human and robot actions.
- Bridging action representation: The method avoids human 6DoF wrist actions because wrist rotations are noisy and finger-to-gripper contact differences decouple rotation from semantic manipulation behavior.These limitations make direct replay of estimated human wrist rotations sub-optimal.
- Bridging action representation: Relative wrist translation in the initial head-camera frame replaces direct 6DoF human wrist actions as the shared signal.The translation-only signal is physically meaningful, robust to noisy rotation estimates, and embodiment-agnostic.
- Architecture: A π0-like end-to-end VLA generates action chunks from language and head- and wrist-camera observations, padding missing human wrist views with blank images.Vision-language tokens provide context to an Action Transformer that generates actions through flow matching.
- Interleaved action tokens and supervision: Interleaved a3D-wrist, a6D-eef, and agripper tokens use attention masks and position ids to handle embodiment-specific missing components.The shared wrist signal is ordered before end-effector actions to support transfer, while gripper signals follow target-reaching actions; unavailable components are masked and excluded from loss calculation.
- Training strategy: Randomly including or substituting a3D-wrist as the robot prediction target explicitly binds the bridging representation to executable robot actions, and this binding is essential for transfer.The training strategy combines human pre-training with human-robot co-training and uses approximately 600 hours of human actions plus approximately 72 hours of generalized robot data.
5 Experiments · 5.1 Evaluation Setups
The experiments evaluate bridging-action transfer across 15 real-world bi-manual manipulation tasks, using human and robot demonstrations and controlled novel-scene trials. They are organized around transfer, scaling, data efficiency, training objectives, executable action alignment, and performance limits.
- 5 Experiments: The experiment asks whether bridging actions transfer beyond generic pick-and-place and scale with large-scale human-only pre-training.It also compares bridging actions with 6DoF human actions and examines few-shot robot post-training efficiency.
- 5 Experiments: The evaluation further tests which training objective is essential for human-robot co-training and whether human-only pre-training aligns with executable robot actions.A final question assesses the performance upper bound of the bridging action.
- 5.1 Evaluation Setups: The study evaluates bridging-action transfer on 15 real-world manipulation tasks grouped by their objects.For each task, the authors collect three hours of task-specific human actions and 100 task-specific robot trajectories.
- 5.1 Evaluation Setups: Each task uses two distinct testing layouts with detailed progress-scoring criteria.The evaluation scenes include varied distractors and differ from the training data.
- 5.1 Evaluation Setups: Robot-only pick-and-place training is insufficient for downstream tasks, whereas human-robot co-training transfers manipulation knowledge to robot actions.The bridging action additionally benefits from large-scale human-only pre-training and few-shot robot demonstrations.
- 5.1 Evaluation Setups: Each evaluation scene is tested four times, yielding eight trials per task after resetting the robot pose and object layout before each rollout.The resets follow a pre-recorded mask to support relatively fair comparisons.
5.2 The bridging action is scalable and essential for manipulation skill transfer
The bridging action is essential for transferring manipulation skills beyond pick-and-place, as robot-only training performs poorly while human–robot co-training substantially improves progress and success. It also scales effectively with large-scale human-only pre-training and can benefit from additional few-shot robot demonstrations.
- Scalable human pre-training: Additional few-shot robot demonstrations further benefit the bridging action after human-only pre-training and human–robot co-training.Figure 6 compares this setting with robot-only training, co-training, and large-scale human-only pre-training.
- Beyond pick-and-place: Robot pick-and-place training alone achieves very low performance on all downstream evaluation tasks.This rules out solving the tasks through generalization from large-scale pick-and-place data alone.
- Beyond pick-and-place: Co-training human and robot actions substantially improves both task progress and success rate.The improvement demonstrates that human manipulation knowledge transfers effectively through the bridging action.
- Scalable human pre-training: Large-scale human-only pre-training substantially improves performance, confirming that the bridging action representation is scalable and effective.The policy is pre-trained using only the bridging action before human–robot co-training.
5.3 Why translation-only instead of 6DoF human wrist actions?
The paper compares mainstream 6DoF human wrist actions with translation-only bridging actions for human–robot co-training. Translation-only actions produce more stable, natural, and handle-aligned robot manipulation than the noisy, twisted, or distorted behaviors induced by 6DoF actions.
- Bridging actions yield a natural wrist pose aligned with the microwave door handle.
- The baseline extracts relative 6DoF wrist actions by treating the human hand as another robotic embodiment.
- Co-training with 6DoF human actions produces noisy and twisted robot behaviors.Qualitatively, the robot can yield a distorted, off-target wrist pose.
- Translation-only bridging actions produce more stable manipulation behaviors than the 6DoF baseline.The comparison reports this superiority qualitatively and quantitatively in Table 2.
5.4 Human-only pre-training improves post-training data efficiency
Human-only pre-training with bridging actions improves post-training data efficiency when downstream evaluation uses few-shot robot trajectories. Despite seeing only non-executable human bridging actions during pre-training, the model achieves higher average progress and success rate after post-training.
- Experiment: The study evaluates whether bridging actions improve post-training efficiency using few-shot robot data without robot trajectories during human-only pre-training.The comparison uses 10 robot trajectories per task for few-shot post-training.
- Training setup: During human-only pre-training, the model predicts only the bridging signal L3D-wrist FM, while post-training includes all action components.This separates non-executable human-action supervision in Stage I from executable-action learning in Stage III.
- Result: Despite no exposure to executable actions during pre-training, the model substantially improves average progress and success rate during post-training.The result indicates that pre-training knowledge transfers to more efficient few-shot robot post-training.
5.5 Training bridging actions for robot data is essential for manipulation transfer · 5.6 Human-only pre-training aligns with the executable robot action space
Robot data must also be trained with the bridging action objective: removing it reduces overall success from 38.33% to 12.50%. Human-only pre-training on wrist translations accelerates convergence of the remaining executable robot action components during co-training.
- 5.5 Training bridging actions for robot data is essential for manipulation transfer: During human-robot co-training, robot data randomly adds or substitutes the bridging action a3D-wrist for the 6D end-effector action objective.This augmentation supervises bridging actions directly on robot data.
- 5.5 Training bridging actions for robot data is essential for manipulation transfer: Removing the bridging training objective degrades performance across all task groups, reducing overall success from 38.33% to 12.50%.The comparison uses checkpoints co-trained on human and robot actions, with and without bridging-objective augmentation, after large-scale human-only pre-training.
- 5.5 Training bridging actions for robot data is essential for manipulation transfer: The ablation shows that supervising bridging actions on robot data is essential for manipulation skill transfer.Table 4 summarizes the bridging-objective ablation during co-training.
- 5.6 Human-only pre-training aligns with the executable robot action space: Training-loss comparisons during co-training examine whether the bridging-action objective aligns with 6DoF end-effector actions.The analysis compares an experiment initialized from scratch with another initialized from human-only pre-training.
- 5.6 Human-only pre-training aligns with the executable robot action space: Although human-only pre-training supervises only wrist translations, it accelerates convergence of 6DoF end-effector and gripper action losses.This convergence behavior is shown during human-robot co-training.
- 5.6 Human-only pre-training aligns with the executable robot action space: Human-only pre-training therefore aligns wrist-translation supervision with the other action components used by the robot.The aligned components are the 6DoF end-effector action and gripper action.
5.7 The bridging action aligns with the executable robot actions · 5.8 The upper bound of the bridging objective · 5.9 Failure Case Analysis
The bridging action aligns closely with executable robot end-effector actions across diverse tasks. Its transfer performance strengthens when embodiment gaps, visual gaps, and action noise are reduced, while failures remain concentrated in contact-rich tasks requiring precise end-effector configurations.
- 5.7 The bridging action aligns with the executable robot actions: The predicted translation-only wrist action and 6DoF end-effector action align closely when generated from the same vision-language input and projected into the head-camera frame.This alignment is observed across diverse manipulation tasks.
- 5.7 The bridging action aligns with the executable robot actions: Figure 10 visualizes the close correspondence between bridging and 6DoF end-effector predictions across diverse tasks.Both action types are projected onto the head camera using the robot’s state.
- 5.8 The upper bound of the bridging objective: The upper-bound experiment treats 100 task-specific robot demonstrations per task as human data after converting them into translation-only wrist actions.It applies the human-action training objective without an observation gap and with substantially less action noise.
- 5.8 The upper bound of the bridging objective: Reducing the visual gap and action noise makes manipulation-knowledge transfer increasingly efficient under the same bridging training objective.The upper-bound analysis uses task-specific in-lab robot actions rather than human demonstrations.
- 5.9 Failure Case Analysis: Failure cases mainly occur when success requires both reaching the correct interaction region and maintaining a precise end-effector configuration during contact-rich manipulation.The representative tasks are “insert the straw into the cup” and “open the drawer”.
- 5.9 Failure Case Analysis: In representative failures, the policy shows clear task intent but cannot securely grasp the straw or rotate the wrist to establish valid pulling contact.These critical-step failures are visualized in Figure 12.
6 Conclusions
The paper studies learning task-specific bimanual robot skills from human motion data using a translation-based bridging action compatible with both embodiments. It also introduces interleaved action sequences to handle missing action components, while acknowledging limits for fine rotational adjustments and thin-object pickup.
- Limitations and Future Work: Using only wrist translation limits transfer on tasks requiring fine rotational adjustments.The limitation arises from adopting wrist translation as the bridging action for in-the-wild human actions.
- Limitations and Future Work: After co-training, the robot struggles to pick up thin objects because of the observation and embodiment gap and noise in human actions.The paper attributes this difficulty mainly to those two factors.
- Conclusions: The proposed translation-based bridging action is compatible with both robot and human action data.The work investigates the feasibility of learning task-specific bi-manual robot skills from human motion data.
- Conclusions: Interleaved action sequence representations address potentially missing action components across different data sources.This representation is introduced to handle action components that may be absent in different embodiments or datasets.