PaperlayerView exact revision on arXiv

Source-linked AI summary

You Only Teach Once: Learn One-Shot Bimanual Robotic Manipulation from Video Demonstrations

Huayi Zhou, Ruixiang Wang, Yunxin Tai, Yueci Deng, Guiliang Liu, Kui Jia

arXiv:2501.14208v2cs.ROcs.CV

TL;DR

Bimanual manipulation remains difficult because two arms must coordinate in a high-dimensional action space, while taxonomy-based methods and teleoperation have limitations in coverage or scalability. YOTO extracts and injects hand-motion patterns from a one-shot video demonstration, proliferates training examples, and trains a bimanual diffusion policy; it reports nearly 60% average success across five tasks and validation across synchronous and asynchronous scenarios.

  • Problem

    Bimanual manipulation requires coordinated dual-arm control in a high-dimensional action space, while predefined taxonomies may not cover arbitrary tasks and teleoperation requires substantial data that is inconvenient to scale.

    Source: §I · p. 1

  • Method

    YOTO extracts hand motion from a one-shot human demonstration, converts trajectories into keyframes, proliferates training examples, and trains a customized bimanual diffusion policy.

    Source: §I · p. 2

  • Results

    BiDP achieved nearly 60% average success across five tasks, and YOTO was validated on five complex long-horizon tasks including synchronous and asynchronous manipulation.

    Source: §I · p. 2, B. Results Comparison · p. 10

  • Takeaways & Limitations

    The authors present YOTO as a standardized approach for transferring human motion to robots, generating scalable training data, and learning intricate dual-arm tasks.

    Source: §VI · p. 11

    The reported validation covers five complex bimanual tasks; broader task coverage is not specified in the conclusion.

    Source: §VI · p. 11

Abstract

from arXiv · show

Bimanual robotic manipulation is a long-standing challenge of embodied intelligence due to its characteristics of dual-arm spatial-temporal coordination and high-dimensional action spaces. Previous studies rely on pre-defined action taxonomies or direct teleoperation to alleviate or circumvent these issues, often making them lack simplicity, versatility and scalability. Differently, we believe that the most effective and efficient way for teaching bimanual manipulation is learning from human demonstrated videos, where rich features such as spatial-temporal positions, dynamic postures, interaction states and dexterous transitions are available almost for free. In this work, we propose the YOTO (You Only Teach Once), which can extract and then inject patterns of bimanual actions from as few as a single binocular observation of hand movements, and teach dual robot arms various complex tasks. Furthermore, based on keyframes-based motion trajectories, we devise a subtle solution for rapidly generating training demonstrations with diverse variations of manipulated objects and their locations. These data can then be used to learn a customized bimanual diffusion policy (BiDP) across diverse scenes. In experiments, YOTO achieves impressive performance in mimicking 5 intricate long-horizon bimanual tasks, possesses strong generalization under different visual and spatial conditions, and outperforms existing visuomotor imitation learning methods in accuracy and efficiency. Our project link is https://hnuzhy.github.io/projects/YOTO.

I. INTRODUCTION

YOTO learns bimanual skills from a single human video demonstration, then expands the resulting data to train a specialized diffusion policy. The paper frames this approach against coordination and state-complexity challenges, and reports validation on five complex long-horizon tasks.

A. Bimanual Robotic Manipulation

  • Bimanual control is challenging because the arms must coordinate while operating in a higher-dimensional action space; taxonomy-based methods may not cover arbitrary tasks, and teleoperation data is costly to scale.The paper contrasts these approaches with learning from human action videos.Source: §I · pp. 1–2

B. Learn from Human Hand Videos

  • YOTO extracts human-hand features, reduces noisy trajectories to discrete keyframes, and uses inverse-kinematics interpolation and a motion mask to transfer coordinated movements to two arms.The motion mask records and replays the order of dual-hand movements.Source: §I · p. 2

I. INTRODUCTION

  • Demonstration proliferation varies object poses and analogous objects, then augments object point clouds to generate diverse training examples from one teaching.The authors report collecting about 300 demonstrations in 8 hours through auto-rollout for a well-taught task.Source: §I · p. 2
  • BiDP trains on manipulated-object observations, predicts keyposes, and uses the motion mask to organize alternating or synchronous arm actions into a unified policy.The paper positions its fixed-base, parallel-jaw setup and one-shot data generation in contrast to teleoperation-based approaches.Source: §I · p. 2, A. Bimanual Robotic Manipulation · p. 2
  • YOTO was validated on five complex long-horizon bimanual tasks, including synchronous and asynchronous manipulation.The framework is presented as compatible with most bimanual tasks.Source: §I · p. 2

C. Visuomotor Imitation Learning

The paper situates BiDP within visuomotor imitation learning and formulates bimanual imitation from shared visual observations and robot proprioception. Its setup uses a binocular camera and fixed-base dual arms with parallel-jaw grippers.

C. Visuomotor Imitation Learning

  • Visuomotor imitation learning trains action policies from visual observations and demonstrations; BiDP adds a motion-mask condition and task-related object point clouds for bimanual learning.The cited approaches include action chunking and diffusion-based policies.Source: C. Visuomotor Imitation Learning · p. 3

III. HARDWARE SYSTEM

  • The system uses contralaterally placed fixed-base arms with parallel-jaw grippers and a binocular stereo camera that supplies left and right images for post-processing.Each arm has an approximately 880 mm span, and each gripper opens to 80 mm.Source: §III · p. 3

IV. METHOD

  • The method comprises hand-motion extraction and injection, demonstration proliferation from one teaching, and a visuomotor bimanual diffusion policy.The next stages explain how one-shot demonstrations are expanded and how diffusion-based imitation is adapted to bimanual tasks.Source: §IV · p. 4, A. Problem Formulation · p. 4

A. Problem Formulation

  • The task formulation assumes shared visual observations from a binocular camera, robot proprioception, and demonstrations pairing observations with actions for learning an observation-conditioned imitation policy.Actions include each arm’s target 6-DoF pose and binary gripper state.Source: A. Problem Formulation · p. 4

B. Hand Motion Extraction and Injection

The extraction pipeline captures a human demonstration and converts hand motion into robot-usable trajectories, states, and keyframes. It uses stereo-based back-projection to address instability in directly estimated hand trajectories.

B. Hand Motion Extraction and Injection

2) High-Level Features Extraction:

  • The pipeline detects both hands, estimates their 3D shapes, and tracks hand centers using selected joints such as fingertips to form 3D trajectories.The hand model uses 21 predefined joints.Source: 2) High-Level Features Extraction · p. 4
  • Directly retargeting estimated hand trajectories is unreliable because hand classification and 3D shape predictions can be erroneous and temporally inconsistent; stereo back-projection is used for more stable trajectories in the camera space.The method projects estimated 3D points into the image and lifts them back into 3D with stereo matching.Source: 2) High-Level Features Extraction · p. 4
  • The method estimates hand open/closed state from object contact and simplifies hands into lower-dimensional grippers, with Fig. 3 illustrating extracted trajectories and corresponding keyframes for both hands.The authors describe contact-based state estimation as more trustworthy than relying solely on the hands.Source: 2) High-Level Features Extraction · p. 4, Figure 3 · p. 5

3) Robot Actions Injection:

The method converts hand-motion trajectories into discrete keyframes, verifies and injects those robot actions, and uses timestamps to schedule the two arms. This reduces the trajectory to a smaller set for checking, but mixed synchronized and asynchronous tasks are not considered.

3) Robot Actions Injection:

  • Keyframes summarize demonstrations as intermediate end-effector poses, letting the robot learn the next pose and use a sampling-based planner to reach it.They can be extracted using changes in gripper state or local velocity/acceleration extrema.Source: 3) Robot Actions Injection · p. 5
  • K is around 10 in these tasks and much smaller than J, making errors faster to verify and correct.The injected poses are transformed from camera coordinates to robot coordinates using a pre-measured hand-eye calibration matrix.Source: 3) Robot Actions Injection · p. 5
  • A motion mask derived from recorded timestamps schedules dual-arm coordination across the keyframes.The mask indicates whether each arm holds or moves at a keyframe.Source: 3) Robot Actions Injection · p. 5
  • The approach does not consider long-horizon tasks that mix synchronized and asynchronous keyframes.For fully synchronous tasks, the two arms’ motion-mask values remain the same.Source: 3) Robot Actions Injection · p. 5

C. Demonstration Proliferation from One Teaching

One-shot teaching supports demonstration proliferation through real-robot auto-rollout and point-cloud geometry augmentation. In auto-rollout, editable keyframes are adjusted for changed objects before replaying the task.

C. Demonstration Proliferation from One Teaching

1) Auto-Rollout Verification in Real-World:

  • For auto-rollout, keyframes associated with a manipulated object can be edited to match its changed position before the robot replays the task.The method also permits replacing objects with similar-shaped ones in the same position.Source: 1) Auto-Rollout Verification in… · pp. 5–6
  • Auto-rollout is reported as more efficient than master-slave arm synchronization and drag-and-drop teaching, which are hampered by multiple operators and higher failure rates.The comparison covers data collection methods; the supplied table caption reports completion-time comparisons across three tasks.Source: 1) Auto-Rollout Verification in… · p. 6, Table I · p. 6

2) Geometric Transformation of Point Clouds:

Point-cloud geometry transformations generate additional object-observation and action pairs, with transformations restricted to the robot’s reach. BiDP learns bimanual keyframe actions using manipulated-object point clouds and a motion-mask-based action representation.

2) Geometric Transformation of Point Clouds:

  • Point-cloud geometry augmentation applies controlled rotations and translations to manipulated objects and updates corresponding keyframes to generate new observation-action pairs in batches.The transformations are restricted to the robot arm’s reachable area; the moving range can be measured during auto-rollout.Source: 2) Geometric Transformation of Point… · p. 6

D. Bimanual Diffusion Policy Learning

  • BiDP uses manipulated-object point clouds as observations and learns from datasets expanded from the initial one-shot demonstration.The dataset includes motion masks that encode coordination strategies and can contain hundreds or thousands of trajectories.Source: D. Bimanual Diffusion Policy Learning · p. 6
  • The diffusion policy predicts denoising gradients from observations, noisy actions, and a diffusion timestep, using a point-cloud encoder and convolutional U-Net.The approach uses SIM(3)-equivariant architectures for point-cloud input.Source: D. Bimanual Diffusion Policy Learning · p. 6
  • The policy’s training objects include everyday manipulated-object instances collected for each of five bimanual tasks to improve and verify generalizability.The objects were not intentionally customized.Source: Figure 4 · p. 7
  • The motion mask reorganizes actions into time-ordered left- and right-arm sequences, removing potentially redundant actions in asynchronous tasks.This representation unifies the bimanual policy form and is also compatible with single-arm tasks.Source: D. Bimanual Diffusion Policy Learning · pp. 6–7

V. EXPERIMENTS

The experiments are framed around hand-motion extraction quality, the effect of YOTO strategies on policy learning, generalization beyond the training domain, and compatibility with complex long-horizon tasks.

V. EXPERIMENTS

  • The experiments ask whether the framework is compatible with diverse complex long-horizon bimanual tasks.The stated scope includes compatibility with a variety of such tasks.Source: §V · p. 7
  • The evaluation asks whether trained BiDP models generalize outside the in-distribution domain.Source: §V · p. 7
  • The experiments examine the quality of extracted hand motions and whether YOTO strategies improve bimanual policy learning.Source: §V · p. 7

A. Experiment Setups

YOTO is evaluated on five complex, long-horizon bimanual tasks spanning synchronous and asynchronous collaboration, with varied objects and primitive skills. Demonstrations start from one-shot teaching and are expanded across object positions and categories for policy training.

1) Tasks:

  • Five long-horizon tasks cover synchronous and asynchronous collaboration, varied object types, and skills including pulling, placing, unscrewing, revolving, and lifting.The tasks are pull drawer, pour water, unscrew bottle, uncover lid, and open box.Source: 1) Tasks · p. 7

2) Demonstrations:

  • One-shot teaching and multiple objects per task provide starting demonstrations for collecting diverse trajectories to train closed-loop policies.The task demonstrations include diverse object positions and categories.Source: 2) Demonstrations · pp. 7–8
  • Position variations and object replacements diversify real-robot demonstrations, then geometric transformations expand training data 100-fold to 5K–24K trajectories per task.The setup uses 3 position variations for pull drawer and pour water, and 9 for the other tasks.Source: 2) Demonstrations · p. 8

3) Baselines:

Across five long-horizon tasks, BiDP outperforms the compared visuomotor policies, with nearly 60% average success; its performance also degrades least in out-of-distribution evaluation. The experiments compare policy designs, trajectory extraction, and generalization under randomized placements and novel objects.

4) Metrics:

  • Evaluation uses randomized initial placements, reports substep average lengths and final success rates, and tests novel objects in two OOD tasks.OOD trials are quadrupled relative to the regular evaluation.Source: 4) Metrics · p. 8

B. Results Comparison

  • Nearly 60% average success across five tasks: BiDP significantly outperforms all compared policies on long-horizon bimanual manipulation.The reported average is summarized across all five tasks.Source: Table V · p. 9, B. Results Comparison · p. 10
  • Projected hand points improve trajectory trends over raw reconstructed 3D points, while filtered keyframes lifted into 3D provide end-effector keyposes.The illustrated tasks are pull drawer and uncover lid; the authors note the projection approach suits stationary cameras with known parameters.Source: Figure 5 · p. 8, B. Results Comparison · p. 9
  • Geometric transformation augmentation produced the most prominent growth in the pull-drawer ablation, and combining the strategies yielded the best results.The ablation uses 243 training episodes; the reported data-volume trend is that more expanded data leads to better performance.Source: Table III · p. 8, Figure 6 · p. 8, B. Results Comparison · pp. 9–10
  • Diffusion-based methods outperform ACT, while more sophisticated 3D perception architectures outperform the modified DP baseline.The comparison covers ACT, DP, DP3, EquiBot, and BiDP.Source: 3) Baselines · p. 8, B. Results Comparison · p. 10
  • In novel-object OOD tests, BiDP outperforms EquiBot and shows the least performance degradation among the compared methods.The paper attributes this advantage to geometric-transform demonstration expansion and object-only point-cloud inputs.Source: 4) Metrics · p. 8, Table VI · p. 9, B. Results Comparison · pp. 10–11
  • Real-robot visualizations show learned movements across five tasks; two additional tasks demonstrate one-shot action extraction and robot rollouts without further proliferation or policy training.The authors describe the framework as compatible with a range of bimanual tasks, while noting that these additional examples were not quantitatively evaluated.Source: Figure 7 · p. 10, B. Results Comparison · p. 11, Figure 8 · p. 11

VI. CONCLUSION AND LIMITATION

YOTO transfers one-shot human video demonstrations to dual-arm robots by extracting hand features, simplifying motions into keyframes, and scaling demonstrations for a bimanual diffusion policy. It is evaluated on five complex tasks, while the authors identify vision-related verification demands and hardware and task-scope limitations.

VI. CONCLUSION AND LIMITATION

  • YOTO extracts fine-grained hand features from one-shot video demonstrations, converts motion into keyframes, and uses auto-rollout and geometric transformations to generate diverse training examples.BiDP simplifies observations, predicts keyposes, and reorganizes action spaces for efficient training.Source: §VI · p. 11
  • Across five complex tasks, YOTO demonstrates performance in both synchronous and asynchronous settings.The authors describe the framework as a method for transferring human motions to robots and generating training data at scale.Source: §VI · p. 11
  • Vision-based hand-trajectory extraction still requires careful real-robot verification; the fixed workbench and parallel gripper constrain flexibility and functionality.The authors also identify specialized tool use, highly dynamic tasks, and interactive collaboration with people as under-explored.Source: §VI · p. 11

APPENDIX

The appendix details BiDP’s inputs, architecture, training and evaluation, and the selected tasks and object-processing pipeline.

1) Spaces of observation and action:

  • Each arm uses 13-dimensional proprioception and a 7-dimensional action space, comprising end-effector position, orientation, gravity direction, and gripper state or action.Proprioception includes position, a six-dimensional orientation representation, gravity direction, and gripper opening; actions include position offset, angular velocity, and gripper action.Source: 1) Spaces of observation and action · p. 17
  • The observation horizon is one initial left-arm observation, with the right arm’s initial state fixed; predicted action length is K for asynchronous tasks and 2K for synchronous tasks.The two synchronous tasks require the 2K prediction length, which the authors state is not reducible.Source: 1) Spaces of observation and action · p. 17

2) Network architecture:

  • BiDP uses a four-layer, 128-dimensional SIM(3)-equivariant PointNet++ encoder and an 8-step DDIM scheduler for faster inference.The noise-prediction network inherits hyperparameters from Diffusion Policy; DDIM is used instead of DDPM’s up to 100 denoising steps.Source: 2) Network architecture · p. 17

3) Sampling of point cloud:

  • Sampling 1,024 point-cloud points per observation speeds training without reported performance loss, while larger samples increase storage and training time.Increasing the point count to 2,048 or more yielded minimal evaluation improvement; all policy models train on a GeForce RTX 3090 Ti with 24 GB memory.Source: 3) Sampling of point cloud · p. 17

4) Training and evaluation:

  • Training uses 500 or 1,000 epochs for fully expanded demonstrations and 2,000 or 4,000 for under-expanded or unexpanded data; real-world evaluation uses randomly initialized object placements over dozens of episodes.The first two tasks use the lower epoch count in each pair and the last three tasks use the higher; the final checkpoint is evaluated, recording mean average length and success rate.Source: 4) Training and evaluation · p. 17
  • Florence-2 detection and SAM2 segmentation produce object masks for point-cloud observations, with inaccurate masks occasionally corrected manually.The authors exclude these manual corrections from failed evaluation trials because they do not involve significant elements of bimanual manipulation; Fig. 9 shows detection and segmentation examples.Source: 4) Training and evaluation · p. 17, Figure 9 · p. 17

B. Details of Our Selected Bimanual Tasks

  • The appendix also gathers details about the selected bimanual tasks, including object sizes, hand trajectories, keyframes, and valid workbench areas.Fig. 10 presents centimeter-level object size information.Source: B. Details of Our Selected Bimanual… · p. 17

1) Real size of manipulated objects:

The appendix documents the objects, extracted hand motions, keyframe choices, and placement areas used to characterize the selected tasks.

1) Real size of manipulated objects:

2) Hand motion extraction results:

3) Determination of keyframes in each task:

  • Keyframes can be extracted from gripper-state changes or velocity and acceleration extrema, but synchronous tasks need added intermediate frames for smoother, more stable actions.The simpler keyframe abstraction works effectively for the first three strictly asynchronous tasks but leaves too few keyframes for the latter two synchronous tasks.Source: 3) Determination of keyframes in each… · p. 19

4) Effective area on the workbench:

  • Object placement stays within the fixed manipulators’ reachable area, while geometric transformations spread training objects across that valid area without excessive displacement.Fig. 12 distinguishes predefined placement points from approximate augmented positions; test positions and orientations can still be new and out of distribution within the restricted area.Source: 4) Effective area on the workbench · p. 19, Figure 12 · p. 19

C. Evaluation Results and Performance Analysis

The appendix supplements the main evaluation with qualitative real-robot rollouts, two additional tasks, and examples of execution failures.

1) More qualitative examples:

  • Third-person rollout samples cover all five evaluation tasks and illustrate object-category and location variation without using those recordings for training or testing.The sequences show the two arms and more complete scenes, supplementing the binocular camera’s limited field of view.Source: 1) More qualitative examples · pp. 21–22

2) Rollouts of two new tasks:

  • Additional real-robot rollouts demonstrate reorienting pens into a cup and flipping a basket, representing reorientation and rearrangement skills.The basket task turns the non-prehensile basket 180 degrees so its mouth faces upward.Source: 2) Rollouts of two new tasks · p. 22

3) Analysis of failure cases:

  • BiDP still has real-world execution failures, including grasping errors, spills, object collisions, coordination failures, and box wings that rebound or fail to open.The examples span all five tasks and are presented as directions for future exploration.Source: 3) Analysis of failure cases · p. 22

Related papers

Loading 2501.14208v2…