Source-linked AI summary
ALOHA Unleashed: A Simple Recipe for Robot Dexterity
Tony Z. Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, Ayzaan Wahid
TL;DR
The paper asks how far imitation learning can be pushed for dexterous manipulation with deformable objects and complex contact dynamics. It combines large-scale ALOHA 2 data collection with Transformer-based Diffusion Policies and demonstrates dexterous behaviors across real and simulated tasks. The approach also shows recovery behaviors, while remaining limited by task-specific models, one-second replanning, and substantial demonstration requirements.
Problem
The paper investigates whether imitation learning can handle dexterous, long-horizon bimanual tasks involving deformable objects, complex contact dynamics, and high precision.
Method
The method collects large-scale teleoperated demonstrations and trains a separate Transformer-based Diffusion Policy for each task using multiview observations and proprioceptive state.
Results
The recipe demonstrates dexterous behaviors on 5 real-world and 3 simulated tasks, including autonomous shoelace tying and shirt hanging.
Takeaways & Limitations
Large-scale data collection combined with an expressive diffusion-based architecture can support challenging bimanual dexterous behaviors involving deformable objects.
Takeaways & Limitations
Policies are trained for one task at a time, replan every 1 second, and require many time-consuming human demonstrations per task.
Abstract
from arXiv · showhide
Recent work has shown promising results for learning end-to-end robot policies using imitation learning. In this work we address the question of how far can we push imitation learning for challenging dexterous manipulation tasks. We show that a simple recipe of large scale data collection on the ALOHA 2 platform, combined with expressive models such as Diffusion Policies, can be effective in learning challenging bimanual manipulation tasks involving deformable objects and complex contact rich dynamics. We demonstrate our recipe on 5 challenging real-world and 3 simulated tasks and demonstrate improved performance over state-of-the-art baselines. The project website and videos can be found at aloha-unleashed.github.io.
1 Introduction
The paper asks whether imitation learning can scale to dexterous manipulation involving deformable objects, complex contacts, long horizons, and precise bimanual coordination. It argues that large-scale data collection paired with an expressive diffusion-based architecture enables such policies on real and simulated tasks.
- Dexterous tasks remain difficult because they combine deformable objects, complex contact dynamics, many manipulation steps, high-dimensional bimanual coordination, and high precision.
- Scaling imitation learning to dexterous manipulation is uncertain because collecting sufficiently precise datasets covering system-state variation appears prohibitive.
- Over 26,000 demonstrations for 5 real tasks and over 2,000 demonstrations for 3 simulated tasks support the paper’s large-scale data-collection protocol.
- The approach combines diverse demonstrations with a transformer architecture trained using diffusion loss to denoise action trajectories from multiple views.
- The evaluation covers 5 real-world tasks, including shoelace tying and hanging clothes, plus 3 simulated tasks, while examining data complexity and out-of-distribution robustness.
2 Related Work
Related work spans imitation-learning architectures, bimanual manipulation, and methods for scaling real-world robot data collection. ALOHA Unleashed focuses these ideas on dexterity and robust handling of deformable and articulated objects.
- Imitation learning: Imitation-learning research has progressed from motor primitives to end-to-end models using visual backbones, sequence models, action tokenization, energy-based models, diffusion, and VAEs.
- Bimanual manipulation: Bimanual manipulation has been studied through model-based optimization, reinforcement learning, imitation learning, and keypoint-based modulation of motor primitives.
- Scale up robot learning in the real world: Real-world robot-learning datasets have been scaled through teleoperation, autonomous programming or reinforcement learning, and wearable or handheld devices.
- Scale up robot learning in the real world: The policy architecture uses camera-view embeddings, a Transformer encoder, and a Transformer decoder that iteratively denoises action chunks during inference.
- Scale up robot learning in the real world: The paper combines scalable data collection with dexterity-focused learning for deformable and articulated objects, targeting autonomous shoelace tying and shirt hanging.
3 Method
ALOHA Unleashed combines scalable bimanual teleoperation with a Transformer-based Diffusion Policy trained separately for challenging real and simulated tasks. The method uses multiview visual and proprioceptive inputs to generate action chunks for dexterous manipulation.
- ALOHA Unleashed pairs scalable teleoperation with a Transformer-based neural network trained using a Diffusion Policy formulation on the ALOHA 2 bimanual platform.
- Policy: Each task receives a separate Diffusion Policy that models multimodal actions from 4 camera views, proprioceptive state, and a 14-degree-of-freedom action space.
- Policy: The policy uses four ResNet50 vision backbones, a 85M-parameter Transformer encoder, and a 55M-parameter Transformer diffusion denoiser.
- Policy: Training uses JAX on 64 TPUv5e chips with batch size 256 for 2M steps, approximately 265 hours.
- Policy: Inference denoises sampled action chunks for 50 iterations and executes the resulting 50 actions open loop; a full pass takes 0.043 seconds on an RTX 4090 GPU.
- Data Collection: The data protocol enables 35 non-expert operators to collect over 26k episodes for 5 real tasks across 10 robots, 2 buildings, and 8 months.
4 Results
Across five real-world tasks, the policies learn dexterous behaviors including reorientation, retries, relative gripper control, mode switching, and precise visual insertions. Performance depends on demonstration quantity and quality, while Diffusion Policy outperforms L1 regression on ShirtMessy and most simulated tasks, and generalization remains bounded by the training distribution.
- Task Performance: The five-task evaluation measures complete-task success, with additional progress breakdowns for GearInsert and RandomKitchen.GearInsert reports success after inserting one, two, or three gears; performance generally decreases with each additional stage.
- Learned Dexterous Behaviors: The policies learn reorientation, recovery and retry behaviors, relative gripper control, mode switching, and millimeter-precision insertion using visual feedback.Examples include wrist-camera view augmentation for FingerReplace insertion and transitions from shirt flattening to hanger reaching or shoe straightening to loop tying.
- Data Ablations: Filtering ShirtEasy demonstrations raises success from 30% with all sampled episodes to 55% with the shortest 50%, then lowers it to 40% with only the shortest 25%.The shortest 25% contains 541 episodes, illustrating a trade-off between clean demonstrations and dataset size.
- Ablations: 25% success for L1 regression versus 70% for Diffusion Policy on ShirtMessy shows a substantial advantage for diffusion loss in this comparison.The comparison uses similarly sized 150M-parameter models and a well-tuned action-chunking L1 implementation.
- Ablations: Diffusion Policy outperforms ACT on all three simulated tasks except DoubleInsertion, with evaluations spanning different object-position initializations.The comparison uses XS-LowRes models and 50 rollouts, with three seeds for Diffusion Policy.
- Generalization: Generalization is demonstrated on unseen shirts, a different robot and building, and varied initial states, but failures occur for shirt and lace configurations absent from training.ShirtMessy handles tilted, wrinkled, and rightside-up shirts but usually fails on 180-degree or face-down shirts; Lace models fail on tipped, flipped, or tangled-shoe states.
5 Conclusion
ALOHA Unleashed presents a simple recipe combining large-scale demonstrations with a Transformer-based Diffusion Policy for dexterous behaviors in real and simulated environments. The approach is limited by per-task training, slow replanning, and substantial demonstration requirements.
- Over 26k demonstrations are collected on ALOHA 2 to train dexterous robot behaviors.
- A Transformer-based Diffusion Policy is trained on the collected data.
- The recipe demonstrates dexterous behaviors in both real and simulated environments.
- Policies are trained for only one task at a time rather than using one model for multiple tasks.
- The policy replans every 1 second, which may be too slow for highly reactive tasks.
- Many human demonstrations are required per task, making data collection time consuming.
A Additional Simulation Experiments
Additional simulation analyses examine evaluation protocols, model variants, chunk size, and diffusion sampling. Results indicate that ACT can outperform Diffusion in some settings, while task and data regime strongly affect the preferred configuration.
- Evaluation Protocol: Simulation results use 50 evaluations per checkpoint every 100k training steps, reporting the maximum evaluation score across checkpoints.
- Sim Analysis: ACT can outperform Diffusion in certain settings, while Diffusion generally performs better on most tasks and especially the real Shirt task.
- Sim Analysis: ACT with a smaller network is easier to tune and often performs better in low-data regimes.
- Evaluation Curves: The XS-LowRes simulation variant uses a single vision encoder per camera with images resized to 256x256.
- Chunk Size: 66 is the best single-seed score for chunk size 10, compared with 58 ± 3 averaged over three seeds for chunk size 50 on SingleInsertion.
- Chunk Size: Chunk-size tuning is task dependent, and chunk size 50 was used for real experiments because it showed good qualitative performance across tasks.
- Diffusion Steps: Reducing DDIM diffusion steps from 50 to 25 or 2 has little effect on this simulated task.
B.1 Protocol Documents
Operators receive task-specific collection instructions, with researchers reviewing initial test episodes before the remaining data are collected. Some tasks also require brief in-person strategy tutorials.
- Operators receive protocol instructions and collect 5–10 test episodes for researcher quality review.
- ShirtMessy and Lace operators receive a short in-person tutorial on the preferred task strategy.
- Operators collect the remaining data after the initial test episodes.
B.2 Initial State Distributions
The RandomKitchen evaluation uses varied initial states, with visualizations illustrating the range of starting configurations used for evaluation and data collection.
- Figure 8 visualizes 16 logged RandomKitchen initial states to show the task's initial-state variance.
- The varied initial states used for evaluation were also used to vary states during data collection across tasks.
C Hardware Variance
The paper measures base-position variation across 12 ALOHA 2 robots, using millimeter measurements tied to the workcell segments shown in Figure 9.
- Measurements of 12 ALOHA 2 robots quantify variation in their base positions.The measurements were taken several months after data collection and evaluation, though similar discrepancies existed during experiments.
- Figure 9 labels the workcell segments used to measure robot-position variance.
- Table 5 reports robot base-position measurements in millimeters and references Figure 9 for the measured segments.