Source-linked AI summary
Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation
Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, Sergey Levine
TL;DR
Robot learning lacks a simple way to train one policy across robots with different sensors, actuators, and control frequencies. CrossFormer uses a flexible transformer over 900K trajectories from 20 embodiments, and real-world evaluations show specialist-level performance while outperforming prior cross-embodiment learning. Its current results do not yet demonstrate significant positive transfer across embodiments.
Problem
Robot platforms and tasks often have limited data, while heterogeneous sensors, actuators, and control frequencies complicate training one policy across embodiments.
Method
CrossFormer tokenizes variable observations and task specifications, processes them with a shared transformer, and uses action-specific heads to produce variable-dimensional actions without manual space alignment.
Results
CrossFormer matches specialist policies and significantly outperforms prior cross-embodiment learning across 900K trajectories spanning 20 robot embodiments.
Takeaways & Limitations
A single network can control single- and dual-arm systems, wheeled robots, quadcopters, and quadrupeds while performing comparably to specialist methods.
Takeaways & Limitations
The results do not yet show significant positive transfer across embodiments, and larger models may face inference-speed limits for high-frequency embodiments.
Abstract
from arXiv · showhide
Modern machine learning systems rely on large datasets to attain broad generalization, and this often poses a challenge in robot learning, where each robotic platform and task might have only a small dataset. By training a single policy across many different kinds of robots, a robot learning method can leverage much broader and more diverse datasets, which in turn can lead to better generalization and robustness. However, training a single policy on multi-robot data is challenging because robots can have widely varying sensors, actuators, and control frequencies. We propose CrossFormer, a scalable and flexible transformer-based policy that can consume data from any embodiment. We train CrossFormer on the largest and most diverse dataset to date, 900K trajectories across 20 different robot embodiments. We demonstrate that the same network weights can control vastly different robots, including single and dual arm manipulation systems, wheeled robots, quadcopters, and quadrupeds. Unlike prior work, our model does not require manual alignment of the observation or action spaces. Extensive experiments in the real world show that our method matches the performance of specialist policies tailored for each embodiment, while also significantly outperforming the prior state of the art in cross-embodiment learning.
1 Introduction
Cross-embodied policies can exploit diverse robot data, but varying sensors, actions, and control frequencies make one general policy difficult to train. CrossFormer addresses this with a flexible transformer and matches specialist performance across substantially different robots.
- General-purpose robot policies can transfer visual representations and skills across embodiments while reducing engineering for robot-specific policy design.
- Robot systems differ widely in camera views, proprioception, joint configurations, action outputs, and control frequencies, challenging unified policy training.
- CrossFormer tokenizes variable observations and uses action readout tokens with embodiment-specific heads to predict actions of any dimension without manual action-space alignment.
- 900K trajectories across 20 embodiments support a policy spanning robots from quadrupeds with proprioception and 12 joints to bimanual systems with 3 cameras and 14 joints.
- Real-world experiments find performance matching target-only training and prior specialist methods, while outperforming prior cross-embodiment learning without negative transfer.
2 Related Work
Prior cross-embodied learning used varied strategies but generally operated at smaller scale or imposed shared observation and action spaces. CrossFormer extends flexible pretraining to more heterogeneous embodiments, environments, and real-world tasks.
- Earlier approaches used embodiment representations, domain randomization, modular policies, or model-based reinforcement learning, typically with limited data, simulation, or robot coverage.
- Several multi-robot methods require robots to share the same observation and action spaces, restricting heterogeneous sensor and actuator combinations.
- CrossFormer targets simultaneous control of robots with disjoint sensor and actuator sets, including robot arms and quadrupeds.
- Prior flexible transformer policies handled varying spaces but focused mainly on robot arms, whereas CrossFormer evaluates navigation, quadrupedal locomotion, high-frequency bimanual arms, and broader real-world settings.
- CrossFormer trains on a 900K-trajectory subset of OXE together with navigation, Franka, and quadruped data, extending beyond prior single-arm subsets.
3 Designing a Cross-Embodied Policy
CrossFormer represents heterogeneous observations and actions as token sequences processed by a shared transformer, with modality-specific encoders and action heads. This design supports varied embodiments, task specifications, and action chunk sizes.
- Designing a Cross-Embodied Policy: The central design challenge is accommodating varying sensors, action representations, and control frequencies without discarding observations or aligning action spaces.
- Designing a Cross-Embodied Policy: A transformer serializes all available observation types and decodes variable-length actions, while action chunking improves temporal consistency and reduces compounding error.
- Designing a Cross-Embodied Policy: Observations and task specifications are tokenized by modality, processed by a shared causal decoder-only transformer, and mapped through embodiment-specific action heads.
- 3.1 Training data: The training mixture spans 20 embodiments with widely varying observation spaces, action spaces, and control frequencies.
- 3.2 Tokenizing variable observation types and task specifications: Images and proprioception are tokenized across variable camera configurations, while tasks can be specified with language instructions or goal images.
- 3.3 Predicting variable length actions: Action readout tokens feed continuous-action L1 heads, with readout-token counts matched to embodiment-specific action chunk sizes.
- Evaluation settings: The evaluation settings cover manipulation, navigation, and aviation across a wide range of tasks and embodiments.
- 3.4 Training details: Missing observations are masked while observation and readout token groups retain fixed context-window locations for each batch element.
4 Evaluation
CrossFormer is evaluated across diverse real-world robot tasks against single-robot training, task-specific prior methods, and Yang et al. It performs comparably to target-only training and outperforms the prior cross-embodiment method in the reported settings.
- Evaluation design: The evaluation tests whether one policy matches target-only training and the best prior imitation-learning method across multiple robot embodiments.The comparisons include CrossFormer, a target-robot-only version of the same architecture, and setting-specific prior methods.
- Comparison to Yang et al.: CrossFormer outperforms Yang et al. by 3x overall on third-person tabletop manipulation and common navigation tasks.Yang et al. aligns actions for navigation and manipulation and uses only one camera view at a time.
- Cross-embodiment evaluation: 73% average success for CrossFormer versus 67% for the single-robot dataset baseline across embodiments.The reported average excludes no embodiments from the CrossFormer-versus-single-robot comparison.
- Cross-embodiment evaluation: 73% average success for CrossFormer versus 51% for the best prior method, excluding Go1 because no suitable prior imitation-learning method exists.CrossFormer performs similarly to the best prior method in each evaluation setting according to the authors.
- Comparison to Yang et al.: With equal LoCoBot training data, CrossFormer significantly outperforms Yang et al. across cornering, obstacle avoidance, and sharp-turning scenarios.The authors also report smoother navigation, straighter lines, and fewer stops and starts.
5 Discussion and Conclusion
The paper concludes that CrossFormer controls many robot embodiments with specialist-level performance and improves over prior cross-embodiment learning, while identifying transfer, data mixing, and inference-speed limitations.
- Conclusion: CrossFormer matches specialist policies across single-arm, dual-arm, wheeled, aerial, and quadrupedal embodiments while outperforming prior cross-embodiment learning.The same policy is reported to control these diverse embodiments using one set of network weights.
- Limitations: The experiments do not yet show significant positive transfer across embodiments.The authors anticipate greater positive transfer from larger datasets containing more embodiments.
- Limitations: Hand-picked sampling weights are used to avoid over-training repetitive datasets and under-training data relevant to evaluation settings.The authors suggest larger models might eventually fit all data equally well without weighting.
- Limitations: Scaling model size can limit inference speed for high-frequency robots, despite successful application to fine-grained bimanual manipulation.The paper identifies further research and hardware improvements as needed for controlling high-frequency embodiments with large models.
A Training Data
CrossFormer’s training mixture combines datasets from Open X-Embodiment with additional project-collected data, using sampling weights that up-weight several target datasets.
- Sampling weights: Bridge, ALOHA-multi-task, GNM, Go1-walk, and Franka-tabletop receive increased sampling weights.
- Training data: The training mixture uses datasets from Open X-Embodiment and additional data collected for this project.
B Training Hyperparameters
The model uses a 130M-parameter transformer system with pretrained ResNet-26 image encoders and is trained with image augmentation and goal-relabeled observations.
- Architecture: The complete model has 130M parameters, including ResNet-26 image encoders and action heads.The ResNet-26 encoders are initialized with ImageNet-pretrained weights.
- Training procedure: Training uses color jitter, random resizing and cropping, hindsight goal relabeling, and uniformly sampled future observations as goals.
- Training procedure: Training took 80 hours on a TPU V5e-256 pod.
C Evaluation Setups
The evaluation spans manipulation, navigation, quadrupedal locomotion, and quadcopter control, using embodiment-specific sensors, action heads, and task conditions. It includes a zero-shot quadcopter setting and reports task-specific success or normalized reward metrics.
- Evaluation platforms and tasks: Evaluations cover single-arm, bimanual, wheeled navigation, quadrupedal walking, and quadcopter corner-turning tasks.The setups include WidowX, Franka, ALOHA, LoCoBot, Go1, and Tello platforms.
- Observation and action setups: The manipulation experiments use over-the-shoulder views for single-arm systems and three camera views for ALOHA bimanual control.WidowX and Franka use single-arm heads, while ALOHA uses one overhead and two wrist cameras with a bimanual head.
- Navigation setup: LoCoBot evaluation tests path-following, obstacle avoidance, and sharp corner-turning with a graph-based planner and distance function.The policy uses one camera view and a navigation head within the planner-based setup.
- Locomotion setup: Go1 evaluation reports reward over 25 minutes normalized by the reward achieved by the RL-trained expert policy that generated the data.The policy directly controls the quadruped’s joints rather than higher-level navigation waypoints.
- Metrics: For ground and aerial navigation, the success metric is the proportion of subgoals reached on the path toward the goal.The evaluation table also reports the number of trials in parentheses and uses normalized reward for Go1.
- Aerial navigation setup: Tello evaluation requires zero-shot generalization because the policy is not trained on quadcopter data, and it tests corner turning in three locations.The navigation head outputs 2-D relative waypoints while maintaining a static height.
D Evaluation Results
Table 3 provides a detailed, task-level breakdown of the real-world evaluation results. It compares CrossFormer with target-robot training and the best prior method on target-robot data.
- Detailed results: Table 3 reports real-world evaluation results broken down by task.The passage identifies the table as a detailed breakdown without specifying individual task outcomes.