Source-linked AI summary
Cross-Embodiment Robot Foundation World Models with Latent Actions
Huang Huang, Sriram Yenamandra, Arjun Majumdar, Elie Aljalbout, Tushar Nagarajan, Tsung-Yen Yang, Akshara Rai, Michael Rabbat, Li Fei-Fei, Jiajun Wu, Tingfan Wu, Franziska Meier
TL;DR
Diverse robot embodiments and action spaces make it challenging to build world models that generalize across robots. LAC-WM learns a shared latent action space and compares it with explicit-action conditioning. It improves reported downstream results and scales positively with more pretraining embodiments, unlike EAC-WM.
Problem
Robot embodiments have diverse structures, sensors, actuators, and control strategies, making it challenging to learn world models across embodiments and adapt to unseen robots.
Source: §1 · p. 1Method
LAC-WM learns unified latent actions from image transitions and uses an action projector to map raw actions from unseen embodiments into that space.
Source: §1 · p. 2Results
Takeaways & Limitations
LAC-WM's unified action representations support cross-embodiment generalization, and its downstream performance scales positively with the number of pretraining embodiments.
Source: §7 · p. 9Absolute task success remains limited on the dexterous manipulation benchmark.
Source: §7 · p. 9
Abstract
from arXiv · showhide
The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments. We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments. This unified action space improves the world model's performance when adapted to previously unseen robot embodiments. We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels. Our results show that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. We evaluate both models on dexterous manipulation tasks and a modified LIBERO benchmark. LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM's downstream performance to scale positively with the number of embodiments used during pretraining. In contrast, the disjoint action space in EAC-WM leads to decreased performance as the number of pretraining embodiments increases. These results highlight the importance of a unified action space for efficient cross-embodiment learning, addressing a key challenge in robotics. Project website: https://lacwm.github.io/
1. Introduction
Robot world models must learn across diverse embodiments and adapt to unseen robots. LAC-WM learns a shared latent action space, and its reported transfer results outperform explicit-action conditioning.
1. Introduction
- The paper proposes a unified latent action space aligning representations across human and robot embodiments, while retaining task-relevant information.It hypothesizes that motion-only encoders produce disjoint embodiment-specific spaces that limit transfer to unseen robots.Source: §1 · p. 1
- The work asks whether latent actions align cross-embodiment representations better than explicit labels, and whether unified actions improve world-model learning.The motivation is that diverse robot structures, sensors, actuators, and control strategies make cross-embodiment learning and adaptation challenging.Source: §1 · p. 1
- LAC-WM encodes consecutive image observations into latent actions for future-frame prediction, then uses an action projector to map explicit actions into that space.The architecture also decodes latent actions into motion labels during training.Source: §1 · p. 2, Figure 1 · p. 2
- LAC-WM improves task success by up to 46.7% on dexterous manipulation and raises modified-LIBERO average success from 82.4% to 92.0% over EAC-WM.Its downstream planning performance also scales positively with more pretraining embodiments, while EAC-WM shows the opposite trend.Source: §1 · p. 2
2. Related Work
Prior robot world models often avoid heterogeneous action conditioning or limit it to one embodiment. LAC-WM instead learns a unified action space from labeled cross-embodiment data and compares it with explicit action encoding.
World Models in Robotics
- Prior manipulation world models commonly use action-free pretraining or add action conditioning only during finetuning; Zhu et al. (2025) study action conditioning for one embodiment without cross-embodiment transfer.LAC-WM addresses heterogeneous action spaces with a unified latent space.Source: §2 · p. 2
Latent Action Modeling
- Earlier latent-action work explored game policies and robot policies trained from unlabeled human videos, while LAC-WM learns continuous actions from labeled cross-embodiment datasets for world models.Its motion-decoding loss follows Nikulin et al. (2025).Source: §2 · pp. 2–3
Latent Action World Models
- Prior latent-action world models did not combine diverse-embodiment pretraining with raw-action inference or demonstrate unseen-robot adaptation alongside explicit-action comparisons.LAC-WM uses labeled cross-embodiment data and compares learned latent actions with directly encoded motion labels.Source: §2 · p. 3
3. Method
LAC-WM learns latent actions from image transitions and uses them to predict future visual embeddings. An action projector then maps raw actions from an unseen robot into the learned space for imagined rollouts.
3. Method
- During pretraining, the IDM encodes image observations into latent actions for the FDM to predict the next image latent; the motion decoder reconstructs motion labels.Image-prediction and motion-decoding losses jointly supervise the modules.Source: §3 · p. 3
3.1. Latent Action Conditioned Robot World Model
- The IDM maps consecutive visual embeddings to latent actions, and the FDM uses the current embedding and latent action to predict the next embedding.Cross-augmentation uses independently augmented observation pairs to mitigate the risk that the IDM encodes future information directly into its latent action.Source: §3.1 · p. 3
- A motion decoder reconstructs ground-truth motion from latent actions, using continuous actions and an auxiliary decoding loss to mitigate shortcut solutions.The training objective combines image-latent prediction and supervised motion-decoding losses.Source: §3.1 · p. 3
3.2. Adapting to Unseen Robot Embodiments
- For an unseen embodiment, an action projector maps raw robot actions into the latent space; staged finetuning adapts the IDM, FDM, and projector before inference uses the projector and FDM for rollouts.The projector is a two-layer MLP; the FDM is frozen while the projector is first trained, then jointly finetuned with it.Source: §3.2 · p. 4
4. Experiments
The experiments compare explicit action encoders with a cross-embodiment setup pretrained on three datasets and fine-tuned on an unseen bimanual robot. In transfer visualizations, IDM embeddings produce trajectories closer to references than the other tested embeddings.
4.1. Baseline
- EAC-WM encodes explicit robot end-effector actions or human hand poses with separate action encoders, then adapts to an unseen embodiment using a new encoder and LoRA fine-tuning of the FDM.The explicit-action baseline uses the same FDM architecture as LAC-WM.Source: §4.1 · p. 4
4.2. Model Pretraining
- Pretraining uses EgoDex, Agibot, and Droid, sampling 50,000 trajectories from each and chunking actions into five-step sequences.The datasets include human manipulation, a bimanual humanoid robot, and a Franka arm; action labels include robot end-effector actions and human hand poses.Source: §4.2 · p. 4
4.3. Model Finetuning
- Fine-tuning evaluates adaptation on a bimanual Franka setup with Allegro hands, using a 22-task RoboCasa dataset of object-picking and placement trajectories.LAC-WM uses a three-stage fine-tuning procedure; EAC-WM uses a single stage, which the authors report performs better than its multi-stage alternative.Source: §4.3 · pp. 4–5
- IDM embeddings yield trajectories close to reference motions, while EAE and IDM without motion decoding show inaccurate patterns and visual artifacts.The figure compares robot-to-human and human-to-robot transfer.Source: Figure 3 · p. 5
5. Latent Action Analysis
The analysis finds that IDM embeddings form a more unified cross-dataset action space than explicit action embeddings, and transfer more consistently between robot and human motions. Motion decoding further improves the coherence of the learned representations.
UMAP Visualization
- IDM embeddings cluster across all three datasets, whereas EAE and shared EAE* embeddings remain dataset-separated, indicating that explicit action embeddings do not share a common representation.The comparison uses 7,000 validation examples; EAE* tests whether separation is due to separately initialized encoders.Source: §5 · p. 5
- Without motion decoding, Agibot and EgoDex embeddings cluster together apart from Droid; the authors hypothesize that similar egocentric views explain this pattern.They suggest visual-only latent actions may encode perceptual features rather than task-relevant information, risking disjoint representations across differing visual distributions.Source: §5 · p. 5
Action Latent Transfer
- In 8-frame transfer tests, IDM embeddings produce semantically similar movements across embodiments, while EAE transfers poorly and IDM without motion decoding shows visual artifacts.With motion decoding, IDM embeddings capture embodiment-invariant motion without residual source-embodiment artifacts.Source: §5 · p. 6
Latent Action Analysis
- The authors conclude that latent action models unify representations better than direct action labels, and that motion decoding further improves action-space coherence.They also report PCA analysis of latent actions in Section A.7.Source: §5 · p. 6
6. Robot Task Evaluation
The evaluation tests imagined rollouts and world-model-guided action selection on dexterous manipulation and unseen LIBERO tasks. LAC-WM leads EAC-WM in prediction and planning, with scaling results favoring unified actions as pretraining diversity grows.
6.1. Action-conditioned Imagined Roll-out
- Across unseen instances and categories, LAC-WM achieves the best image and video quality, followed by EAC-WM, while EAC-WM-S performs worst.The evaluation reports image-quality metrics and the video-quality metric FVD.Source: Table 1 · p. 6, §6.1 · p. 6
6.2. Robot Planning with Action Selection
- For dexterous planning, the world model rolls out candidate action sequences toward subgoals and selects sequences by predicted-image embedding distance.The procedure repeats across subgoals; the reported metrics include task success and subgoal embedding distances.Source: §6.2 · pp. 6–7
- As pretraining embodiment count grows, LAC-WM's planning performance improves, while EAC-WM's success rate decreases and its δf and δfg increase or remain nearly unchanged.The models are evaluated after fine-tuning on BFA; results are averaged over three seeds.Source: Figure 4 · p. 7, §6.2 · p. 8
- With either more embodiments at fixed sample count or more samples across fixed embodiments, LAC-WM improves; EAC-WM degrades with added embodiments and improves only with more same-embodiment data.Figure 5 contrasts adding a third embodiment at fixed trajectories with scaling from 24k to 150k trajectories across three embodiments.Source: Figure 5 · p. 7, §6.2 · p. 8
- LAC-WM-guided action selection significantly outperforms VLA-only baselines, particularly on S.R.L and S.R., and improves over EAC-WM-S and EAC-WM.The evaluation covers 100 episodes per split across all tasks, with three random seeds.Source: §6.2 · p. 7, Table 2 · p. 8
- On the Unseen Categories validation split, LAC-WM achieves a 38% higher success rate than the variant trained without motion decoding loss or cross-augmentation.The ablation compares LAC-WM with LAC-WM-MD-CA across three seeds.Source: Table 3 · p. 8, §6.2 · p. 8
- 92.0% versus 82.4% with EAC-WM: LAC-WM achieves a 9.6-point absolute gain in average success rate and leads on all five unseen LIBERO tasks.The comparison uses 125 total episodes across the five tasks.Source: Table 4 · p. 9, §6.2 · p. 9
7. Conclusion and Limitations
LAC-WM learns more unified representations and scales positively with more pretraining embodiments, unlike EAC-WM; absolute dexterous task success remains limited in unseen settings.
7. Conclusion and Limitations
- LAC-WM learns more unified representations and outperforms EAC-WM in planning, with downstream performance rising as pretraining embodiments increase while EAC-WM declines.The comparison uses latent-action conditioning for LAC-WM and explicit action labels for EAC-WM.Source: §7 · p. 9
- Absolute task success remains limited on the dexterous manipulation benchmark, reflecting the difficulty of dexterous manipulation in unseen settings.The authors identify stronger VLA policies and more robust planning as directions for improvement.Source: §7 · p. 9
- Motion labels in pretraining enable direct comparison with explicit-action supervision, while auxiliary motion decoding promotes coherent latent actions without requiring labeled data.The authors say the approach is compatible with, and likely to benefit from, large-scale label-free video pretraining.Source: §7 · p. 9
- Real-world data collection and hardware validation are future work for evaluating LAC-WM in physical environments.Source: §7 · p. 9
A.1. Pretraining Dataset Details
Pretraining uses three datasets with embodiment-specific pose formats, and the appendix illustrates how the resulting world models predict future observations.
A.1. Pretraining Dataset Details
- EgoDex, Agibot, and Droid provide motion labels; EAC-WM receives concatenated end-effector pose and camera-motion vectors.Source: §A.1 · p. 13
- Wrist poses use 9 dimensions (3 translation, 6 rotation); Agibot uses 20 end-effector dimensions across two hands, while Droid uses 10 for one arm.Source: §A.1 · p. 13
- EgoDex uses 138 end-effector dimensions across both hands, including 60 finger-position dimensions; camera motion is 9D for EgoDex and Agibot, and zero for Droid.Source: §A.1 · p. 13
- LAC-WM's motion decoder assigns the latent action's first half to end-effector pose and its second half to camera pose.Source: §A.1 · p. 13
- LAC-WM accurately predicts future observations in imagined rollouts, while EAC-WM-S and EAC-WM show excessive robot-arm motion.Source: Figure 6 · p. 13
A.2. BFA Dataset
The BFA data description covers synthetic pick-and-place generation and the dataset's trajectory composition; an example figure shows model-based action-sequence selection.
A.2. BFA Dataset
- MimicGen and RoboCasa generate synthetic pick-and-place data from 10 teleoperation trajectories, with objects placed left or right at least 0.1 meter from their initial position.Source: §A.2 · p. 13
- The dataset contains 7,265 success and failure trajectories, 20% successful, from two Franka arms mounted on a mobile platform.Source: Figure 7 · p. 14
- In the action-selection example, three sequences yield eight future predictions, and the sequence with the smallest predicted-goal embedding distance is selected.The top-row sequence is selected in the illustrated example.Source: Figure 7 · p. 14
A.3. Image decoder
A custom Vision Transformer–based decoder reconstructs pixels from V-JEPA2 embeddings, with pretraining on DROID followed by fine-tuning on BFA.
A.3. Image decoder
- The decoder maps 16 × 16 tokens of dimension 1408 for a 256 × 256 image into a 1024-dimensional decoder space, then processes them with 12 self-attention layers and 32 heads each.Source: §A.3 · p. 14
- After self-attention, features are linearly mapped back to pixel space and trained with mean-squared error loss.Source: §A.3 · p. 14
- The pixel decoder is pretrained to reconstruct DROID images and then fine-tuned on the BFA dataset.Source: §A.3 · p. 14
A.4. Model Training and Inference
The appendix describes model training and inference, training-data action selection, the dexterous-manipulation VLA, and latent-action PCA analysis. The PCA examples indicate meaningful motion patterns across embodiments.
A.4. Model Training and Inference
- Both models are pretrained for 80,000 iterations, with temporal attention added for the final 20,000; training takes about four days on 64 H200 GPUs.Inference takes approximately 0.1 seconds per batch of 10 while predicting eight future frames at 256×256 resolution on an NVIDIA L40S; model hyperparameters are listed in Table 5.Source: §A.4 · pp. 14–15, Table 5 · p. 14
A.5. Action Selection from Training Dataset
- Training-dataset action sequences are used to avoid potential bias from VLA-generated samples and compare models under multiple fine-tuning strategies.The evaluation uses 50 episodes per split, repeated with three random seeds; the strategies include staged and end-to-end fine-tuning variants.Source: §A.5 · p. 15
- In Figure 8, LAC-WM selects actions that complete the pick-and-place task, while the other compared methods fail to grasp and lift the object.The comparison includes VLA-mean, VLA-random, EAC-WM-S, and EAC-WM.Source: Figure 8 · p. 16
A.6. Vision-Language Action Model for Dexterous Manipulation
- The dexterous VLA fuses vision, language, proprioception, and prior actions, then uses a diffusion head to predict future end-effector poses and finger angles.The visual and language encoders are pretrained and frozen; proprioception and action encoders and the three-layer transformer trunk are trained from scratch.Source: §A.6 · p. 16
- The policy predicts 100-step action sequences at 15 Hz, deploys each chunk before replanning, and is trained on 2,000 successful pick-and-place demonstrations.Source: §A.6 · p. 16
A.7. Latent Action PCA Analysis
- PCA projections and clustering of latent-action sequences from three pretraining datasets reveal consistent, meaningful motion patterns across embodiments.Representative cluster examples include motions such as opening or closing a gripper and moving left or right.Source: §A.7 · p. 17, Figure 9 · p. 17