Source-linked AI summary
Latent Action as Intention Enables Efficient Future Imagination for World Action Models
Xiang Li, Yupeng Zheng, Songen Gu, Huailiang Ma, Feng Yu, Xian Nie, Shanshuai Yuan, Yujie Zang, Weize Li, Shuai Tian, Moyang Liu, Ya-Qin Zhang, Wenchao Ding
TL;DR
WAMs face a trade-off between future-aware generalization and the latency of generating future observations. LAWA moves future imagination into compact latent actions, achieving stronger performance than Fast-WAM while matching Joint-WAM-level performance at lower latency. Across simulation, zero-shot, and real-world evaluations, the results support retaining future imagination through a compact latent-action pathway.
Problem
Future observation generation improves future-aware robot control but incurs substantial test-time latency, while removing it can reduce generalization, particularly with scarce demonstrations.
Method
LAWA represents future intentions with compact latent actions, trained using a mask-supervised tokenizer and action-free videos, then jointly denoises intentions with executable action chunks without future-video inference.
Results
Across RoboCasa, LIBERO-Plus, and real-world tasks, LAWA improves over Fast-WAM, retains Joint-WAM-level performance, and reduces inference latency relative to Joint-WAM by 42.9%.
Takeaways & Limitations
Future imagination need not be discarded for efficiency: compact latent actions provide an effective trade-off among performance, generalization, and latency.
Abstract
from arXiv · showhide
World action models (WAMs) improve robot control by modeling how observations evolve, but generating future observations at test time incurs substantial latency. Fast-WAM removes this process for efficiency; however, our matched implementations show lower generalization for Fast-WAM than for future-aware alternatives, especially with scarce robot demonstrations and in out-of-distribution scenarios. To bridge this gap, we introduce **LAWA**, a WAM architecture that uses compact latent actions as an operational representation of future intentions, enabling efficient test-time future imagination without generating future observations. Specifically, a discrete tokenizer enhanced by action-free pre-training produces manipulation-centric codebook targets. LAWA jointly denoises a continuous latent state anchored to these targets with executable action chunks while omitting the future-video branch at inference. On RoboCasa, LAWA achieves state-of-the-art average success rates of 65.6% and 80.8% in the few-shot and full data settings, improving over the matched Fast-WAM baseline by 9.6 and 4.5 points, respectively. It also preserves the performance level of the matched Joint-WAM variant while requiring 42.9% lower inference latency. LAWA also demonstrates competitive zero-shot robustness on LIBERO-Plus and superior performance on real-world tasks. These results show that future imagination need not be discarded: retaining it with compact latent actions yields an effective trade-off among performance, generalization, and latency. Code and models will be released.
1 Introduction
WAMs improve robot control by modeling future visual dynamics, but future observation generation is costly and removing it can weaken generalization. LAWA retains future-aware control through compact latent intentions, action-free video training, and inference without future-video generation.
- Future observation prediction helps represent task progress and dynamics, but iterative denoising creates a test-time bottleneck for robot control.
- Fast-WAM removes future-video denoising at inference but generalizes worse than matched Joint-WAM, especially with few-shot supervision.
- LAWA treats temporally structured latent actions as future intentions that encode task-relevant transitions without reconstructing all future visual details.
- LAWA jointly denoises latent intentions and action chunks at inference, discarding the future-video branch while preserving future-aware action generation.
- Action-free egocentric pre-training reverses LAWA’s initial disadvantage against Joint-WAM and provides complementary training diversity.
- 65.6% and 80.8% are LAWA’s RoboCasa average success rates in few-shot and full-data settings, exceeding matched Fast-WAM by 9.6 and 4.5 points.
2 Related Work
Related WAMs and latent-action methods explore future visual prediction, compressed transition representations, and action generation. LAWA combines these directions through latent intentions and a structured attention design that omits future-video denoising at inference.
- World Action Models: WAMs augment visuomotor policies with predictive modeling so actions can account for how scenes may evolve.
- Latent Actions for Robotic Manipulation: Latent-action methods compress visual transitions into action-like variables, enabling video pre-training without action labels.
- Latent Actions for Robotic Manipulation: LAWA treats latent actions as future intentions and uses them to retain future-aware action generation without explicitly predicting future observations.
3 Method
LAWA retains future imagination as compact latent-action intentions rather than generating future observations at inference, combining a tokenizer, latent-action modeling, and executable action prediction. Its two-stage training uses action-free pre-training followed by jointly trained video, latent-action, and action experts under structured attention.
- Latent Action Tokenizer: The tokenizer maps observation sequences to discrete latent-action tokens that capture dynamic transitions without action labels.A causal forward decoder reconstructs the next observation, forcing the representation to retain transition information beyond static appearance.
- Manipulation-centric Auxiliary Supervision: Mask-prediction supervision biases tokenizer representations toward hands, manipulators, and interaction dynamics that may be underrepresented by video reconstruction alone.The auxiliary decoder uses projected latent actions as prompts for predicting corresponding manipulation-centric masks.
- Action-free Egocentric Pre-training: Action-free egocentric pre-training expands tokenizer training beyond robot data while aligning motion-speed distributions across heterogeneous sources.During policy training, the pre-trained tokenizer is frozen and supplies target latent actions.
- Latent Action as Intention: LAWA predicts compact latent action sequences as future intentions, preserving Fast-WAM’s efficient inference interface without future-video generation.At inference, it encodes the current observation once and jointly denoises latent actions and executable actions from noise.
- Multi-model Joint Attention: LAWA jointly couples video, latent-action, and action experts through multi-model joint attention with modality-specific access constraints.Current-observation tokens cannot access the future, while latent-action and action tokens receive the contexts permitted by the structured attention mask.
- Training Objective: Training proceeds in two stages: tokenizer pre-training on action-free videos, then flow-matching optimization of the policy experts.The video, latent, and action terms respectively preserve world modeling, match codebook-derived transition targets, and learn executable control.
4 Experiments
Experiments evaluate LAWA on RoboCasa and LIBERO-Plus, including few-shot, full-data, zero-shot, ablation, attention, and latency analyses. Across these settings, LAWA improves over matched Fast-WAM, matches Joint-WAM performance on RoboCasa, and reduces inference latency relative to Joint-WAM.
- RoboCasa: LAWA achieves 65.6% few-shot and 80.8% full-data average success on RoboCasa, exceeding matched Fast-WAM by 9.6 and 4.5 points.These results use 100 demonstrations per task in the few-shot setting and full data across 24 tabletop tasks.
- Latent-action analysis: Perturbing LAWA’s latent-action state lowers RoboCasa success from 80.8% to 52.2% with Gaussian noise and to 56.4% with temporal shuffling.The interventions test whether action generation uses both latent-action content and temporal organization.
- Pre-training and ablations: Egocentric pre-training improves LAWA by 5.9 and 4.5 points in few-shot and full-data settings, compared with Fast-WAM gains of 1.5 and 1.7 points.Without egocentric pre-training, LAWA trails Joint-WAM by 3.4 and 2.0 points; after pre-training, it reaches 65.6% and 80.8%.
- Performance–latency Trade-off: 338.5 ms per action chunk is 42.9% lower than Joint-WAM’s 593.1 ms, while Fast-WAM is faster at 196.5 ms but has substantially lower success.Latency is measured on identical hardware, with Table 5 reporting results on one NVIDIA A800 GPU.
- Real-world tasks: LAWA outperforms Fast-WAM by 31.2, 36.3, and 33.8 points with 25%, 50%, and 100% of demonstrations, respectively, across four real-world tasks.In the 25% setting, LAWA reaches 40.0% average success versus Fast-WAM’s 33.8% with full training data, including 45.0% and 30.0% on Block and Laboratory.
5 Conclusion
LAWA preserves future-aware control by representing future intentions with compact latent actions instead of generating future observations. Across benchmarks, it improves efficiency and supports scalable learning from action-free videos.
- LAWA represents future intentions with compact latent actions, jointly denoising them with executable actions without observation prediction.This preserves future-aware action generation while avoiding costly visual prediction.
- LAWA improves over Fast-WAM while retaining Joint-WAM-level performance with 42.9% lower latency.
- LAWA supports scalable learning from action-free videos across RoboCasa, LIBERO-Plus, and real-world tasks.