Source-linked AI summary
World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis
Yi Yang, Zhihong Liu, Siqi Kou, Yiyang Chen, Yanzhe Hu, Jianbo Zhou, Boyuan Zhao, Zhijie Wei, Xiao Xia, Xueqi Li, Pengfei Liu, Zhijie Deng
TL;DR
Existing world-action models emphasize next visual-state prediction, limiting semantic reasoning and extrapolation. WLA combines textual intentions, physical dynamics, and action synthesis in an autoregressive framework, achieving strong multi-task performance, state-of-the-art memory-dependent manipulation results, and favorable inference efficiency.
Problem
Existing world-action models focus on next visual-state prediction, burdening models with low-level details and restricting semantic reasoning and extrapolation.
Method
WLA uses an autoregressive backbone with World and Action Experts to jointly model textual intentions, physical dynamics, and executable actions through meta-query conditioning.
Results
WLA-0 achieves strong multi-task performance, state-of-the-art results on memory-dependent manipulation tasks, and favorable inference efficiency.
Takeaways & Limitations
Learning new tasks from action-free videos suggests a promising direction for scalable cross-embodiment robot learning.
Takeaways & Limitations
Real-world experiments cover only a small set of bimanual tasks on one robot platform, while video-based task learning relies on simulated robot videos.
Abstract
from arXiv · showhide
We propose world-language-action (WLA) models as a new class of embodied foundation models. WLA takes textual instructions, images, and robot states as inputs to jointly predict textual subtasks, subgoal images, and robot actions, conjoining the \emph{world modeling interface} to learn from extensive egocentric videos as in the world-action model (WAM) and the \emph{language reasoning} capacities to solve complex long-horizon tasks as in vision-language-action (VLA) models. At the core of WLA lies an \emph{autoregressive (AR)} Transformer backbone, instead of a bidirectional diffusion Transformer as in WAMs, to predict the \emph{next state}, comprising the \emph{semantic-level} textual intention and complementary \emph{fine-grained} physical dynamics. The physical dynamics are supervised by the world modeling objective based on a dedicated World Expert, and are leveraged to ease the characterization of the state-action correlation for the Action Expert. WLA leverages meta-queries to make the world prediction \emph{implicitly} impact the action generation so that the former can be disabled during inference. The world prediction can also be activated to enable test-time scaling for improved robot control. Our WLA-0 prototype, with 2B active parameters, achieves 40 ms per inference on an NVIDIA RTX 5090. Evaluations across simulated and real-world environments demonstrate that WLA-0 achieves state-of-the-art multi-task and long-horizon learning abilities, e.g., 92.94\% success rate on RoboTwin2.0 Clean and 56.5\% success rate on RMBench. WLA-0 also holds the promise to learn novel tasks directly from \emph{cross-embodiment robot videos} without action annotations.
1 Introduction
The introduction proposes world-language-action (WLA) models, which combine textual intention, physical dynamics, world modeling, and action synthesis through an autoregressive Transformer. WLA-0 uses end-to-end meta-query conditioning and demonstrates efficient, generalizable, long-horizon robot control, including learning from cross-embodiment videos without action annotations.
- Motivation: WLA models predict next states as both textual intentions and low-level physical dynamics, connecting compact semantic representations to fine-grained motion control.Textual intentions provide abstract, generalizable future-state representations, while physical dynamics describe transitions between visual states.
- WLA Architecture: WLA uses an autoregressive Transformer to generate textual subtasks and model next states, contrasting with WAMs’ bidirectional diffusion Transformer architecture.The design inherits language modeling and context-management capabilities from existing vision-language models.
- Learning Framework: A dedicated World Expert supervises physical dynamics by predicting subsequent visual states, while meta-queries condition world prediction and guide the Action Expert end to end.This avoids the need for ground-truth low-level dynamics and replaces explicit latent-action two-stage training with an end-to-end framework.
- Efficiency: ∼40 ms inference latency on an RTX 5090 enables real-time adaptation, while disabling the World Expert during inference preserves efficiency.WLA-0 predicts static future visual frames with a lightweight diffusion Transformer rather than full video clips.
- Results: WLA-0 achieves strong generalization and long-horizon performance, including state-of-the-art results on RMBench and learning new tasks from cross-embodiment videos without action annotations.The introduction attributes RMBench gains to language-based planning, memory use, and error correction, and reports robustness in dynamic and out-of-distribution settings.
2 Related Work
Related work combines world modeling and embodied control through visual-state prediction, inverse dynamics, and video representations, while WAMs extend this direction with large-scale video training. VLA models add language for instruction following and long-horizon planning, motivating WLA’s unified approach.
- World Modeling and Action Generation: World-modeling methods enhance embodied policy learning by predicting future visual states and using explicit inverse dynamics or visual prediction as intermediate reasoning.Later approaches exploit internal video representations and diffusion models to guide action prediction through implicit inverse dynamics.
- World Action Models: WAMs combine world modeling and action generation using pretrained video-generation models or Mixture-of-Transformers architectures for task understanding, video prediction, and robot control.Training on web and egocentric videos provides interaction priors that improve downstream generalization and data efficiency.
- World Action Models: Most WAMs use video-generation backbones without language generation, limiting high-level planning and reasoning.This limitation motivates WLA’s integration of language capabilities with world modeling and action generation.
- Vision-Language-Action Models: VLA models leverage pretrained vision-language representations to support instruction following, high-level planning, behavioral steering, and movement beyond reactive control.Subsequent work shows language can bridge high-level goals and low-level control in long-horizon tasks through chain-of-thought reasoning and hierarchical approaches.
3 Methodology
WLA unifies multimodal world modeling, language-based subtask reasoning, and action generation with an autoregressive Transformer that predicts textual intentions and physical dynamics. Its World Expert, Action Expert, implicit training interface, and optional test-time scaling support efficient or more computationally intensive robot control.
- Unified Multimodal Modeling: WLA maps images, text, and robot states to images, text, and robot actions, supporting heterogeneous supervision from image-text pairs, robot demonstrations, and egocentric videos.This formulation combines the strengths of world-action models and vision-language-action models.
- Next-State Prediction: The AR Transformer predicts next states through complementary high-level textual intentions and low-level physical dynamics.Textual intentions provide a semantic blueprint for state evolution, while physical dynamics capture state transitions.
- Textual Intention Learning: Textual intention learning decomposes user instructions into intermediate textual subtasks, using a pretrained VLM backbone and a recursively updated memory buffer for long-horizon context.The memory buffer is updated by appending newly predicted subtasks to historical context.
- Physical Dynamics Modeling: The World Expert models compact physical dynamics with meta-queries and predicts future visual states from the current state representation.It predicts VAE feature representations for the target frame, optionally alongside depth maps, rather than semantic features or the full video clip.
- Implicit World-Action Interface: World modeling guides action generation through shared parameter learning, allowing the World Expert to be discarded during inference instead of conditioning actions on explicitly predicted future images.This implicit interface reduces test-time latency relative to image-then-act world-action models.
- Test-Time Scaling: ∼40 ms inference latency enables real-time action prediction on an RTX 5090 in efficient mode, while test-time scaling samples candidate action chunks and executes the highest-valued imagined trajectory.The value model scores World-Expert-predicted future frames, enabling rejection of potentially failing trajectories before real-environment execution.
4 Experiments
Experiments show that WLA-0 performs strongly across simulated multi-task, long-horizon, and real-world manipulation settings. Its language-based reasoning, memory utilization, world modeling, and video supervision support robust control and transfer, including from cross-embodiment videos.
- RoboTwin 2.0: 92.94% success rate is achieved by WLA-0 in clean RoboTwin 2.0 environments across 50 coordinated dual-arm tasks.The evaluation averages success rates over 100 trials per task; language-based subtask prediction is disabled because of the benchmark’s short horizons.
- LIBERO: 98.6% average success is achieved on LIBERO, outperforming all WAM and VLA baselines.A single model is trained across Spatial, Object, Goal, and Long suites, with strong performance observed after only 30k training steps.
- RMBench: 56.5% average success rate makes WLA-0 the best-performing model on RMBench’s long-horizon, memory-dependent bimanual tasks.WLA-0 substantially outperforms Fast-WAM and nearly doubles Mem-0, while removing the subtask prediction loss tests the contribution of language-based supervision.
- Real-World Experiments: WLA-0’s real-world performance benefits from historical context and future-state prediction, enabling adaptation to rotating-bin dynamics under synchronous execution.Motus loses track of the rotating bin because of high inference latency, whereas π0.5 misestimates turntable velocity without history conditioning.
- Cross-Embodiment Video Transfer: +Unseen Same-Emb. Video nearly triples the baseline success rate, while +Unseen Cross-Emb. Video remains highly competitive on five unseen RoboTwin 2.0 tasks.The model acquires the novel “beat” action solely from video observations and aligns visual observations with actionable control knowledge across embodiments.
5 Conclusion
WLA unifies world modeling, language reasoning, and action synthesis through an autoregressive backbone with World and Action Experts. Experiments show strong multi-task and memory-dependent manipulation performance, favorable inference efficiency, and potential for learning new tasks from action-free videos.
- 5 Conclusion: WLA integrates world modeling, language reasoning, and action synthesis in a unified embodied framework.Its autoregressive language backbone uses World and Action Experts to model textual subtasks and physical dynamics.
- 5 Conclusion: WLA-0 achieves strong multi-task performance, state-of-the-art memory-dependent manipulation results, and favorable inference efficiency.These capabilities support long-horizon reasoning and real-time robot control.
- 5 Conclusion: Learning new tasks from action-free videos suggests a promising direction for scalable cross-embodiment robot learning.The conclusion identifies this capability as a basis for scaling learning across robot embodiments without action annotations.
6 Limitations · A Acceleration Techniques
The paper identifies limited real-world evaluation and simulated-only video supervision as current limitations. An acceleration stack combining CUDA Graph Capture, operator fusion, and precomputation/caching reduces WLA-0 latency from approximately 116 ms to under 40 ms.
- 6 Limitations: WLA’s real-world evaluation covers only a small set of bimanual tasks on one robot platform, limiting established generality across embodiments and domains.The paper calls for broader evaluations across diverse embodiments and task domains.
- 6 Limitations: WLA’s video-based task learning experiments currently use simulated robot videos for supervision.The paper presents cross-embodiment video learning as requiring further validation beyond this setting.
- A Acceleration Techniques: ∼116 ms to under 40 ms: three complementary optimizations reduce WLA-0 inference latency dominated by Python dispatch and many small CUDA-kernel launches.The bottlenecks are especially pronounced in the iterative DiT denoising loop.
- A Acceleration Techniques: CUDA Graph Capture replaces eager execution with replayed graphs using fixed-address GPU buffers, removing per-step Python dispatch and reducing kernel-launch overhead.The method is especially beneficial for the multi-step DiT head.
- A Acceleration Techniques: Operator Fusion uses custom Triton kernels to combine adjacent operations, reducing launch overhead and intermediate memory traffic in the VLM and DiT.Fused components include normalization, projections, attention, activation, positional, and decoder-layer computations.
- A Acceleration Techniques: Precomputation and Caching reuse invariant VLM quantities, DiT quantities, and cross-attention K/V tensors across denoising steps.Cached or precomputed items include embeddings, masks, RoPE tables, action-time encodings, timestep-MLP outputs, and AdaLN parameters.
B Simulation Benchmarks · B.1 RoboTwin 2.0
The RoboTwin 2.0 simulation benchmark evaluates methods under different instruction settings and reports per-task results separately. For WLA-0, the setup uses 32 flow-matching inference steps and absolute end-effector position actions.
- B Simulation Benchmarks: RoboTwin 2.0 evaluates LingBot-VA under seen instructions, while other methods use unseen instructions.This distinction applies to the comparisons in Table 1.
- B.1 RoboTwin 2.0: Per-task RoboTwin 2.0 results are reported separately in Table 7.The passage distinguishes these per-task results from the main evaluation setting.
- B.1 RoboTwin 2.0: WLA-0 uses 32 flow-matching inference steps for the RoboTwin 2.0 evaluation.The stated configuration is part of the benchmark implementation.
- B.1 RoboTwin 2.0: Preliminary experiments found that using fewer inference steps can induce robotic-arm jitter.This observation motivates the 32-step configuration.
- B.1 RoboTwin 2.0: WLA-0 represents actions using absolute end-effector positions.The passage specifies the action representation used in the benchmark.
- B.1 RoboTwin 2.0: The benchmark implementation therefore combines a 32-step flow-matching procedure with absolute end-effector position actions.These are the two implementation details explicitly specified for WLA-0.
B.2 LIBERO
In LIBERO, the study compares single-frame and multi-frame World Expert prediction targets for action learning, finding that multi-frame supervision substantially reduces success rate and may interfere with convergence.
- World Expert prediction target: Multi-frame prediction achieves a substantially lower success rate than single-frame prediction, suggesting dense visual supervision can slow convergence and interfere with action learning.With n = 32, the single-frame model predicts only o_t+32, while the multi-frame model jointly predicts o_t+8, o_t+16, o_t+24, and o_t+32; both use 30k training steps and global batch size 256.
B.3 RMBench
On RMBench, WLA-0 achieves the best average success rate. Its advantage over comparison methods is attributed to synchronizing progress tracking with action generation, whereas others lack explicit memory traces and language-level progress planning.
- B.3 RMBench: WLA-0 achieves the best average success rate on RMBench.RMBench illustrations cover Battery Try, Blocks Ranking Try, Cover Blocks, and Press Button.
- B.3 RMBench: π0.5, X-VLA, and Fast-WAM generate actions from visual observations and instructions but lack explicit memory traces and language-level progress planning.These limitations make it difficult to infer the current executable subtask when the next action depends on previous trials.
- B.3 RMBench: WLA-0’s advantage over Mem-0 mainly comes from tighter synchronization between progress tracking and action generation.Mem-0 instead relies on a separate Subtask End Classifier to detect visually subtle subtask transitions.
C Real-World Experiments
Real-world experiments represent actions as robot-arm joint angles and use 10-step flow-matching inference, with results reported in Table 5 and World Expert-generated images visualized in Figure 10.
- Experimental Setup: Real-world actions are represented by the robot arm’s joint angles, with flow-matching inference performed in 10 steps.Full results are reported in Table 5.
- World Expert Visualization: Figure 10 visualizes images generated by the World Expert during inference.These visualizations accompany the real-world experiments.
D Learning New Tasks from Videos
WLA-0 is evaluated for learning unseen tasks from unseen-task videos, including human egocentric videos that depict real-world props resembling RoboTwin 2.0 objects. The evaluation compares Seen-Action with an augmented setting using unseen human-egocentric videos across five unseen RoboTwin 2.0 tasks.
- Training setup: Training uses a 0.1 loss weight for unseen-task videos, a 1:1 seen-to-unseen video sampling ratio, 50k steps, and global batch size 256.These settings apply to both unseen same-embodiment and unseen cross-embodiment video training.
- Human egocentric videos: The human-video evaluation collects real-world props resembling objects in the RoboTwin 2.0 simulation environment.This tests whether unseen tasks can be learned from human egocentric videos.
- Unseen-task evaluation: WLA-0 is tested on five unseen RoboTwin 2.0 tasks by comparing Seen-Action against +Unseen Human-Ego Video.The augmented setting uses unseen-task human egocentric videos.
- Task examples: One illustrated RoboTwin-style task repeatedly rearranges three colored cubes and presses a button until the arrangement succeeds.The button and cubes are initially arranged in a random row on the table.