PaperlayerView exact revision on arXiv

Source-linked AI summary

World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis

Yi Yang, Zhihong Liu, Siqi Kou, Yiyang Chen, Yanzhe Hu, Jianbo Zhou, Boyuan Zhao, Zhijie Wei, Xiao Xia, Xueqi Li, Pengfei Liu, Zhijie Deng

arXiv:2606.05979v1cs.ROcs.AI

TL;DR

World-action models largely predict visual next states, restricting semantic reasoning and extrapolation. WLA combines textual intentions and physical dynamics in an autoregressive framework with World and Action Experts. WLA-0 shows strong multi-task performance and state-of-the-art results on memory-dependent tasks, nearly doubling the best baseline on RMBench [14].

  • Problem

    Current world-action methods focus almost exclusively on predicting next visual states, restricting their capacity for semantic reasoning and extrapolation.

    Source: §1 · p. 1

  • Method

    WLA uses an autoregressive backbone to model textual intentions and physical dynamics, with meta-query signals guiding World and Action Experts to predict visual states and executable actions.

    Source: §1 · pp. 1–2, §5 · p. 9

  • Results

    WLA-0 shows strong multi-task performance, state-of-the-art results on memory-dependent manipulation, and favorable inference efficiency, nearly doubling the best baseline on RMBench [14].

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Learning new tasks from action-free videos suggests a promising direction for scalable cross-embodiment robot learning.

    Source: §5 · p. 9

    Real-world experiments cover only a small set of bimanual tasks on one robot platform, and video-based task-learning experiments rely on simulated robot videos.

    Source: §6 · p. 9

Abstract

from arXiv · show

We propose world-language-action (WLA) models as a new class of embodied foundation models. WLA takes textual instructions, images, and robot states as inputs to jointly predict textual subtasks, subgoal images, and robot actions, conjoining the \emph{world modeling interface} to learn from extensive egocentric videos as in the world-action model (WAM) and the \emph{language reasoning} capacities to solve complex long-horizon tasks as in vision-language-action (VLA) models. At the core of WLA lies an \emph{autoregressive (AR)} Transformer backbone, instead of a bidirectional diffusion Transformer as in WAMs, to predict the \emph{next state}, comprising the \emph{semantic-level} textual intention and complementary \emph{fine-grained} physical dynamics. The physical dynamics are supervised by the world modeling objective based on a dedicated World Expert, and are leveraged to ease the characterization of the state-action correlation for the Action Expert. WLA leverages meta-queries to make the world prediction \emph{implicitly} impact the action generation so that the former can be disabled during inference. The world prediction can also be activated to enable test-time scaling for improved robot control. Our WLA-0 prototype, with 2B active parameters, achieves 40 ms per inference on an NVIDIA RTX 5090. Evaluations across simulated and real-world environments demonstrate that WLA-0 achieves state-of-the-art multi-task and long-horizon learning abilities, e.g., 92.94\% success rate on RoboTwin2.0 Clean and 56.5\% success rate on RMBench. WLA-0 also holds the promise to learn novel tasks directly from \emph{cross-embodiment robot videos} without action annotations.

1 Introduction

WLA addresses WAMs’ limited semantic reasoning and extrapolation by predicting both high-level textual intention and low-level physical dynamics. Its autoregressive backbone connects this next-state prediction to action synthesis.

Introduction

  • Unlike bidirectional-diffusion WAMs [42] [46] and VLAs [27], WLA uses an autoregressive Transformer to predict textual subtasks and physical dynamics that guide actions.Textual intention provides a generalizable abstract representation, while physical dynamics describe transitions between visual states.Source: §1 · p. 1
  • A dedicated World Expert predicts subsequent visual states from current states and physical dynamics, enabling end-to-end training of latent transitions without ground-truth dynamics.Meta-queries condition the World Expert and guide the Action Expert to produce executable actions.Source: §1 · p. 2
  • A lightweight SANA-600M diffusion Transformer predicting static future frames suffices, and implicit parameter updates let WLA disable the World Expert during inference.WLA-0 runs at approximately 40 ms per inference on an RTX 5090.Source: §1 · p. 2
  • WLA-0 matches leading WAMs on simulation benchmarks, halves baseline Stack Cup completion time, and sets RMBench SOTA at nearly twice the best baseline.It has 2B running parameters and no pretraining; test-time scaling further improves simulation performance.Source: §1 · p. 2
  • WLA-0 learns new tasks from cross-embodiment robot videos without action annotations, showing promising steerability and cross-embodiment generalization.WLA can use heterogeneous data with or without action annotations.Source: §1 · pp. 1–2

2 Related Work

Related work advances embodied control through visual world prediction and language-guided planning, while identifying limitations in language reasoning and physical-dynamics supervision. WLA unifies world modeling, language-guided planning, and action prediction.

2.1 World Modeling for Policy Learning

  • Policy-learning work moved from explicit inverse dynamics [19] [20] [30] to visual-prediction CoT reasoning [48] and implicit inverse dynamics guided by video diffusion [23] [40].Visual prediction serves as an intermediate reasoning step for action generation [47] [56] [57].Source: §2.1 · pp. 2–3
  • WAMs use pretrained video-generation models [28] [31] or MoT architectures [5] [38] [53], learning interaction priors from web and egocentric videos [52].Most WAMs rely on video-generation backbones without language generation, whereas WLA uses a VLM backbone to preserve native language reasoning and planning.Source: §2.1 · p. 3

2.2 Language-Guided Embodied Control

  • VLA models leverage pretrained VLM representations [4] [18] [26] and language-based CoT and hierarchical planning [24, 39, 54] to bridge goals to actions, but struggle with physical dynamics and visual supervision [32].WLA unifies world modeling, language-guided planning, and action prediction.Source: §2.2 · p. 3

3 Methodology

WLA unifies multimodal instruction, observation, and robot-state inputs to predict textual intentions, future visual states, and action chunks. Its world model trains action generation implicitly, while inference can disable that model for efficiency or activate it to compare candidate actions.

3 Methodology

  • WLA maps images, text, and robot states to textual intentions, future observations, and n-step action chunks executed in a receding-horizon loop.It supports heterogeneous supervision from image-text pairs, robot demonstrations, and egocentric videos.Source: §3 · p. 3

3.1 World-Language-Action Models

  • An autoregressive Transformer predicts next states through complementary high-level textual intentions and low-level physical dynamics.The textual stream guides behavior semantically, while the dynamics representation grounds intentions in motion patterns.Source: §3.1 · p. 3
  • WLA predicts temporally aligned subtasks spanning the upcoming action horizon and uses a recursively updated memory buffer as long-horizon context.The backbone is initialized from a pretrained VLM, and joint training uses language cross-entropy plus world-modeling and action flow-matching losses.Source: §3.1 · pp. 3–4
  • A World Expert predicts future visual states from compact physical-dynamics representations produced by causal-attention meta-queries appended to the autoregressive backbone.The World Expert also receives the original-state representation.Source: §3.1 · p. 4
  • The World Expert predicts VAE features for only the target future frame, can optionally predict depth maps, and leaves fine-grained visual detail to the expert.The paper reports that full-clip supervision slows training without improving performance.Source: §3.1 · p. 4
  • Shared-parameter learning lets the world-model objective guide action generation without conditioning on predicted future images at inference, so the World Expert can be discarded to reduce latency.This is the paper’s implicit alternative to image-then-act inference.Source: §3.1 · p. 4

3.2 Inference Optimization

  • In efficient mode, WLA disables the World Expert and achieves approximately 40 ms inference latency on an RTX 5090.The paper also describes additional acceleration techniques in Appendix A.Source: §3.2 · p. 5
  • In test-time scaling mode, WLA samples candidate action chunks, predicts their future frames, and executes the chunk with the highest value-model score.The value model can help reject potentially failing trajectories in imagined space before they affect the real environment; the imagination horizon can be extended autoregressively.Source: §3.2 · p. 5

4 Experiments

WLA-0 is evaluated on simulated and real-world manipulation benchmarks, where it demonstrates strong task performance and inference efficiency. Experiments also test whether video supervision enables learning novel tasks, including across robot embodiments.

4.1 Implementation Details

  • WLA-0 uses a 2.1B-parameter RynnBrain backbone, a SANA-600M World Expert, and a 390M flow-matching Action Expert, with 64 meta-queries.The prototype has 3.4B total parameters; action chunks are 8 for LIBERO and 32 elsewhere.Source: §4.1 · p. 5

4.2 Evaluation in Simulation Environment

  • WLA-0 achieves a 92.94% success rate in clean RoboTwin 2.0 environments and remains robust under domain randomization.The World Expert ablation indicates that future-state prediction refines action generation; WLA-0 matches or outperforms prior pipelines while using fewer parameters and no embodied pretraining.Source: §4.2 · p. 6
  • WLA-0 achieves 98.6% average success on LIBERO, rising to 98.9% with test-time scaling using 6 candidates and an imagination horizon of 2.The model outperforms all WAM and VLA baselines and shows strong performance after 30k training steps.Source: §4.2 · p. 6
  • WLA-0 achieves a 56.5% average success rate on RMBench, outperforming Fast-WAM and nearly doubling Mem-0.Removing subtask prediction loss reduces average success from 56.5% to 17.25%, supporting the value of language-level subtask reasoning and memory tracking.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Real-World Experiments

  • In real-world trials, WLA-0 performs comparably to pretrained baselines, generalizes across OOD objects and scenarios, and achieves the highest success rate on Dispose Trash.The paper attributes this dynamic-task result to historical context, future-state prediction, and low inference latency.Source: §4.3 · p. 7, Figure 3 · p. 7
  • On Stack Cup, with all models using 10 flow-matching steps and action chunks of 32 (Motus generates one frame per call), WLA-0 achieves the lowest completion time and inference latency, reducing latency by ∼40× versus Motus.Source: §4.3 · p. 7

4.4 Learning New Tasks from Videos

  • Adding unseen-task video supervision nearly triples baseline success and enables WLA-0 to acquire the novel “beat” action solely from video observations.Unseen cross-embodiment video maintains a highly competitive success rate, demonstrating transfer from ARX-X5 videos to the Aloha-AgileX setting.Source: §4.4 · p. 8, Table 3 · p. 8, Figure 4 · p. 8

5 Conclusion

WLA unifies world modeling, language reasoning, and action synthesis through an autoregressive backbone with World and Action Experts. Experiments report strong multi-task performance, state-of-the-art memory-dependent manipulation results, favorable inference efficiency, and potential for learning from action-free videos.

5 Conclusion

  • WLA integrates world modeling, language reasoning, and action synthesis using an autoregressive backbone with World and Action Experts.It models semantic-level textual subtasks and fine-grained physical dynamics to enable long-horizon reasoning and real-time robot control.Source: §5 · p. 9
  • Experiments show WLA-0 achieves strong multi-task performance, state-of-the-art results on memory-dependent manipulation tasks, and favorable inference efficiency.The conclusion also reports real-time robot control.Source: §5 · p. 9
  • WLA-0’s ability to learn new tasks from action-free videos suggests a direction for scalable cross-embodiment robot learning.Source: §5 · p. 9

6 Limitations · A Acceleration Techniques

WLA’s real-world evaluation remains narrow, while inference optimizations reduce latency from ∼116 ms to under 40 ms. The appendix describes CUDA graph replay, operator fusion, and precomputation and caching as complementary optimizations.

6 Limitations

  • Real-world experiments cover a small set of bimanual tasks on one robot platform, while video-based task learning uses simulated robot videos.The authors call for broader evaluation across embodiments and task domains to establish generality.Source: §6 · p. 9

A Acceleration Techniques

  • Inference optimizations reduce WLA-0 latency from ∼116 ms to under 40 ms.The bottlenecks include Python dispatch overhead and launching many small CUDA kernels, especially in the iterative DiT denoising loop.Source: A Acceleration Techniques · p. 13
  • CUDA Graph replay captures a forward pass with fixed-address GPU buffers and replays it to remove per-step Python dispatch and reduce kernel-launch overhead.This is especially beneficial for the multi-step DiT head.Source: A Acceleration Techniques · p. 13
  • Custom Triton kernels fuse VLM RMSNorm [55], QKV projection with RoPE, SwiGLU [44], and decoder layers, plus DiT AdaLayerNorm [42], attention, and GELU [22].Fusion reduces launch overhead and intermediate memory traffic; DiT fusion also covers position-embedding addition.Source: A Acceleration Techniques · p. 13
  • Precomputation and caching reuse invariant VLM embeddings, masks, RoPE tables, and image indices, as well as DiT encodings, timestep outputs, and AdaLN parameters.Cross-attention K/V tensors are also cached and reused across denoising steps for a prediction.Source: A Acceleration Techniques · p. 13

B Simulation Benchmarks · B.1 RoboTwin 2.0

The RoboTwin 2.0 benchmark setup distinguishes instruction splits across methods and specifies WLA-0’s inference procedure and action representation.

B.1 RoboTwin 2.0

  • LingBot-VA [31] is evaluated with seen instructions, while the other methods use unseen instructions in Table 1.Per-task results are reported in Table 7.Source: §B.1 · p. 13
  • WLA-0 uses 32 flow-matching inference steps because preliminary experiments found that fewer steps can induce robotic-arm jitter.Source: §B.1 · p. 13
  • WLA-0 represents actions as absolute end-effector positions.Source: §B.1 · p. 13

B.2 LIBERO

On LIBERO, preliminary experiments find that multi-frame World Expert prediction underperforms single-frame prediction, potentially because dense visual supervision interferes with action learning.

B.2 LIBERO

  • Multi-frame prediction achieves substantially lower success than single-frame prediction, suggesting dense visual supervision may slow convergence and interfere with action learning.For n = 32, the single-frame model predicts only o_t+32, while the multi-frame model predicts o_t+8, o_t+16, o_t+24, and o_t+32; both were trained for 30k steps with global batch size 256.Source: §B.2 · p. 13

B.3 RMBench

On RMBench, WLA-0 achieves the best average success rate. Its progress tracking is more tightly synchronized with action generation than Mem-0’s separate transition-classifier approach.

B.3 RMBench

  • WLA-0 achieves the best average success rate on RMBench.The passage does not provide the numerical rate or the full comparison values.Source: §B.3 · p. 13
  • WLA-0 repeatedly infers the current executable subtask before acting, supporting more stable progress tracking than Mem-0’s separate Subtask End Classifier.Mem-0 can switch subtasks incorrectly when its classifier detects subtle transitions too early or too late, disrupting subsequent actions.Source: §B.3 · p. 14

C Real-World Experiments

Real-world robot experiments represent actions with the arm’s joint angles and use 10-step flow-matching inference; results and World Expert images are presented in Table 5 and Figure 10.

C Real-World Experiments

  • Real-world experiments use joint-angle actions and 10-step flow-matching inference, with results in Table 5 and generated World Expert images in Figure 10.The passage does not provide the results or image contents.Source: C Real-World Experiments · p. 14

D Learning New Tasks from Videos

WLA-0 did not learn five unseen RoboTwin 2.0 tasks from human egocentric videos, and the authors conjecture that a human-to-simulation domain gap caused the failure. Their video-training setup mixed unseen-task videos with seen-task data at a 1:1 ratio.

D Learning New Tasks from Videos

  • Adding human egocentric videos did not enable WLA-0 to learn five unseen RoboTwin 2.0 tasks.The authors conjecture that a domain gap between real-world human videos and simulation was primarily responsible, and plan to better align the data in future work.Source: D Learning New Tasks from Videos · p. 14, Table 6 · p. 15
  • For unseen-task video training, models used a 1:1 seen-data-to-video sampling ratio, a 0.1 video loss weight, and 50k training steps.The human-video experiment used 100 egocentric manipulation videos for each of the five unseen tasks, mixed with seen-task data.Source: D Learning New Tasks from Videos · p. 14

Related papers

Loading 2606.05979v1…