PaperlayerView exact revision on arXiv

Source-linked AI summary

World-Action Models for Robot Learning and Control: A Survey

Zuxing Lu, Hongjia Zhai, Guanzhi Wang, Huajian Zeng, Jiaqi Yang, Jingyu Liu, Lei Cheng, Yuantai Zhang, Yuheng Qiu, Zezhou Cheng, Ivan Laptev, Danfei Xu, Benjamin Riviere, Giuseppe Loianno, Eric Xing, Xingxing Zuo

arXiv:2609.16074v1cs.ROcs.CV

TL;DR

Robots need to anticipate action consequences, yet reactive policies often lack explicit future-state modeling and separate predictive models can mismatch control utility. This survey defines WAMs and synthesizes their designs, applications, and evaluation; it concludes that useful predictions must be action-grounded, consistent, uncertainty-aware, and suitable for closed-loop control.

  • Problem

    Reactive policies often omit how alternative actions affect future states, while separate predictive models can mismatch predictive fidelity and control utility.

    Source: §I · pp. 1–2

  • Method

    The survey defines WAMs and organizes their designs, applications, evaluation protocols, and open challenges through a robotics-oriented synthesis.

    Source: §I · p. 2

  • Results

    The survey concludes that useful predictions must be grounded in robot actions, preserve spatial and temporal consistency, expose uncertainty, support closed-loop improvement, and satisfy real-time constraints.

    Source: §VII · p. 15

  • Takeaways & Limitations

    Progress depends on coupling multimodal and 3D representations, action alignment, neural simulation, reinforcement learning, real-robot feedback, and benchmarks measuring task success, safety, robustness, and transfer.

    Source: §VII · p. 15

Abstract

from arXiv · show

Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs), which couple future world prediction with executable action generation. This survey provides a robotics-oriented review of WAMs. We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. We further review applications of WAMs in manipulation, navigation, and autonomous driving, and we summarize the datasets, benchmarks, metrics, and protocols used to evaluate WAM systems. Finally, we discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference. Taken together, this survey aims to provide a concise technical foundation for integrating predictive world modeling with action generation, toward more reliable embodied robot intelligence. Project page: https://rcl-robotics.github.io/Awesome-World-Action-Models.

I. INTRODUCTION

This survey frames World-Action Models as a way to connect prediction of future states with executable robot actions, then organizes their design, applications, evaluation, and open challenges. It also situates WAMs among world models and reactive VLA policies.

I. INTRODUCTION

  • Modular systems can be brittle under distribution shift, while reactive policies often omit how alternative actions change future states or task progress, –[39],,,,.The survey identifies these gaps in open-world, long-horizon, and contact-rich settings.Source: §I · pp. 1–2
  • WAMs couple predictive world representations with executable action generation, jointly predicting future states and actions or inferring actions through inverse dynamics [19], [22],, [61]–[63].The survey distinguishes this predictive structure from reactive VLA policies and conventional model-based reinforcement learning.Source: §I · p. 2
  • The taxonomy organizes methods by components, transitions, action interfaces, architectures, training, data modalities, and scaling, spanning latent dynamics, video-action, diffusion, inverse-dynamics, and unified multimodal systems.Source: §I · p. 2
  • The survey reviews manipulation, navigation, autonomous driving, and generalist robotics, distinguishing prediction used for representation learning, planning, synthetic data, or imagined policy optimization.Source: §I · p. 2
  • It summarizes datasets, benchmarks, metrics, and protocols, and identifies challenges in action grounding, spatial consistency, neural simulation, real-time inference, transfer, and safe deployment.Source: §I · p. 2

A. Definition

  • The formulation treats interaction as a POMDP and distinguishes world models’ future-state prediction from VLAs’ direct action generation and WAMs’ joint prediction of state trajectories and actions.Latent states encode interaction histories for predicting dynamics, while latent actions bridge high-level commands and low-level motor control.Source: A. Definition · p. 3

B. Foundational Research Threads

World models and video generative models contribute complementary foundations for WAMs: latent-dynamics prediction for planning and future visual-scene prediction from history and language.

B. Foundational Research Threads

1) World Model and Model-based Reinforcement Learning:

  • MBRL uses a dynamics model for planning and policy optimization, while world models learn latent dynamics from states and actions.Representative world models use deterministic and stochastic latent states, with later explicit and implicit approaches learning latent dynamics.Source: 1) World Model and Model-based… · p. 3

2) Video Generative Models:

  • Video generative models predict future visual observations from text, images, or video history, providing visual priors for long-horizon prediction and planning.They are characterized as visual predictive planners rather than explicit dynamics models.Source: 2) Video Generative Models · p. 3

3) Latent Action Pre-training:

Latent-action pretraining learns motion representations from visual transitions, while VLAs generate actions directly from visual and language inputs; WAMs bring future-state prediction and action generation together.

3) Latent Action Pre-training:

  • Latent Action Pretraining learns latent actions from visual state transitions to support downstream policy learning, including from action-free videos.The inferred latent actions encode motion dynamics in a structured space described as transferable across domains and embodiments.Source: 3) Latent Action Pre-training · pp. 3–4, Figure 2 · p. 4

4) Vision-Language-Action Models:

  • VLAs map visual observations and language instructions to actions, using autoregressive sequence modeling or flow matching to model action distributions.The cited formulation conditions action generation on visual and language inputs.Source: 4) Vision-Language-Action Models · p. 4

C. From These Threads to World-Action Models

  • WAMs unify dynamics modeling, visual prediction, latent-action learning, and direct action generation by modeling future observations and the actions responsible for them.Their generative process can be decomposed into policy generation, forward dynamics, inverse dynamics, and visual planning.Source: C. From These Threads to World-Action… · p. 4
  • WAMs generate actions either jointly with future observations or after first generating a future visual plan and inferring actions through inverse dynamics.The two pathways form the basis of the survey's transition-modeling paradigms.Source: C. From These Threads to World-Action… · p. 4

III. ARCHITECTURE DESIGN & MODEL TAXONOMY

The taxonomy organizes WAM architectures by components and learning paradigms, with language encoders conditioning tasks and visual encoders defining predictive representations.

III. ARCHITECTURE DESIGN & MODEL TAXONOMY

  • WAMs are organized by components, transition paradigms, architectures, training pipelines, and data modalities and scaling strategies.The component view identifies language interface, visual encoder, predictive backbone, and action decoder as functional modules.Source: §III · p. 4, Figure 3 · p. 5

A. Model Components

  • The component decomposition separates general representation from task-specific action mechanisms, supporting a consistent basis for architecture comparison.Source: A. Model Components · p. 5

1) Language Encoder:

  • Language encoders provide instruction-conditioning representations through global text encoders, dense sequence encoders, or autoregressive LLMs.These categories are exemplified by CLIP, T5, and LLMs, respectively.Source: 1) Language Encoder · p. 5

2) Visual Encoder:

  • Visual encoders compress observations while preserving semantic, spatial, and temporal information for WAM prediction.Approaches include 2D and video VAEs, pretrained representation encoders, 3D or multi-view encoders, and reused VLM backbones.Source: 2) Visual Encoder · p. 5

3) Model Backbone:

WAM designs vary in their temporal backbone and action decoder, and in whether future observations and actions are generated jointly or through inverse dynamics.

3) Model Backbone:

  • WAM backbones either learn multimodal dynamics with unified Transformers from scratch or use pretrained video foundation models as temporal priors.The latter shifts emphasis toward aligning visual predictions with control.Source: 3) Model Backbone · p. 5

4) Action Decoder:

  • Action decoders use discretized action tokens, latent actions inferred from state transitions, or continuous diffusion and flow-matching action heads.Tokenization can lose precision, while diffusion and flow-based heads have higher inference overhead.Source: 4) Action Decoder · pp. 5–6

B. Transition Modeling Paradigms

  • Transition modeling follows joint prediction, which couples observation and action generation, or inverse dynamics, which predicts future states before inferring actions.These are described as complementary paradigms.Source: B. Transition Modeling Paradigms · p. 6, Figure 4 · p. 6

1) Joint Prediction:

  • Joint prediction generates the next action and observation conditioned on the current observation and instruction, unifying action generation with future-state prediction.It can function as a direct policy and as an auxiliary learning signal that enriches action representations through foresight; later systems include coordinated video-and-action diffusion and alignment of pretrained generative dynamics with embodiment-specific control.Source: 1) Joint Prediction · p. 6

2) Inverse Dynamics Model:

Inverse-dynamics WAMs predict future observations before inferring actions. WAM architectures also vary in whether they unify or separate world and action processing, while training commonly uses pre-training followed by post-training.

2) Inverse Dynamics Model:

  • Inverse-dynamics WAMs first predict the next observation from the current state and language instruction, then infer the action from that transition.This factorized “plan-then-act” approach contrasts with joint prediction, which generates future observations and actions simultaneously.Source: 2) Inverse Dynamics Model · p. 6

C. Architecture Structures

  • The architecture taxonomy crosses two choices: a unified backbone versus separate visual and action experts, and joint prediction versus sequential inverse dynamics.End-to-end systems process modalities through a shared pathway; dual systems separate world modeling and action generation through an intermediate interface.Source: C. Architecture Structures · p. 6, Figure 5 · p. 7, 2) Dual Systems · p. 7

1) End-to-end Systems:

  • End-to-end WAMs jointly optimize world modeling and action generation, with video-generation backbones and multimodal alignment strategies supporting this unified design.Cosmos Policy represents non-image signals as latent-frame-like representations, while ViPRA learns latent actions from video before aligning them with robot visual representations.Source: 1) End-to-end Systems · p. 7

2) Dual Systems:

  • Dual-system WAMs separate predictive world representations from action generation, enabling modularity and interface-level controllability at the cost of added boundary complexity.Common interfaces include MoT-style routing and cross-attention transfer to an action-decoding module.Source: 2) Dual Systems · p. 7

D. Policy Training Pipeline

  • WAM training generally has two stages: pre-training learns spatiotemporal and action representations from large-scale video, then post-training adapts the model through fine-tuning, augmentation, and reinforcement learning.The figure depicts this pipeline as pre-training followed by post-training.Source: D. Policy Training Pipeline · p. 7

1) Pre-Training:

WAM pre-training develops spatiotemporal and action representations and aligns video with actions. Post-training adapts pretrained models for deployment, while modality and data scale shape the information and supervision available to them.

1) Pre-Training:

  • Pre-training combines spatiotemporal representation learning, action representation learning, and video-action alignment.Methods use video backbones and multi-view training, explicit or latent action representations, and alignment objectives including forward dynamics, inverse dynamics, action generation, and video generation.Source: 1) Pre-Training · pp. 7–8

2) Post-Training:

  • Post-training adapts pretrained WAMs through policy fine-tuning, data augmentation, and reinforcement learning, including optimization through imagined rollouts.Fine-tuning can adapt large backbones with parameter-efficient methods; generated trajectories can add appearance variation or recover pseudo-actions for policy learning.Source: 2) Post-Training · pp. 8–9

E. Data Modality, Scaling, and Mixture

  • Data modality determines what information WAMs observe, while data scale influences learned-dynamics richness and the availability of action supervision.The data-pyramid view contrasts broad world priors from large action-free video corpora with action grounding from smaller embodied trajectories.Source: E. Data Modality, Scaling, and Mixture · p. 9, Figure 7 · p. 9

1) Data Modality:

  • Language, multi-view video, 3D geometry, contact and auditory feedback, and proprioception provide complementary task or state information for WAM decisions.Multi-view prediction consistency remains challenging, while tactile and auditory signals can reveal interaction events that are difficult to infer from vision alone.Source: 1) Data Modality · p. 9

2) Data Scaling and Mixture:

WAM training draws on internet videos, egocentric demonstrations, and embodied trajectories, which contribute visual dynamics priors or direct action supervision. Applications span manipulation and navigation, using video-action policies, structured state models, imagined policy interactions, and predictive look-ahead.

2) Data Scaling and Mixture

  • Internet videos and egocentric demonstrations mainly provide visual-dynamics priors, whereas embodiment-specific trajectories supply direct action supervision.Embodied datasets provide action-conditioned supervision across tasks, embodiments, and contact-rich interactions, bridging broad visual priors and controllable prediction.Source: 2) Data Scaling and Mixture · p. 10

IV. APPLICATIONS

  • The survey organizes WAM applications by manipulation, navigation, and autonomous driving, where prediction serves different roles across domains.Those roles include representation learning, imagined look-ahead at inference, and synthetic trajectories for policy improvement.Source: §IV · p. 10

A. Robot Manipulation

  • Manipulation is the most common WAM testbed because it combines visual prediction with physically constrained action.Useful rollouts must model object states, contact dynamics, occlusions, and task progress, not only visual plausibility.Source: A. Robot Manipulation · p. 10

1) Video-Action and Unified Policies:

  • Video-action and unified manipulation policies use predictive pre-training to connect future observations and robot actions through trajectory-level representations.Some systems jointly generate observations and actions; others use predicted future states or goals as intermediate control targets, while predictive objectives can improve latent representations without online video rollout.Source: 1) Video-Action and Unified Policies · p. 10

2) Structured State Models:

  • Geometry- and physics-oriented models predict control-relevant representations when RGB rollouts underspecify contact-rich manipulation.Examples model dynamic scenes with Gaussian representations, deformable interactions with particles, point-cloud dynamics, or physics-informed constraints; other models capture contact, compliance, or event cues.Source: 2) Structured State Models · p. 10

3) Imagination-based Policies:

  • Imagination-based manipulation methods optimize or evaluate policies through imagined interaction and use rollouts to close loops with policy failures and synthetic trajectories.Their central question is whether predicted futures improve executable control in contact-rich settings, under distribution shift, and over long horizons.Source: 3) Imagination-based Policies · p. 10

B. Navigation

  • Navigation WAMs mainly provide look-ahead by predicting future egocentric views, semantic maps, or latent states for waypoint or action selection.Navigation emphasizes long-horizon spatial reasoning under partial observability rather than contact-sensitive control.Source: B. Navigation · p. 10

1) Look-Ahead Planning:

Navigation WAMs use predicted views, maps, or latent states to support waypoint selection and long-horizon control. In driving, prediction supports scene generation, occupancy forecasting, and unified planning; evaluation must connect predictive quality with downstream control utility and deployment constraints.

1) Look-Ahead Planning:

  • Look-ahead navigation methods predict future panoramas, semantic maps, poses, or failure probabilities to support waypoint and action selection.These systems keep planning modular: a world model supplies candidate futures and a downstream planner selects actions.Source: 1) Look-Ahead Planning · p. 10

2) Latent Planning and WM-RL:

  • Latent navigation methods use compact plans or world-model states for policy learning and long imagined rollouts without requiring explicit image generation.Their practical value is assessed through waypoint selection, long-horizon consistency, and robustness under partial observability.Source: 2) Latent Planning and WM-RL · pp. 10–11

C. Autonomous Driving

  • Driving WAMs must account for real-time, safety, and multi-agent constraints, and use prediction for control as well as scenario synthesis and scene evolution.The survey groups these roles into controllable video generation, occupancy-based forecasting, and unified perception-prediction-planning.Source: C. Autonomous Driving · p. 11

1) Generation and Structured Scene Prediction:

  • Driving models either generate future traffic scenes for augmentation or evaluation, or forecast semantic occupancy aligned with free space, dynamic agents, and collision-related evaluation.Occupancy-based methods replace pixel futures with planning-oriented 3D states.Source: 1) Generation and Structured Scene… · p. 11

2) Unified Driving Agents and WM-RL:

  • Unified driving WAMs couple scene prediction with action generation, while imagined-rollout RL depends strongly on action-conditioned dynamics fidelity.Evaluation must connect visual realism, traffic consistency, safety, and closed-loop driving performance.Source: 2) Unified Driving Agents and WM-RL · p. 11

V. DATASETS, BENCHMARKS, AND EVALUATION METRICS

  • Datasets supply demonstrations, trajectories, or video-language pairs; benchmarks define protocols and metrics; simulators provide environments for closed-loop policy tests.The survey organizes evaluation resources by these three resource types.Source: §V · p. 11

A. Metric Families and Evaluation Protocols

  • Task-level metrics reflect control objectives, but informative WAM evaluation also reports predictive quality and downstream control utility.Trajectory error does not guarantee safety under feedback, and visual-fidelity metrics alone are insufficient for action grounding; latency, collision, comfort, and infraction metrics capture deployment constraints.Source: A. Metric Families and Evaluation… · p. 12

B. Robot Demonstration and Human Video Corpora

The surveyed resources separate action-grounded robot demonstrations from unpaired human video and span diverse manipulation, driving, navigation, and RL evaluations. The section also identifies action alignment as a key WAM challenge and outlines stage-wise adaptation as a common strategy.

B. Robot Demonstration and Human Video Corpora

  • Paired robot demonstrations provide action grounding, whereas unpaired human videos expand motion and language priors and are typically evaluated after action learning or robot-data fine-tuning.Robot resources primarily support pretraining and embodiment adaptation; human video does not directly specify robot actions.Source: B. Robot Demonstration and Human Video… · p. 12

C. Simulated Manipulation Task Suites

D. Autonomous Driving Datasets and Simulators

  • Driving datasets support open-loop prediction while protocols such as CARLA support closed-loop assessment, requiring measures of realism, structured prediction, planning utility, and safety.Video realism alone is insufficient to judge a driving WAM.Source: D. Autonomous Driving Datasets and… · p. 13

E. Navigation Environments, Datasets, and Benchmarks

  • Navigation evaluation spans instruction and goal protocols, and combines arrival, efficiency, route-fidelity, and pose-consistency measures relevant to look-ahead WAMs.Predicted egocentric views or latent states may aid waypoint selection without photorealistic future frames.Source: E. Navigation Environments, Datasets,… · p. 13

F. Foundational RL Benchmarks

  • Foundational RL benchmarks diagnose predictive-model capabilities but do not replace embodied robot evaluation; FMB bridges this gap with real-world manipulation success and cycle time.Their abstractions omit many sensing, contact, embodiment, and safety constraints.Source: F. Foundational RL Benchmarks · pp. 13–14

VI. OPEN CHALLENGES AND FUTURE DIRECTIONS

  • WAM development faces four major bottlenecks: action grounding, spatial consistency, closed-loop policy improvement, and real-time inference.These challenges arise because WAMs integrate perception, prediction, and action generation in a multi-objective design and training problem.Source: §VI · p. 14

A. Decoupling Motion Intent and Action Alignment

  • WAMs separate task-level motion reasoning from low-level control, creating a need to stage motion-intent learning and action alignment while preserving visual priors.The challenge is intensified by imbalance between visual and action data.Source: A. Decoupling Motion Intent and Action… · p. 14

1) Stage-wise Motion-to-Action Alignment:

  • A common alignment strategy pretrains video models on large-scale visual data, then adapts them with embodied data to align predicted motion with executable actions.IDM-style WAMs are often suited to this staged optimization.Source: 1) Stage-wise Motion-to-Action Alignment · p. 14

2) Prior Preservation during Action Alignment:

The section considers how to preserve pretrained visual knowledge, separate world prediction from action selection, and improve spatial consistency and memory for control. It presents decoupled objectives and structured 3D representations as design directions, while noting remaining integration challenges.

2) Prior Preservation during Action Alignment:

  • Parameter freezing, action-alignment layers, and residual action heads can confine control-specific updates, while stronger anti-forgetting mechanisms may help retain pretrained visual knowledge.Prior preservation matters during action alignment.Source: 2) Prior Preservation during Action… · p. 14

B. World–Action Factorization

  • World prediction models action-conditioned state transitions for predictive fidelity, while action selection maps states to actions to maximize cumulative rewards.The joint trajectory distribution can be factorized into world and agent components.Source: B. World–Action Factorization · p. 14

1) Decoupled Learning Objectives:

  • Separate objectives let the world model focus on accurate simulation and the agent model on action selection; joint optimization may produce optimistic states exploitable by the policy.Such exploitation may degrade real-world performance.Source: 1) Decoupled Learning Objectives · p. 14

2) Decoupled Learning and Inference:

  • A decoupled world model can serve as a simulator for offline RL, runtime planning, and model predictive control, although bias control and component coordination remain challenges.The passage describes this direction as promising.Source: 2) Decoupled Learning and Inference · p. 14

C. Spatial and Multi-View Consistency

  • Single-view 2D prediction can struggle with geometry, object permanence, and cross-view consistency, degrading state estimation, planning, and inverse-dynamics decoding.The stated applications include manipulation and autonomous driving.Source: C. Spatial and Multi-View Consistency · p. 14

1) Explicit 3D Spatial Representations:

  • Point clouds, occupancy fields, and 3D Gaussians offer structured, control-relevant alternatives to independent view generation, but must be integrated with scalable generative backbones and efficient action decoding.Source: 1) Explicit 3D Spatial Representations · p. 14

2) Egocentric Observations:

  • Egocentric video supplies manipulation-intent and hand-object cues, and fixed inter-camera extrinsics tend to provide stronger multi-view consistency.Source: 2) Egocentric Observations · p. 14

D. Long-Horizon Memory

  • Long-horizon tasks require retaining progress, past interactions, and occluded information; simply extending the observation window is costly and adds redundant visual context.These constraints motivate compact, selective memory mechanisms.Source: D. Long-Horizon Memory · p. 14

1) Model-Integrated Memory Architectures:

The section surveys memory strategies, neural simulation for policy improvement, and methods for reducing WAM inference cost. It concludes that reliable progress requires action-grounded, consistent predictions, closed-loop improvement, and real-time performance.

1) Model-Integrated Memory Architectures:

  • MEM combines video-based short-term memory with text-based long-term memory, while EventVLA stores task-critical visual events and MemoryVAM injects episodic memory into prediction and action decoding.These model-integrated approaches often require architecture-specific training.Source: 1) Model-Integrated Memory Architectures · p. 14

2) Memory-Augmented Agentic Harnesses:

  • Harness VLA adds an agentic planner to a frozen VLA to maintain execution traces, success rules, and failure models for planning, verification, and recovery without modifying the policy.Source: 2) Memory-Augmented Agentic Harnesses · p. 14

E. Neural Simulators and Closed-Loop Policy Learning

  • Imitation learning relies heavily on high-quality expert demonstrations and has limited closed-loop correction and improvement from failure, motivating closed-loop policy learning.Training skilled teleoperators is costly, and low-quality or failed demonstrations are typically discarded.Source: E. Neural Simulators and Closed-Loop… · p. 15

1) Neural Simulators:

  • RSSM-style neural simulators learn action-conditioned dynamics for policy evaluation and improvement, but imagined prediction errors can bias long-horizon optimization.Source: 1) Neural Simulators · p. 15

2) Closed-Loop Policy Learning:

  • Neural-simulator policy optimization can use task rewards, goal-based objectives, or learned reward models, while reward-free and offline learning remain challenges at scale.Long-horizon credit assignment is critical for improving policies from predicted futures; SSL-RL may replace the IDM-based expert in dual systems.Source: 2) Closed-Loop Policy Learning · p. 15

F. Inference Latency and Computational Efficiency

1) Few-Step Action Generation:

  • Consistency models, mean-flow, rectified-flow, and related methods reduce sampling cost, but must preserve action-conditioned transitions, temporal consistency, and control-relevant state information.Source: 1) Few-Step Action Generation · p. 15

2) Deployment Optimization:

  • Quantization, pruning, fusion, batching, distillation, and KV-cache reuse can improve deployment efficiency, alongside reducing rollout horizon, imagined trajectories, replanning frequency, and decoding cost without sacrificing control fidelity.Source: 2) Deployment Optimization · p. 15
Loading 2609.16074v1…