PaperlayerView exact revision on arXiv

Source-linked AI summary

World Models for Embodied Intelligence: From Plausible to Controllable to Actionable

Nanjie Yao, Hao Wang, Chong Cheng, Zhikang Chen, Wenzhe Li, Jiafei Lyu, Li Shen, Peilin Zhao, Zongqing Lu, Gao Huang, Steven Hoi, Dacheng Tao, Deheng Ye

arXiv:2609.16697v1cs.ROcs.AI

TL;DR

The paper addresses which predictive capabilities make world models useful for embodied behavior, beyond visual plausibility. It proposes a capability ladder and grounding–improvement matrix, then uses them to survey embodied world modeling and identify evaluation challenges. Its conclusion shifts evaluation toward task-relevant state, intervention effects, and measurable closed-loop gains.

  • Problem

    Existing surveys leave implicit which predictive capabilities improve embodied behavior by organizing world models mainly around architecture, representation, or application.

  • Method

    The paper introduces Plausible, Controllable, and Actionable capability levels and a 3 × 4 matrix linking grounding dimensions with improvement loops.

  • Results

    The framework organizes methods, datasets, benchmarks, and evaluation protocols across manipulation, navigation, locomotion, autonomous driving, and general embodied learning.

  • Takeaways & Limitations

    World-model evaluation should focus on whether predictions preserve task-relevant state, reflect intervention effects, and improve closed-loop behavior.

  • Takeaways & Limitations

    Long-horizon state consistency remains difficult because identity, geometry, and dynamics can drift during rollouts, while unresolved uncertainty grows over time.

Abstract

from arXiv · show

World models connect perception and decision-making in embodied intelligence by maintaining hidden state, anticipating consequences, comparing interventions, and adapting when execution departs from expectations. Although progress is often measured by visual fidelity, their value lies in improving behavior. Before reaching for a cup, a person anticipates its weight and resistance to grasping, shaping the hand before contact. Such anticipation is coarse and rarely pictorial, yet it guides action. This raises a central question: which predictive capabilities improve behavior? Existing surveys, organized by architecture, output modality, or application domain, leave this question implicit. We introduce three progressively stronger capability levels: Plausible models preserve task-relevant temporal, geometric, or physical structure; Controllable models additionally predict how interventions alter that structure; and Actionable models translate predictions into measurable gains in planning, action, learning, evaluation, verification, recovery, or data selection. We complement this hierarchy with a 3 x 4 matrix crossing geometry, physics, and action grounding with improvement loops centered on data, rewards, policies, and the model itself. Using this framework, we survey manipulation, navigation, locomotion, autonomous driving, and general embodied learning, tracing technical progressions, clarifying capability requirements, and examining datasets, benchmarks, and evaluation protocols. We identify challenges in long-horizon consistency, uncertainty calibration, causal intervention testing, latency, verification and recovery, and cross-embodiment transfer. This perspective shifts evaluation from visual plausibility toward whether predictions capture task-relevant state, reflect intervention effects, and improve the closed-loop behavior of embodied agents.

1 Introduction

The survey reframes world models around the predictive capabilities that improve embodied behavior, rather than visual realism alone. It introduces a Plausible-to-Controllable-to-Actionable ladder and a grounding–improvement matrix for organizing methods and evaluation.

  • 1 Introduction: World models should preserve task-relevant state, predict intervention effects, and produce measurable downstream improvements in embodied decision making.These requirements define progressively stronger Plausible, Controllable, and Actionable capabilities.
  • 1 Introduction: Existing surveys organize world modeling mainly by architecture, representation, application, or embodiment rather than by capability progression toward decision utility.The proposed framework complements these perspectives by making predictive state, intervention fidelity, and downstream utility explicit.
  • 1 Introduction: The decision-centered ladder spans Plausible, Controllable, and Actionable models across architectures, representations, and application domains.Plausible models preserve temporal, geometric, or physical structure; Controllable models predict intervention effects; Actionable models improve downstream decisions or updates.
  • 1 Introduction: The 3 × 4 matrix connects geometry, physics, and action grounding with data, reward, policy, and model-self improvement loops.It links represented structure to the mechanism through which predictive knowledge affects an embodied agent.
  • 1 Introduction: The review organizes methods across manipulation, navigation, locomotion, autonomous driving, and general embodied learning, while examining datasets, benchmarks, and evaluation protocols.It also identifies open problems involving uncertainty, latency, verification, recovery, and cross-embodiment transfer.

2 Foundations, Definitions, and Boundaries

World models are framed as predictive models of agent–environment dynamics whose value depends on supporting embodied reasoning and decision making, not merely reconstructing observations. The section distinguishes functional scope and capability from representation choices, while tracing the field’s progression from visual and latent prediction toward controllable and decision-oriented systems.

  • Foundations and progression: The field progresses from action-conditioned visual foresight and latent imagination to value-equivalent prediction, uncertainty-aware model exploitation, and joint future–action generation.Representative systems connect predicted futures to model-predictive control, policy optimization, planning, conservative learning, or action decoding.
  • Definitions and scope: World models predict task-relevant environmental change from interaction history and optional actions to support embodied reasoning or decision making.They may model pixels, features, latent states, objects, geometry, physical variables, symbolic states, rewards, or values.
  • Definitions and scope: World-model predictions can support planning, imagined policy training, simulation, direct action generation, or evaluation of predicted transitions and uncertainty.These uses are functional interfaces rather than mutually exclusive architectural classes.
  • Boundaries: Visual plausibility alone is insufficient: a world model must capture task-relevant dynamics, grounded intervention effects, or measurable decision improvements to support stronger capability claims.Purely reactive vision-language-action policies remain outside scope unless predicted futures influence action generation, policy learning, candidate selection, or execution.
  • Capability versus representation: A model’s representation space does not determine its capability: pixel, latent, structured, or 3D models must instead be judged by task-relevant prediction, action response, and downstream utility.Larger backbones and longer rollouts likewise do not imply stronger world modeling.

3 Unified Taxonomy

The unified taxonomy evaluates world models through a cumulative Plausible–Controllable–Actionable ladder and two complementary views of grounding and improvement pathways. It requires explicit tests of task-relevant state, intervention effects, and measured downstream gains rather than inferring capability from architecture, representation, or use alone.

  • Unified taxonomy: The taxonomy separates claim strength from implementation choices through cumulative Plausible, Controllable, and Actionable levels.The ladder is complemented by grounding dimensions and update loops that describe what is predicted and how it changes the system.
  • Capability ladder: Plausible models preserve task-relevant temporal, geometric, or physical structure, while Controllable models additionally predict grounded effects of interventions.Actionable models use those predictions to produce measurable improvements in decisions or learning.
  • Evaluation protocols: Plausibility tests should report task-relevant state error and constraint satisfaction across prediction horizons, scene shifts, or uncertainty so temporal drift remains visible.Geometry, physics, and action-grounded tests target properties such as identity, pose, contact, material response, and observed consequences.
  • Intervention fidelity: Controllability requires held-out transitions showing that paired interventions produce distinct, correct grounded effects; changing prompts or sample appearance is insufficient.Recorded-action conditioning alone cannot isolate intervention effects because unobserved conditions may influence both actions and outcomes.
  • Decision utility: Actionability requires a downstream decision or update to improve a measured outcome under a matched controller and compute budget.Qualitative imagined rollouts do not satisfy this criterion.
  • Grounding and improvement loops: Grounding evaluates geometry, physics, and action effects, while data, reward, policy, and model-self loops identify where predictions enter the learning or decision cycle.Matrix cells represent possible, concurrent interfaces rather than capability levels or mutually exclusive method categories, and occupying one does not establish actionability.

4 Plausible World Models

Plausible world models preserve task-relevant state rather than merely generating sharp frames, progressing from compact representations to metric geometry, physical feasibility, and persistent long-horizon consistency. Each stage makes increasingly meaningful properties testable, while introducing distinct limits in cost, sensing, and rollout duration.

  • 4.1 Foundations: Plausibility requires preserving task-relevant structure—including object identity, spatial layout, physical regularity, memory, and rollout stability—rather than maximizing image quality.Evaluation asks whether predicted state remains measurable through time, occlusion, and distribution shift.
  • 4.1 Foundations: The progression moves from compact latent or object-centric states to metric scene representations, constrained dynamics, and persistent state across long rollouts.The stages are ordered by added predicted-state requirements, not publication chronology.
  • 4.1 Compact Latent and Object-Centric State: Compact predictive states can support forward prediction without reconstructing every pixel, but slot indices alone do not reveal physical location or distinguish drifting and lost objects.Object-centric models structure embeddings as entities and relations, while embedding predictors avoid pixel reconstruction.
  • 4.2 Metric Scene State: Metric scene models forecast occupancy, depth, point clouds, or displacement in shared physical frames, extending measurable prediction across driving, manipulation, and indoor robotics.Reported errors can be expressed in meters, occupancy, or point displacement, with cross-view agreement as a training constraint.
  • 4.3 Interaction Fidelity: Physics-aware predictors constrain support, collision, contact, friction, and material response, allowing interaction feasibility to be scored separately from visual appearance.Benchmarks evaluate physical outcomes and collision events independently of frame quality.
  • 4.4 Persistence and Long-Horizon Consistency: Long-horizon plausibility requires preserving the same object identity and location after rollout, not merely retrieving similar-looking content or extending a video.State persistence remains distinct from longer video duration and can fail when sensors leave and revisit regions.

5 Controllable World Models

Controllable world models predict how interventions change task-relevant state, distinguishing command compliance from smooth but unresponsive generation. The literature progresses from fixed action conditioning through open and measurable controls to policy-coupled rollouts, while exposing trade-offs in labels, grounding, and execution fidelity.

  • 5.1–5.4 Capability Criterion: A Controllable model must predict intervention-induced state changes, with realized changes matching requested changes while unrelated evolution remains consistent across matched rollouts.Accepting an action input is insufficient if the model ignores the command while producing a realistic continuation.
  • 5.1–5.4 Progression: The progression moves from fixed motor or ego commands to open latent or language interventions, measurable geometric or physical variables, and policy-driven interactive rollouts.The stages widen the control set and increasingly connect prediction with action generation.
  • 5.1 Action Condition: Fixed action-conditioned models establish the conditioning interface and separate scene realism from command compliance, but remain tied to logged action labels from one embodiment.Examples span motor actions, end-effector trajectories, ego actions, and graded driving controls; ACT-Bench separately measures action fidelity.
  • 5.2 Open Control: Open-control models reduce dependence on labeled actions through latent actions, goals, keyboard-like interaction, and language-specified interventions.Latent actions can be recovered from unlabeled video, while language methods compose instructions from lower-level primitives.
  • 5.2 Open Control: Latent and linguistic interventions are difficult to compare directly with realized changes because their internal units lack external measurement.This creates a measurement gap even as the control interface becomes more flexible.
  • 5.3 Grounded Control: Structure-grounded control expresses interventions in measurable units such as camera pose, occupancy, object paths, joint configurations, and contact forces, at the cost of sensing, reconstruction, or annotation.These variables make requested changes readable in physical units and remove reliance on human ratings.
  • 5.4 Policy-Driven Rollouts: Policy-coupled models combine action-conditioned prediction with action generation and long-horizon rollout, but policy-selected actions leave rollout fidelity unresolved.Different systems expose visual observations, actions, and rewards or couple generation with generalist policy execution.

6 Actionable World Models

Actionable world models use prediction to change a decision or learning update and demonstrate a downstream gain against a matched alternative. Their roles range from candidate selection and action decoding to simulator-based training or evaluation and runtime correction, with utility constrained by latency and verification costs.

  • 6 Actionability Criterion: Actionability requires prediction to alter system behavior and improve a measured outcome over a matched alternative that does not use the prediction.Prediction accuracy or controllability alone does not establish downstream utility.
  • 6 Functional Stages: The four functional placements are external candidate selection, action decoding from imagination, learned simulation for policy training or evaluation, and runtime verification with correction or recovery.These stages are alternative placements in the decision process rather than a mandatory pipeline.
  • 6.1 Plan and Select: In plan-and-select systems, a world model predicts consequences for externally proposed actions, subgoals, poses, or symbolic states and supplies a score for comparison.Learned scores can replace hand-specified rewards, while safety-oriented models admit only high-probability safe inputs.
  • 6.1 Plan and Select: Candidate-selection utility is bounded by rollout cost: proposal count and model-call frequency constrain whether search fits within a control period.Every candidate consumes a model rollout.
  • 6.2 Imagine and Decode Actions: When prediction moves inside the policy, predictive pretraining or imagined futures can improve downstream visual control and decode executable actions.ContextWM uses in-the-wild video pretraining to improve sample efficiency, while other systems extract robot actions from generated futures.
  • 6.3 Simulate for Training and Evaluation: Learned simulators use action-conditioned rollouts for policy training or evaluation, including comparisons against real-robot behavior and evaluation from real initial frames.Simulation-based evaluation is judged by agreement with real-robot evaluation, while long-rollout reliability and inference speed remain practical concerns.
  • 6.4 Runtime Verification, Correction, and Recovery: Runtime correction treats the residual between imagined and observed execution as a control signal for continuing, replanning, truncating stale actions, or updating the model state.Its value depends on timely arrival, and false alarms on correct executions impose their own cost.

7 Four Improvement Loops

The four improvement loops describe how world-model predictions repeatedly alter data, evaluative feedback, policies, or the predictor itself. Actionable claims require measured gains, while sustained improvement requires evaluation across repeated updates.

  • 7 Four Improvement Loops: A coupled iteration collects or filters experience with C and updates any subset of the world, critic, and policy models with U.The subscript k distinguishes repeated updates from single-use capability demonstrations.
  • Data Loop: The data loop acquires, generates, or curates transitions, including targeted counterfactual failures filtered for task validity, visual coherence, and kinematic safety.Repeated error-driven collection still requires independent checks of synthetic transition coverage.
  • Reward Loop: The reward loop turns predicted transitions into preference, progress, value, feasibility, or risk signals, but accurate average prediction may still misrank controller candidates.Examples include rollout judges, progress-value models, and critics with predictive heads or future-latent objectives.
  • Policy Loop: The policy loop uses imagined futures to propose or rehearse behavior, while support gates, keyframe initialization, and co-evolution limit—but do not establish—realized improvement.World-model exploitation remains a failure mode, and strict versus loose gates trade off discarded useful experience against unsupported states.
  • Model-Self Loop: The model-self loop updates predictive parameters, state, memory, or uncertainty only from measured deployment errors or verification results targeting a diagnosed weakness.A better architecture alone is not a loop.
  • 7 Four Improvement Loops: The four loops close different paths around a grounded predictor: data targets informative transitions, reward produces evaluative feedback, policy uses imagined experience, and model-self updates the predictor.The loops are concurrent rather than sequential, and one system may run several at once.

8 Embodiments and Domain Transfer

The capability ladder applies across embodiments, but each domain requires different task-relevant variables and operating conditions. Transfer therefore depends on matching representations, dynamics, action semantics, decision logic, and evaluation to the embodiment.

  • 8 Embodiments and Domain Transfer: Manipulation depends on contact and object state, driving on multi-agent futures and safety, navigation on persistent belief and geometry, and locomotion on fast body dynamics and stability.The same reported number can therefore carry different weight across domains.
  • Manipulation: Manipulation models increasingly predict contact, friction, force, and wrench variables directly because pixels can show reach while hiding grasp pose and force trajectories.Tactile signals expose contact forces and friction changes that vision alone does not reveal.
  • Autonomous Driving: Driving models must preserve geometry, traffic rules, multi-agent intent, ego-control effects, and feasible risk-bearing futures rather than reproduce one logged trajectory.Closed-loop evaluation should report route completion, collisions, comfort, rule violations, intervention rate, and latency under one planner budget.
  • Navigation: Navigation requires a belief that persists after places and objects leave view, supporting loop closure, disambiguation, future visibility, traversability, and goal progress.Benchmarks should separate map accuracy from navigation utility, especially during exploration.
  • Locomotion: Locomotion favors compact predictive states that meet the control clock, with actionability assessed through return, falls, constraint violations, and adaptation.A slow visual generator may support high-level planning but cannot replace a high-rate dynamics model inside control; hierarchical rates require aligned states and uncertainties.
  • What Transfers Across Embodiments: Cross-embodiment transfer should be assessed separately for representation, dynamics, action semantics, and decision logic, with consistent reporting of adaptation cost.Object permanence and free-space geometry may transfer widely, whereas contact dynamics require accounting for material, tool, and actuator differences.

9 Data, Benchmarks, and Evaluation

Data and evaluation protocols should match the claimed capability and grounding, preserving executable actions, timing, failures, and relevant multimodal state. Valid comparisons require matched interventions, downstream budgets, controls, calibrated uncertainty, and separate reporting of visual quality and functional utility.

  • 9 Data, Benchmarks, and Evaluation: Plausibility, controllability, and actionability require progressively stronger tests: consistent task-relevant state, expected intervention effects, and improved outcomes under the task objective and system budget.Higher-level claims should retain relevant lower-level tests because no single metric proves all three.
  • Data Requirements: A useful world-model record links state, observations, actions, time, and resulting state, optionally adding goals, calibration, depth, proprioception, force, contact, audio, reward, failures, and recovery.Timing and action semantics are always part of the data.
  • Datasets: Cross-embodiment datasets support action grounding only when control rate, coordinate frames, calibration, and original action definitions are preserved.A normalized action vector alone does not determine transferable meaning.
  • Datasets: Closed-loop datasets should record failure onset, likely cause, recoverable state, corrective action, and final outcome, with paired nominal and perturbed runs isolating prediction-invalidating events.Synthetic failures can extend coverage but require physical and task-validity checks before policy training.
  • Plausibility and Stability: Grounded evaluation adds object, 3D, depth, occupancy, topology, contact, momentum, constraint, and long-horizon drift measures beyond image and video appearance metrics.Long-horizon tests should report error as a function of rollout time and memory retention.
  • Controllability: Controllability should use shared-start paired or branched interventions and measure action adherence, treatment-effect error, counterfactual ranking, contact consistency, and invariance.The intervention changes action, timing, route, force, or trajectory constraint while unchanged variables should remain stable.
  • Actionability: Actionability should be measured inside the decision path using realized utility, safety, latency, decision budgets, and matched baselines that separate predictive structure from extra computation or policy strength.Relevant measures include success, regret, replanning, return, constraint cost, interaction count, calibration, and verification risk.
  • Uncertainty: Uncertainty evaluation should report horizon calibration, action-dependent confidence, selective prediction, and fixed-coverage risk while distinguishing sample diversity from weak dynamics.Diverse samples help only when they cover feasible outcomes and preserve relative risks.

10 Open Problems

The survey identifies open problems at each capability boundary, emphasizing persistent state, causal intervention testing, decision-aware efficiency, joint grounding, calibrated uncertainty, shared evaluation, timing, and reproducibility.

  • 10.1 Persistent State and Drift: Long-horizon models must preserve identity, geometry, and dynamics while supporting correction and uncertainty growth when hidden state remains unresolved.The proposed hide-and-reveal protocol tests whether models retain object pose and physical properties and show when errors begin and can be repaired.
  • 10.2 Intervention Fidelity: Intervention claims require branched data, action perturbations, failed actions, and fixed-state tests that separate command effects from correlations in demonstrations.Cross-embodiment evaluation must preserve both semantic intent and embodiment-specific realization.
  • 10.3 Decision-Aware Efficiency: Action-responsive models remain unusable when latency, uncertainty, or small prediction errors change action rankings, motivating regret evaluation under fixed compute budgets.The benchmark should vary proposal quality, horizon, and model-call budget while comparing selected actions with grounded outcomes.
  • 10.4 Joint Grounding: Most systems specialize in geometry, physics, or action grounding, so evaluation must test conflicting axes and whether modules exchange causal state rather than concatenate features.Examples include geometrically reachable but physically unstable grasps and physically possible motions that violate instructions.
  • 10.5 Uncertainty and Stable Loops: Uncertainty must be action- and horizon-dependent, with defined low-confidence responses, because policy updates can invalidate calibration measured only on held-out training-distribution data.Possible responses include shorter rollouts, conservative values, data requests, fallback policies, or safe stops; loop benchmarks should measure regressions and interaction cost across cycles.
  • 10.6 Evaluation, Access, and Reproducibility: Shared evaluation should report state stability, intervention consistency, regret, calibration, closed-loop utility, timing failures, and domain-specific outcomes across embodied settings.Hierarchical systems also need explicit contracts for timestamps, committed action prefixes, uncertainty, cancellation, sensing delay, and variable inference time.
  • 10.6 Evaluation, Access, and Reproducibility: Reproducibility is constrained by dependence on large video backbones, private robot data, and expensive trials, requiring disclosure beyond released checkpoints.Recommended disclosures include interfaces, data filters, model-call budgets, latency, seeds, and failure cases, with smaller diagnostics and frozen traces used before full training.

11 Conclusion

The conclusion frames world models by the behavior their predictions enable, using a capability ladder and grounding–improvement matrix to organize embodied-intelligence research. It calls for evaluation of state consistency, intervention fidelity, and downstream utility while noting heterogeneous settings and persistent open challenges.

  • 11 Conclusion: The survey organizes world models as Plausible, Controllable, or Actionable according to preserved structure, intervention prediction, and measurable downstream gains.These levels are complemented by a 3 × 4 matrix linking geometry, physics, and action grounding with data, reward, policy, and model-self improvement loops.
  • 11 Conclusion: The framework spans manipulation, navigation, locomotion, autonomous driving, and general embodied learning while separating state consistency, intervention fidelity, and downstream utility.It supports qualitative comparison, but heterogeneous evaluation settings limit direct comparison of reported results.
  • 11 Conclusion: Open challenges include long-horizon consistency, uncertainty calibration, causal intervention testing, latency, verification, recovery, and cross-embodiment transfer.
Loading 2609.16697v1…