World Models for Embodied Intelligence: From Plausible to Controllable to Actionable

• Direct observation-to-action policies lack explicit mechanisms for maintaining task-relevant state, anticipating delayed consequences, and comparing interventions in partially observed, contact-rich, and long-horizon environments.These capabilities are needed to revise behavior when execution departs from expectation.Source: §1 · p. 3
• The Plausible–Controllable–Actionable ladder classifies progressively stronger capabilities: preserving relevant structure, predicting intervention effects, and producing measurable downstream decision or learning benefits.The levels are capability requirements rather than mutually exclusive architecture classes.Source: §1 · pp. 3, 5
• It organizes related methods from latent dynamics and structured prediction through physics-aware, action-conditioned, planning, policy-learning, verification, and correction systems across multiple embodied domains.Coverage includes manipulation, navigation, locomotion, autonomous driving, and general embodied learning.Source: §1 · pp. 3–5
• The survey connects geometry, physics, and action grounding with data, reward, policy, and model-self improvement loops in a unified 3 × 4 matrix.This matrix links represented structure and feasible dynamics to how predictive knowledge enters learning and interaction.Source: §1 · p. 5
• The review covers datasets, benchmarks, and capability-aligned evaluation protocols while identifying challenges in state consistency, intervention fidelity, uncertainty, latency, verification, recovery, and cross-embodiment transfer.These challenges concern whether predictions support reliable decision utility rather than merely visual plausibility.Source: §1 · p. 5

2.1 From Learned Dynamics to World Models

• World models evolved from learned state transitions and visual foresight toward latent, value-aware, and intervention-sensitive predictions that improve downstream decisions rather than merely reconstruct observations.This progression includes model-predictive control, imagined trajectories, tree search, uncertainty management, and continuous grounding with real experience.Source: §2.1 · pp. 6–7

2.2 World Models: Definition and Scope

• A world model predicts task-relevant temporal change from interaction history and optional actions, using pixels, tokens, latent states, objects, geometry, physical variables, symbolic states, rewards, or values.Predictions may support planning, imagined policy learning, simulation, or action generation, while observation decoding remains optional.Source: §2.2 · p. 7
• Visual plausibility alone is insufficient: world models must represent intervention effects and produce measurable decision or learning gains, excluding purely reactive policies without predictive mechanisms.The framework therefore distinguishes Plausible, Controllable, and Actionable capabilities by task-relevant consistency, correct intervention responses, and downstream improvement.Source: §2.2 · pp. 7–8, §2.3 · p. 8

2.3 Predictive Representations

• World-model representations vary by prediction space, semantic content, temporal horizon, uncertainty modeling, and action interface; these choices determine how predictions are queried, not their capability.Observation-, latent-, and structured-state models can predict appearance, geometry, physics, or task variables through deterministic, probabilistic, explicit-action, latent-action, or action-generating interfaces.Source: §2.3 · p. 8
• The taxonomy evaluates representations separately from capability: plausibility requires stable task-relevant prediction, controllability requires correct action-conditioned changes, and actionability requires measurable downstream gains.The capability ladder is cumulative, with progressively stronger requirements for state consistency, intervention effects, and matched-budget decision improvement.Source: §2.3 · p. 8, Figure 2 · p. 9

3 Unified Taxonomy

• The unified taxonomy distinguishes capability levels from design choices: the ladder measures state consistency, intervention fidelity, and decision utility, while grounding and improvement loops describe predicted variables and update paths.Geometry, physics, and action grounding are multi-label descriptors, as are data, reward, policy, and model-self loops.Source: §3 · p. 10

3.1 A Grounding-Conditioned Capability Ladder

• Plausible models preserve task-relevant geometry, physics, or action-state structure over time, but open-loop consistency does not establish counterfactual control.Evaluation may test identity, pose, occupancy, occlusion, multi-view consistency, motion, collision, contact, support, material response, and observed consequences across time.Source: §3.1 · pp. 10–11
• Controllable models predict how commanded interventions alter grounded futures, requiring held-out transitions whose predicted contrasts match true action effects rather than merely producing different-looking samples.Recorded-action conditioning alone cannot isolate intervention effects because unobserved conditions may affect both action selection and outcomes.Source: §3.1 · p. 11
• Actionable models change a downstream planner, policy, evaluator, verifier, or data selector and improve measured utility under matched controller and compute conditions.Relevant outcomes include task return, success, ranking quality, safety, recovery, real-data efficiency, and persistent improvement; qualitative imagined rollouts are insufficient.Source: §3.1 · p. 11

3.2 The Grounding–Improvement Matrix

• The grounding–improvement matrix crosses geometry, physics, and action effects with data, reward, policy, and model-self loops, identifying interfaces rather than capability levels or exclusive method categories.Matrix paths can be coupled across repeated updates, but occupying a cell does not establish actionability without measured improvement over an appropriate matched baseline.Source: §3.2 · p. 12, Table 2 · p. 12

4.1 Compact Latent and Object-Centric State

• Compact latent and object-centric models predict entities, relations, or task-relevant features without reconstructing every pixel, but their representations leave scene location unresolved.C-SWMs [110] use object representations and graph-based relations, while V-JEPA [13] and related methods predict in embedding space; a slot index is not a physical position.Source: §4.1 · pp. 13–14

4.2 Metric Scene State: Occupancy, Points, and Rendering

• Metric scene-state models express future occupancy, depth, displacement, or points in a shared frame, making error measurable in meters while enabling cross-view geometric agreement.This stage spans occupancy, voxel, point-cloud, Gaussian, and bird’s-eye-view representations across driving, manipulation, and indoor robotics, but state cost grows with spatial extent and rollout length.Source: §4.2 · pp. 14–15

4.3 Physics-Grounded Dynamics and Interaction Fidelity

• Physics-grounded dynamics constrain state evolution through support, collision, contact, friction, and material response, allowing feasibility to be scored separately from appearance.Benchmarks such as IntPhys [174], PHYRE [10], CLEVRER, and Physion [14] separate event plausibility from frame quality, while later systems incorporate physical constraints into prediction or generation.Source: §4.3 · p. 15

4.4 Persistent Memory and Long-Horizon Stability

• Persistent world models preserve object identity, layout, and maps across long rollouts through retrieved memory or architectures that carry state further at bounded cost.A longer video alone shows duration; state consistency requires recovered content to remain the same object in the same place.Source: §4.4 · pp. 15–16

4 Plausible World Models

• Plausibility requires preserving task-relevant structure beyond sharp frames, progressing from compact states to metric geometry, physical feasibility, and long-horizon persistence.The progression is conceptual rather than chronological and adds increasingly testable properties to the predicted state.Source: §4 · pp. 12–13, Figure 3 · p. 13, §4.4 · p. 16

5 Controllable World Models

• Controllability requires intervention fidelity: requested and realized changes must align in shared units while unrelated state evolution remains consistent across matched rollouts.Accepting a command without obeying it is insufficient, regardless of whether the intervention is motor, trajectory, latent, language, camera, object, or force based.Source: §5 · p. 16

5.1 Action-Conditioned Prediction

• Fixed action-conditioned models established command compliance but remained tied to logged action labels from a single embodiment, separating scene realism from controllability.Representative systems condition prediction on motor, end-effector, or ego actions; ACT-Bench [4] measures whether generated futures follow instructed trajectories.Source: §5.1 · p. 17, Figure 4 · p. 17

5.2 Open, Latent, and Language-Specified Interventions

• Latent, open, and language-specified interventions reduce dependence on labeled actions, supporting heterogeneous action spaces, interactive environments, keyboard control, free-text commands, and compositional manipulation goals.Because latent and linguistic controls are internal to the model, their effects cannot be compared directly with realized changes.Source: §5.2 · pp. 17–18

5.3 Structure-Grounded Control: Geometry, Trajectory, and Contact

• Measurable geometric and physical controls make intervention fidelity directly testable through camera poses, trajectories, object paths, occupancy, joint states, contact forces, mass, and friction.This stage supports sensor- or annotation-grounded comparison without human ratings, but requires additional annotation, reconstruction, or sensing.Source: §5.3 · pp. 18–19

5.4 Interactive Rollout and Joint World–Action Models

• Policy-driven rollouts place intervention selection inside the model, requiring faithful multi-step prediction as policies react to predicted observations.Ctrl-World [67], iVideoGPT [213], Vid2World [84], DreamDojo [57], Mem-World [264], and EnerVerse [85] exemplify interactive rollout interfaces with memory, causal guidance, or multi-view futures.Source: §5.4 · p. 19
• Joint world–action models couple future prediction with action generation, enabling shared or separate video-action representations for policy, forward-model, inverse-model, and end-to-end control roles.Examples include GR-1 [211], GR-2 [23], PAD [65], Unified World Models [271], WorldVLA [20], DyWA [145], and EVA [200].Source: §5.4 · p. 19

6 Actionable World Models

• Actionability requires predictions to change behavior and improve outcomes relative to a matched alternative, with correctness determined by the prediction’s consumer.Planners, action decoders, learners, evaluators, and runtime verifiers depend on different properties of predicted futures.Source: §6 · p. 20

6.1 Planning, Search, and Value-Based Selection

• Planning-stage models score externally proposed actions, subgoals, poses, or symbolic states, learning task-specific values while search cost is bounded by candidates and model calls.Representative systems include Video Language Planning [44], Safe MPC [112], VIPER [46], and World-Value-Action [121].Source: §6.1 · p. 21

6.2 Policy Learning and Action Decoding from Imagination

• Policy-stage models improve action generation by pretraining on predictive structure, decoding actions from imagined futures, or constraining latent representations without rendering futures at runtime.These approaches remain limited by predictive reliability and inference latency; ContextWM [212] specifically improves downstream model-based reinforcement-learning sample efficiency.Source: §6.2 · p. 22

6.3 Learned Simulators for Policy Training and Evaluation

• Learned simulators host policy training and evaluation, but a model that trains a policy cannot independently certify it because both optimize against the same errors.Benchmarks therefore assess perceptual quality, functional utility, executable interactions, and action-conditioned reliability.Source: §6.3 · p. 23

6.4 Runtime Verification, Correction, and Recovery

• Runtime systems use imagined–observed residuals for diagnosis, verification, correction, and recovery, but their control value depends on timely signals and costly false alarms.Correction can truncate stale action chunks, update predictions, or replan, while recovery returns execution toward earlier or familiar states.Source: §6.4 · pp. 24–25

7 Four Improvement Loops

• Four concurrent loops close different paths around a grounded predictor: data targets transitions, reward scores outcomes, policy uses imagined experience, and model-self repairs predictions.Their effects can reinforce errors as well as correct them, so sustained gains must be established separately for each system.Source: §7 · p. 25, §7.4 · p. 28

7.1 Data Loop

• The data loop acquires, generates, or curates transitions, using failure-directed collection, targeted synthesis, and diagnostics to make experience more informative.DreamGen [93], GigaWorld-0 [190], Interactive World Simulator [205], V-JEPA 2 [6], Dream2Fix [116], and WorldSimProbe [39] broaden or target data generation, but repeated error-driven collection remains unevaluated.Source: §7.1 · pp. 25–26

7.2 Reward and Critic Loop

• The reward loop compresses predicted transitions into preference, return, progress, feasibility, or risk, supplying dense feedback for evaluation and online policy refinement.GE-Sim 2.0 [171], RISE [223], Self-Correcting VLA [135], and WCM [48] show simulator-level and policy-coupled critics, but average critic accuracy may misrank controller candidates.Source: §7.2 · pp. 26–27

7.3 Policy Loop

• The policy loop uses world models to propose, filter, rehearse, or generate behavior at scale, while support gates regulate imagined experience to limit model exploitation.Cosmos Policy [109], World4RL [103], WoVR [104], GigaBrain-0.5 M* [189], WorldSample [221], and World-in-World span offline, simulated, and real-robot updates; support checks do not by themselves establish realized gains.Source: §7.3 · p. 27

7.4 Model-Self Loop

• The model-self loop updates predictive parameters, state, memory, or uncertainty from measured errors or verification, targeting diagnosed weaknesses rather than merely improving architecture.RLVR-World [214] optimizes decoded transition predictions, World Action Verifier [139] updates from cycle-consistency failures, and World-VLA-Loop [137] couples model and policy repair, complicating attribution.Source: §7.4 · pp. 27–28

8 Embodiments and Domain Transfer

• Across embodiments, the ladder holds, but manipulation, driving, navigation, and locomotion require different variables and operating conditions for claims to remain valid.The section rereads prior work by domain rather than capability level.Source: §8 · p. 28, §8.5 · p. 31

8.1 Robotic Manipulation

• Manipulation models must predict contact-relevant state—friction, forces, wrenches, and recovery states—rather than treating contact as a consequence visible in pixels.ParticleFormer [86], PIN-WM [123], OmniVTA [263], and FAWAM [76] make contact or force structure explicit, while CycleVLA [148] and See, Plan, Rewind [40] support recovery.Source: §8.1 · p. 28

8.2 Autonomous Driving

• Driving world models should preserve geometry, rules, multi-agent intent, and control effects while assigning probability to feasible multimodal futures that retain planner-relevant risks.Closed-loop evaluation should report route completion, collisions, comfort, rule violations, intervention rate, latency, and rare events rather than relying on single-trajectory displacement error.Source: §8.2 · pp. 28–29

8.3 Navigation and Exploration

• Navigation actionability depends on persistent beliefs supporting visibility, traversability, and goal progress, so benchmarks should separate map accuracy from the utility of reaching goals.Pathdreamer [111], VISTA [90], VISTAv2 [89], Navigation World Models [11], SparseVideoNav [248], EgoWM [9], and X-MOBILITY [136] trace increasingly action-conditioned prediction.Source: §8.3 · p. 29

8.4 Locomotion and Control

• Locomotion requires compact, fast predictive states for short-horizon dynamics, contact, stability, and control-loop operation; slow visual generators can only complement higher-rate dynamics models.PlaNet [70], Dreamer [69], DayDreamer [215], DreamerV3 [72], and TD-MPC2 [73] illustrate latent-state planning or learning across tasks and embodiments.Source: §8.4 · pp. 29–30

8.5 What Transfers Across Embodiments

• Cross-embodiment transfer may use object-level motion or latent-action interfaces, but a transferable interface is distinct from measured transfer, which requires matched tasks and explicit adaptation-cost reporting.Protocols vary the embodiment, environment dynamics, or both, while reporting frozen and adapted modules and the real transitions consumed.Source: §8.5 · pp. 30–31

9.1 Data Requirements

• World-model data should preserve timing and action semantics alongside observations, states, goals, calibration, depth, proprioception, contact, reward, failures, and recovery outcomes.Cross-embodiment use additionally requires control rate, coordinate frames, calibration, and the original action definition; normalized action vectors alone do not define shared interventions.Source: §9.1 · p. 31
• Closed-loop datasets should include failure onset, likely cause, recoverable state, corrective action, and final outcome, with paired nominal and perturbed runs to isolate invalidated predictions.Synthetic failures can extend coverage only after physical and task-validity checks.Source: §9.1 · p. 31

9.2 A Capability-Aligned Evaluation Protocol

• Capability-aligned evaluation separates appearance from grounded behavior, tests long-horizon stability, branches shared-start interventions, and measures action adherence, treatment effects, decision utility, verification, and recovery.Representative datasets and metrics are illustrative rather than mandatory, while higher-level claims should retain relevant lower-level tests.Source: §9.2 · pp. 31–32, Table 4 · p. 32

9.3 Benchmark Design Principles

• Executable benchmarks should expose repeatable perturbations and branching actions while reporting visual quality separately from functional utility, including model calls and wall-clock time.A predicted future is not actionable if it arrives after the control deadline.Source: §9.3 · p. 32
• Uncertainty evaluation should report horizon and action-dependent calibration, selective prediction, and fixed-coverage risk while distinguishing feasible outcome diversity from weak dynamics.Comparisons should also equalize observation histories, action proposals, downstream decision budgets, and resource usage such as cameras, privileged state, and model calls.Source: §9.3 · pp. 32–33
• Each higher capability requires stronger grounded tests: plausibility uses state errors and drift, controllability uses intervention branches and controls, and actionability uses matched utility, latency, safety, and failure-cost measures.No single visual metric spans the capability ladder, so evaluation must connect predictions to grounded outcomes and decision paths.Source: §9 · p. 31, §9.3 · p. 33

10.1 Persistent State and Drift

• Long-horizon models need correctable persistent state, drift-aware uncertainty, and hide-and-reveal tests that assess identity, pose, physical properties, and error repair.More context and longer rollouts do not establish state persistence; evaluation should show when errors begin and whether they can be repaired.Source: §10.1 · p. 33

10.2 Causal Action Effects

• Causal action modeling requires branched and failed-action data, fixed-state perturbation tests, and embodiment-transfer evaluations preserving both intent and realization.Action-conditioned interfaces and embodiment-independent motion commands provide candidate representations, but transfer must test whether semantic intent survives different robot kinematics and timing.Source: §10.2 · pp. 33–34

10.3 Decision Utility Under Budget

• Actionable models must preserve reward, feasibility, and safety information while compressing predictions, with regret under fixed compute budgets testing the right planning interface.The benchmark should vary proposal quality, horizon, and model-call budget across decoded video, latent state, object state, and 3D-scene interfaces.Source: §10.3 · p. 34

10.4 Joint Grounding

• Joint grounding needs interfaces where geometry, physics, and action representations agree on state, uncertainty, and time, especially when their predictions conflict.Conflict cases include reachable but unstable grasps, physically possible instruction violations, and actions invalidated by coordinate-frame or embodiment changes.Source: §10.4 · p. 34

10.5 Uncertainty and Stable Loops

• Uncertainty must be action- and horizon-dependent, trigger explicit low-confidence behavior, and remain calibrated across policy updates and repeated data, model, and policy cycles.Loop-level benchmarks should measure improvement, regression, diversity, and interaction cost while holding out dynamics and failures unavailable to the generator.Source: §10.5 · p. 34

10.6 Evaluation, Access, and Reproducibility

• Shared evaluation should combine state stability, intervention consistency, regret, calibration, and closed-loop utility with domain-specific metrics, timing contracts, and rare-failure reporting.Hierarchical systems need explicit timestamps, action-prefix, uncertainty, and cancellation contracts, while benchmarks should test sensing delays, variable inference time, deadline misses, and stale actions.Source: §10.6 · p. 35
• Reproducibility requires publishing interfaces, data filters, compute budgets, latency, seeds, and failures, while smaller diagnostic models and frozen traces reduce access barriers.Many systems depend on large video backbones such as Cosmos [2], private robot data, or expensive real trials; independent audits are also needed to prevent shared errors circulating through loops.Source: §10.6 · p. 35

Conclusion

• The capability ladder distinguishes Plausible models that preserve task-relevant structure, Controllable models that predict intervention effects, and Actionable models that produce measurable downstream gains.The framework organizes methods across manipulation, navigation, locomotion, autonomous driving, and general embodied learning.Source: §11 · p. 35
• The accompanying 3 × 4 matrix links geometry, physics, and action grounding to data, reward, policy, and model-self improvement loops.Source: §11 · p. 35
• Evaluation should separately assess state consistency, intervention fidelity, and downstream utility, while heterogeneous settings limit direct comparison of reported results.The framework supports qualitative comparison of capability claims rather than straightforward quantitative ranking across studies.Source: §11 · p. 35
• Progress toward actionability requires grounded predictions to improve closed-loop behavior under explicit data, compute, and latency budgets.Open challenges include long-horizon consistency, uncertainty calibration, causal intervention testing, latency, verification and recovery, and cross-embodiment transfer.Source: §11 · p. 35