Reading, Writing, Sharing, and Interacting with the World through Video

1 Introduction

• Video understanding, generation, and unified models are converging on how observations relate to the evolving world that gives rise to them.World models infer underlying states from sequences before predicting their evolution, while generators increasingly use action- and state-dependent controls.Source: §1 · p. 6
• The survey treats video as a partial, lossy, temporally ordered observation of a latent world and organizes research into reading, writing, sharing, and interacting.The operations respectively infer world properties, synthesize observations, couple understanding and generation, and predict or update state under actions and observations.Source: §1 · pp. 6–7
• The framework distinguishes visual plausibility and temporal coherence from consistency in the underlying world state.It provides a basis for comparing the objectives, capabilities, and evaluation criteria of the four paradigms.Source: §1 · p. 7
• The survey reviews tasks, datasets, evaluation, representations, architectures, reasoning, control, and applications, while examining state persistence, state evolution, and decision-making.It also contrasts conventional recognition and perceptual-fidelity metrics with criteria such as controllability, physical and causal consistency, and downstream utility.Source: §1 · p. 7
• The survey synthesizes design patterns, failure modes, and open directions, including long-horizon consistency, physical and causal grounding, measurable understanding–generation synergy, and standardized evaluation.It also provides a categorized public reading list, taxonomy, code, data, and other resources.Source: §1 · p. 7

2.1 World, Video, and Represented State

• A latent temporal world is a stochastic process of states containing entities, attributes, relations, and relevant physical and dynamic variables.The world evolves under its dynamics, optionally conditioned on an intervention or control input; video is a partial, lossy, temporally ordered trace, and a model’s represented state may be incomplete or inaccurate.Source: §2.1 · p. 10

2.2 Two Forms of Consistency

• Temporal coherence concerns local agreement between neighboring frames, whereas world consistency requires compatibility with an admissible trajectory under specified dynamics and constraints.Neither property implies the other: smooth frames can imply impossible state changes, while a hard cut can remain world-consistent.Source: §2.2 · pp. 10–11
• World consistency is an evaluation target rather than a metric, so benchmarks probe it through revisits, occlusion, and interventions.Because the observation process and admissible trajectories are unobserved, evaluation must test more than temporal coherence.Source: §2.2 · p. 11

2.3 The Four Operations

• Reading infers world properties from video, writing generates video from partial conditions, sharing couples comprehension and generation, and interacting predicts state transitions under actions or conditions.Interacting may roll out passively, under a fixed action sequence, or online; its evaluation concerns input-following, world-consistent changes beyond rendered-frame quality.Source: §2.3 · p. 11

3.1 Video Scenarios

• Video scenarios vary in temporal extent, viewpoint, modality, access to observations, interaction, and domain constraints, imposing complementary demands on what models observe, retain, predict, and control.Regular short clips do not test memory, online response, or control; streaming, long-form, interactive, and domain-specific settings impose distinct requirements.Source: §3.1 · pp. 11–12

3 Preliminaries

• The evidence ladder sets four increasing support levels for claims about the world, with stronger world-modeling claims requiring higher levels.Its benchmarks are representative rather than exhaustive.Source: Table 2 · p. 12

3.2 Video Large Language Models

• Video-LLMs connect video encoders to pretrained language models through connectors, while token count trades retained visual detail against context, memory, and attention costs.Compression can remove brief events, small objects, or long-range relations; the context window must also hold instructions, dialogue, and output.Source: §3.2 · pp. 12–13

3.3 Generative Modeling Foundations

• Generative foundations condition video outputs on content or evolution controls and include autoregressive, masked, diffusion, and flow-matching approaches.Autoregressive models predict ordered units sequentially; masked models predict hidden units from visible context, while diffusion and flow matching learn denoising or transport toward data.Source: §3.3 · pp. 13–14

3.4 Taxonomy of Video World Models

• Video world models learn how video-centered states evolve, excluding models whose primary persistent state is explicit 3D structure or purely symbolic or text-based.The taxonomy distinguishes observation-space from latent-state prediction and passive, action-conditioned, and interactive transition interfaces.Source: §3.4 · pp. 14–15

4.1 Tasks, Data, and Evaluation

• Video-understanding tasks test semantic, spatial, temporal, physical, or causal world properties; text-only, single-frame, shuffled-order, and minimal-pair tests expose answers that bypass video evidence.Semantic tasks range from action recognition to retrieval, captioning, and question answering; newer benchmarks also test hallucination and omission.Source: §4.1 · p. 15, §4.1.1 · p. 15

4.1.2 Spatial Understanding

• Spatial benchmarks test localization, identity persistence, layout, and unseen states, while temporal benchmarks probe event order, duration, long-range evidence, and streaming state maintenance.Examples include tracking through occlusion, inferring out-of-view layouts, long-video questions, and streaming evaluation of latency, false alarms, and missed responses.Source: §4.1.2 · p. 16, §4.1.3 · p. 17

4.1.4 Physical Understanding

• Physical benchmarks test plausible transitions, contact, next states, and mechanisms, whereas causal benchmarks ask why events occurred and how outcomes change under interventions or counterfactuals.MVP and VisPhyWorld [268] test reliance on physical evidence; CausalVQA [108] and DMC-CF [618] directly probe altered outcomes.Source: §4.1.4 · p. 17, §4.1.5 · p. 17

4.2 Video Representation Learning and Encoding

• Video representations use contrastive alignment, masked recovery, latent prediction, and generative objectives, each shaping what visual, semantic, or motion information the model retains.Contrastive alignment can diverge from recoverable physical state; teacher-based masked targets inherit teacher biases, while reconstruction may spend capacity on detail unnecessary for recognition.Source: §4.2 · p. 17, §4.2.1 · p. 18, §4.2.2 · p. 18

4.3.1 Architectures

• Video-LLM architectures balance temporal and spatial encoding, visual token detail, compression, and memory against context limits, with streaming and audio-visual systems adding bounded state and time alignment.Projectors, resamplers, cross-attention, and structure-preserving compressors offer different ways to map visual features into the language model; token reduction risks losing small objects, short events, or precise timestamps.Source: §4.3 · p. 19, §4.3.1 · pp. 19–20

4.3.2 Learning Paradigms

• Video-LLM training progresses from connector alignment to instruction tuning and post-training, while data mixtures, curricula, and reward design shape grounding, task coverage, and reasoning.Preference feedback improves faithfulness only when it distinguishes grounded from unsupported answers; task-specific rewards can target timestamps, spatiotemporal tasks, or separate perception and reasoning.Source: §4.3.2 · pp. 21–22

4.3.3 Test-Time Scaling

• Test-time scaling allocates extra computation to reasoning, query-aware frame selection, iterative revisiting, or answer-path verification, trading latency and repeated visual computation for broader evidence exploration.Explicit operations to fetch crops, frames, retrieval results, or persistent memory mark the gradual boundary with agentic modeling.Source: §4.3.3 · p. 22

4.4 Agentic Video Modeling

• Agentic video modeling treats evidence acquisition as sequential decisions: agents select regions, invoke tools, update reasoning, and use search, persistent memory, or specialized roles to answer.Controllers can replan after new observations, search structured video collections, anticipate relevant stream events, or maintain coarse-to-fine and entity-centric memories.Source: §4.4 · p. 22, §4.4.1 · pp. 23–24

4.4.1 Video Agents

• Video agents combine backbones, observation policies, tool outputs, and memory, while trained search policies can reduce tool cost but tie agents to tools seen in training.Benchmarks assess evidence search, multi-hop retrieval, evidence milestones, and search cost; REPAIR [3] uses paired runs to separate method effects from run-to-run variance.Source: §4.4.1 · p. 24

4.4.2 Tool Use and Retrieval Augmentation

• Tool use fetches finer local evidence on demand, whereas retrieval indexes narrow searches across long videos or corpora.SceneRAG indexes narrative scenes and entity relations for multi-hop retrieval, but depends on segmentation and graph construction; distilling agent traces into one-pass inference trades flexibility for simplicity.Source: §4.4.2 · p. 25

4.5.1 Embodied and Egocentric Perception

• Egocentric assistants must track manipulated objects, task steps, and explanatory earlier events despite occlusion and long stretches of routine activity.EgoTools [447] tests physical-tool affordance, grounding, procedure, and spatial reasoning; EgoPolice covers body-worn footage with rare consequential events and poor visibility.Source: §4.5.1 · p. 25

4.5.2 Autonomous Driving

• Driving comprehension links road geometry, traffic participants, and their relations to prediction, planning, and closed-loop control.DriveLM uses graph-structured VideoQA, NuScenes-SpatialQA tests spatial relations, and OmniDrive adds counterfactual questions about alternative actions.Source: §4.5.2 · p. 25

4.5.3 Domain-Specific Video Analysis

• Surgical video requires structured phase, anatomy, and action evaluation, while sports analysis must localize brief rule-dependent events and instructional video must track procedural progress.CholecT50 and Rendezvous refine surgical outputs into instrument–verb–target triplets; SoccerNet-v2 adds shot segmentation and replay grounding; CrossTask and COIN annotate instructional steps.Source: §4.5.3 · pp. 25–26
• VEU-Bench finds general Video-LLMs handle editing attributes poorly, in some cases below chance.The benchmark tests recognition, reasoning, and judgments of editing attributes within and across shots.Source: §4.5.3 · p. 25
• Video understanding is limited by what encoders, Video-LLMs, and agents keep and ground, not by how many frames they accept.Evaluation should test whether added evidence improves the state estimate, remains traceable to source clips, and survives later queries.Source: §4.5.3 · p. 26

• Writing renders a new video trace from a partial world specification supplied as text, images, audio, motion, or other conditions.Source: §5 · p. 26
• Generation conditions fix some entities, relations, and dynamics while leaving the rest to the model.Text may specify an event, an image an initial appearance, audio a visible rhythm or articulation, and trajectories camera or object motion.Source: §5 · p. 26
• This chapter evaluates video output by fidelity, condition adherence, temporal coherence, and controllable revision, while treating persistent state under occlusion, revisits, long rollouts, and actions in Section 7.Source: §5 · p. 26

5.1 Tasks, Conditions, and Evaluation

• A generation task is defined by its presynthesis conditions, and evaluation assesses visual and temporal quality, condition adherence, and stability of entities, events, and physical relations.Source: §5.1 · p. 26

5.1.1 Generation Tasks by Condition

• Text, image, video, audio, camera, motion, identity, and variable-input conditions define distinct generation tasks and constrain what the model must preserve or produce.Any-conditioned systems must bind cues to the correct entity, region, time span, or modality and reconcile cues of different granularity.Source: §5.1.1 · pp. 26–27
• Training pairs must expose each task’s condition–output correspondence, from caption corpora for text-to-video to synchronized tracks for audio-visual generation.Editing uses source–instruction–target triplets; identity control pairs references and targets from different contexts or shots.Source: §5.1.1 · p. 27

5.1.2 Evaluation and Metrics

• Evaluation profiles vary by task: text-to-video emphasizes semantic alignment, editing source preservation and locality, audio-driven generation synchronization, and motion control trajectory adherence.A single aggregate score can hide the failure that limits a system.Source: §5.1.2 · p. 27
• Metrics probe different properties: FID ignores temporal order, while condition-aware measures test prompt alignment and compositionality, and physical benchmarks test commonsense and mechanics.FVD depends on its feature extractor and sample count; some source-preservation metrics penalize valid alternative futures.Source: §5.1.2 · p. 27, Table 10 · p. 29
• Human and learned evaluators capture judgments beyond fixed feature distances, but learned scores inherit annotation coverage and biases.Evaluation suites combine perceptual, semantic, and temporal dimensions, with VBench-2.0 extending to human fidelity, controllability, creativity, physics, and commonsense.Source: §5.1.2 · pp. 27–28
• Multi-shot and audio-visual evaluation checks cross-shot consistency, identity persistence, sound-event generation, and synchronization over settings extending to minute-scale sequences.Fréchet Audio Distance ignores video, while LSE-D and LSE-C measure speech–mouth alignment.Source: §5.1.2 · p. 28

5.2 Video Latent Representation Learning

• Compressed latent spaces reduce training and sampling cost, but their bottlenecks and decoders determine whether fine texture, fast motion, and small objects survive.Figure 4 contrasts continuous, discrete, hybrid, and semantically aligned designs.Source: §5.2 · p. 29, Figure 4 · p. 31

5.2.1 Continuous Latents

• Per-frame VAEs preserve reusable image priors, while spatiotemporal and causal VAEs shorten sequences; aggressive temporal compression can smooth brief events or fast motion.Causal 3D VAEs avoid future-frame leakage and can serve both image and video generation.Source: §5.2.1 · pp. 29–30

5.2.2 Discrete Tokens

• Discrete video tokenizers determine token sequence length and may use fixed grids or content-adaptive budgets, while codebook-free quantizers avoid learned codebooks that can collapse.Adaptive methods allocate more tokens to segments that need them, and hierarchical codes can separate semantics from fine detail.Source: §5.2.2 · p. 31

5.2.3 Hybrid and Semantically Aligned Representations

• Hybrid representations let continuous latents serve diffusion or flow models and discrete codes serve sequence models, while semantic alignment seeks codes useful for both reconstruction and understanding.V-RAE finds that reconstruction ranking does not consistently predict generation quality, so semantic latents must be tested inside the target generator.Source: §5.2.3 · pp. 31–32

5.3 Generative Video Foundation Modeling

• Generation systems combine model-level objectives and prediction orders with planning systems, and may layer preference optimization over the generator.A system can denoise chunks while rolling out autoregressively, and agents can repeatedly call a flow-based generator.Source: §5.3 · p. 32

5.3.1 Diffusion-Based Generation

• Diffusion models learn denoising targets over corrupted video latents, with backbone attention designs trading interaction scope against computational cost.Streaming variants generate causally by chunks, while long-video methods address gradual loss of appearance and global structure across windows.Source: §5.3.1 · pp. 34–35

5.3.2 Autoregressive and Masked Token Generation

• Autoregressive models predict ordered video units with flexible sequence length but face long decoding chains and errors that propagate through the prefix.Masked generation predicts selected positions from visible context, often through coarse-to-fine refinement or parallel within-frame decoding.Source: §5.3.2 · p. 35

5.3.3 Flow-Based Generation

• Flow matching learns a time-dependent vector field and generates by integrating an ordinary differential equation, with rectified-flow paths that can reduce solver steps.It can use the same DiT backbones as diffusion, whereas normalizing flows are a distinct likelihood-based approach.Source: §5.3.3 · p. 36

5.3.4 LLM/MLLM-Assisted Generation

• LLM- or MLLM-assisted systems structure conditions such as scenes, shots, layouts, or prompts for a separate visual generator, while tighter couplings can interleave language and video modeling.The language model can handle composition and timing while the video model handles appearance and motion.Source: §5.3.4 · p. 36, Figure 6 · p. 36

5.3.5 Agentic Generation

• Agentic generation makes planning executable through a controller that calls tools, inspects intermediate outputs, and revises plans or conditions.Recurring stages include scripting, storyboarding, visual generation, evaluation, and revision.Source: §5.3.5 · pp. 36–37, Figure 6 · p. 36

5.4 Control Strategies

• Control routes are classified by the component through which the signal enters, from inputs and attention maps to latent states, features, parameters, or sampling updates.A single system may combine routes, such as source inversion followed by attention reuse and classifier-free guidance.Source: §5.4 · p. 37, Table 13 · p. 38

5.4.1 Attention-Based Control

• Attention control reuses prompt-to-feature and visual correspondences to edit content while retaining layout, and video methods extend this with inversion or cross-frame feature propagation.TokenFlow propagates diffusion features between frames to produce temporally consistent edits without training.Source: §5.4.1 · pp. 37–38

5.4.2 Adapter-Based Control

• Adapters inject external structure through trainable branches or residual features while keeping most or all of a pretrained generator frozen.Video adapters encode temporal conditions, and unified adapters can train several controls together.Source: §5.4.2 · pp. 38–39

5.4.3 Latent-Space Control

• Latent-space control changes the noisy sample or denoising trajectory, with partial noising trading input fidelity against edit strength and inversion enabling condition or latent changes along a reverse path.Latent conditions can also be constructed directly from point trajectories or planned motion.Source: §5.4.3 · p. 39

5.4.4 Concatenation and In-Context Conditioning

• Concatenation injects aligned condition latents or embeddings into the input or token sequence, while in-context generation places examples or source–target pairs in the sequence.FullDiT models heterogeneous conditions in one sequence with full attention, avoiding separately trained side branches.Source: §5.4.4 · p. 39

5.4.5 Guidance-Based Control

• Classifier-free guidance combines conditional and unconditional predictions at each denoising step, and increasing its guidance scale usually strengthens alignment but reduces diversity.More general guidance can use gradients from external objectives or the generator’s internal attention and feature maps.Source: §5.4.5 · p. 39

5.4.6 Tuning-Based Control

• Tuning stores a subject, style, or source-video prior in learned embeddings or weights, while video tuning must preserve identity under motion and viewpoint changes.Video2LoRA predicts reference-specific LoRA weights in one forward pass, avoiding per-reference optimization.Source: §5.4.6 · pp. 39–40

5.5.1 Creative Content Production

• Video generation enters production as concept shots, narrative footage, advertising or educational variants, and visual-effects edits, through both closed products and released-weight models.Production also requires consistency across shots and revision that does not disturb unaffected elements.Source: §5.5.1 · p. 40, §5.5 · p. 40

5.5.2 Avatars and Character Animation

• Avatar systems map audio or pose controls to visible behavior, while real-time use adds sustained frame rate, bounded delay, and synchronization requirements.MuseTalk reports 30 frames per second at 256 × 256 resolution on a V100 GPU.Source: §5.5.2 · p. 40

5.5.3 Video Plans for Embodied Agents

• Generated video can act as a visual plan from which actions are recovered through inverse dynamics, frame correspondences, reusable primitives, or tracked tool motion.A generated plan stays fixed, unlike an interactive simulator that updates predictions as actions arrive.Source: §5.5.3 · pp. 40–41

5.5.4 Synthetic Data for Embodied AI and Driving

• Synthetic videos can vary driving or robotics conditions for costly or unsafe-to-collect scenarios, but generated data are judged by downstream results on real observations rather than visual fidelity.Robotics data also need action labels, which can be supplied through latent-action or inverse-dynamics models.Source: §5.5.4 · p. 41
• Rendering fidelity and stateful control are separate goals, so control should be evaluated by comparing matched rollouts with and without it, not by rollout quality alone.A good control changes only its target while preserving identity, geometry, and motion over time.Source: §5.5.4 · p. 41

6.1 Unified Architectures

• Video systems meet through shared token sequences, continuous latent spaces, semantic bridges between experts, or agent task states; these architecture families can be combined.The sharing relation links inferred content to generation and generative structure to understanding and reasoning.Source: §6.1 · p. 41, §6 · p. 41, Figure 8 · p. 42

6.1.1 Discrete-Token Unification

• Discrete-token unification gives many input–output combinations one vocabulary and likelihood objective, but tokenizer information loss and long autoregressive prefixes can limit fidelity and accumulate errors.Continuous latent models avoid quantization and fit diffusion or flow generation, yet understanding favors appearance invariance while generation needs detail.Source: §6.1.1 · p. 44, §6.1.2 · p. 44

6.1.3 Decoupled and Modular Unification

• Decoupled systems preserve specialized pathways and replaceable pretrained components, but semantic bridges may omit appearance, geometry, or motion needed by video decoders.Agentic systems instead share controller-maintained task state while delegating work to tools, making workflows dependent on tool selection, intermediate formats, and evaluator reliability.Source: §6.1.3 · pp. 44–45, §6.1.4 · p. 45

6.2 Learning Paradigms

• Learning ranges from isolated experts connected after pretraining, through inverse-task or cycle-based dual learning, to joint co-learning; isolated systems cannot transfer across directions.Dual objectives can provide supervision when paired labels are scarce, while co-learning risks objective imbalance because understanding favors invariance and generation detail.Source: §6.2 · p. 45, §6.2.1 · p. 46, §6.2.2 · p. 46

6.2.4 Synergistic Learning

• Video synergy evidence includes understanding-guided generation and editing, reciprocal data-mixture gains on VBench and MVBench, and improved action and temporal hallucination outcomes from counterfactual clips.Integrated tasks also test decision effects, while controlled bidirectional tests remain limited apart from Lance’s data-mixture ablations.Source: §6.2.4 · pp. 46–47

6.3 Applications

• Applications use shared models for instruction-driven editing, generated intermediate videos for reasoning, and predicted futures to refine driving or embodied plans.Generated reasoning traces can be coherent yet unsupported, so observed evidence should be distinguished from generated hypotheses.Source: §6.3.1 · p. 47, §6.3.2 · p. 47, §6.3.3 · p. 47

6.4 Evaluation

• Evaluation separates per-direction competence, integrated behavior, and measured synergy; video coverage is mature at the first level, emerging at the second, and mostly directional at the third.Per-direction scores should use community protocols and matched single-task baselines; integrated benchmarks assess transformations, while synergy requires matched routes and controlled cross-direction comparisons.Source: §6.4 · p. 48, §6.4.1 · p. 48, §6.4.2 · pp. 48–49

6.4.3 Protocols and Consistency

• Round-trip protocols should score preserved entities, event order, geometry, and dynamics separately from intended changes, while controlling for leakage, overlap, and judge bias.Architectural sharing or mixed input–output tasks alone do not demonstrate synergy; controlled ablation of the cross-direction path is needed.Source: §6.4.3 · p. 49

7 „ Video World Models: Interacting with the World

• Rollouts expose feedback errors, forgotten off-screen content, broken physical or causal structure, and actions that miss their intended effects.Interacting advances state over time and decodes observations when visible output is needed.Source: §7 · p. 49

7.1 Architectures and Training Paradigms

• Figure 10 classifies models by observation versus latent-state prediction and passive, fixed-action, or feedback-driven transitions; training data range from video mixtures to synchronized controls and annotated interventions.One model can occupy several cells depending on its evaluated protocol.Source: §7.1 · pp. 49–50, Figure 10 · p. 50

7.1.1 Observation-Space Passive Prediction

• Passive observation-space predictors forecast from video history alone, modeling multiple plausible futures or continuing a video prefix without action input.They provide dynamics priors and uncertainty estimates for controlled prediction.Source: §7.1.1 · p. 50

7.1.2 Observation-Space Action-Conditioned Prediction

• Action-conditioned observation models receive controls for an entire segment before generation, while interactive variants require causal generation that accepts new controls during rollout.Controls include trajectories, camera poses, robot actions, maps, and planned object states.Source: §7.1.2 · p. 50

7.1.3 Observation-Space Interactive World Models

• Interactive visual models turn emitted frames into history and let each new action alter the ungenerated future, including in multi-actor shared worlds.Examples include game simulators, playable generated scenes, and systems that bind actions to agent identities.Source: §7.1.3 · p. 51

7.1.4 Latent-State Passive Prediction

• Latent passive predictors learn video-derived features, but masked-region prediction using frames on both sides is not causal forecasting and needs a separate future-prediction protocol.These models primarily supply feature spaces and initialization for action-conditioned latent transitions.Source: §7.1.4 · pp. 51–52

7.1.5 Latent-State Action-Conditioned Prediction

• Latent action-conditioned models predict feature evolution under controls, allowing candidate action sequences to be compared with goal representations without rendering frames.The best combination of encoders, context, losses, and planning optimizers depends on the task.Source: §7.1.5 · p. 52

7.1.6 Latent-State Interactive World Models

• Interactive latent systems close the loop with model predictive control, but latent distance can omit task-relevant details, so scorer audits complement closed-loop success.Components may specialize rendering, causal rollout, and compact decision state, while agreeing on consequences, retained state, and scores for the same history and action.Source: §7.1.6 · p. 52

7.2 Core Technical Challenges

• Memory and physical consistency matter across the model grid, while controlled prediction must bind actions to consequences within a response budget.Longer memory raises inference cost, fewer denoising steps can change rollout stability, and stronger action constraints can disturb pretrained visual priors.Source: §7.2 · p. 52

7.2.1 Action-Conditioned Controllability

• Action control requires either learning a latent variable from observed transitions or aligning supplied controls with the right time, view, object, and consequence.Latent actions can encode distractor or camera motion; limited action supervision can anchor them to the agent, and controls may be injected frame by frame or as visual action fields.Source: §7.2.1 · pp. 52–53

7.2.2 Long-Horizon Consistency and Memory

• Long-horizon errors accumulate when model outputs return as context, while finite context loses entities, layouts, and events no longer visible.Training-inference alignment methods expose models to uncertain histories or their own rollouts to reduce the mismatch with deployment.Source: §7.2.2 · p. 53
• Memory systems retrieve past views, use geometry to reduce false matches, or model off-screen object progression; these approaches preserve different kinds of evidence.Retrieval can extend storage and retrieval costs, while compressed and fixed-size memories trade capacity against latency and may drift or lose details needed for rendering.Source: §7.2.2 · pp. 53–54
• Compact memories can bound context, but recurrent state may drift and test-time weight updates add optimization latency.Fixed-size memory methods include recurrent queries, geometry-supervised tokens, and fixed-size spatial or spatiotemporal feature planes.Source: §7.2.2 · p. 54

7.2.3 Real-Time Responsiveness

• Responsiveness measures the delay until an action affects the first output, so display frame rate alone can overstate control-loop speed.Acceleration uses few-step distillation, sparse attention, feature caching, and quantization, but control-sensitive errors can propagate through a rollout.Source: §7.2.3 · pp. 54–55

7.2.4 Physical and Causal Grounding

• Physical grounding concerns contact, rigidity, gravity, and persistence, while causal grounding tests intervention effects and whether unaffected factors stay fixed.Intervention-level supervision and evaluation remain less developed than methods for plausible dynamics.Source: §7.2.4 · p. 55
• Physics can enter through explicit simulation, learned physical alignment, or inference-time planning, but deformable, fluid, fractured, articulated, and partly observed contact remain hard to recover and simulate.Intervention-level evidence varies one action or physical parameter from a fixed initial state and compares both the affected future and unchanged state.Source: §7.2.4 · p. 56

7.3 Applications

• Applications range from generating datasets and training or evaluating policies to forecasting for planners and responding directly to people.The first three support decisions before deployment; planning and human simulation consume predictions online.Source: §7.3 · p. 56

7.3.1 Data Engines

• As data engines, controllable world models generate labeled examples for settings where real observations are expensive, dangerous, or unevenly distributed.Robotics examples include action-labeled trajectories, while driving generators can supply unusual layouts and safety-critical cases.Source: §7.3.1 · p. 56

7.3.2 Policy Training in Imagination

• Imagination training places the world model inside policy optimization, where policy-selected trajectories shape the generated distribution and model errors compound along them.Useful simulation requires accurate action effects, rewards, and resets; updated-policy rollouts can be fed back into the model.Source: §7.3.2 · p. 56

7.3.3 Policy Evaluation Environments

• Policy-evaluation environments replace repeated deployment with generated trials, and their validity depends on preserving real-world policy rankings.A simulator tuned on familiar checkpoints may misrank controllers that visit new states or exploit modeling artifacts.Source: §7.3.3 · p. 57

7.3.4 Predict-then-Act Planning

• Predict-then-act methods use forecasts as evidence for decisions, trading inspectable pixel plans against the cost and error of generation or cheaper, less inspectable latent scores.Some approaches predict under candidate actions, while others train representations using future-prediction supervision and omit the World Expert at test time.Source: §7.3.4 · p. 57

7.3.5 Interactive Simulation for Humans

• Human-interactive simulators accept actions and produce evolving observations, ranging from real-time game worlds to voice-controlled characters and user-steered driving futures.Fixed-control models accept edits between rollouts; interactive systems accept input within the stream.Source: §7.3.5 · p. 57

7.4 Benchmarks and Evaluation

• Evaluation requirements rise from visual and motion quality to action fidelity, persistent state, bounded response time, and downstream decision utility.Claims should progress from visual plausibility through controllable rollout to decision utility.Source: §7.4 · p. 57

7.4.1 Challenge-Aligned Benchmarks

• Challenge-aligned benchmarks test action following, memory under occlusion, physical and temporal validity, real-time response, and calibrated future distributions.Protocols vary in action space and horizon; memory tests distinguish static recall from off-screen change, while stochastic tests compare sampled outcomes with reference probabilities or valid outcomes.Source: §7.4.1 · p. 57, Long-horizon consistency and memory · pp. 58–59, Table 19 · p. 58

7.4.2 Application-Aligned Benchmarks

• WorldArena finds that perceptual-quality and data-engine, policy-evaluation, and planning-utility rankings need not agree.Application-aligned benchmarks also compare model predictions with real robot trials, test planning transitions, or execute generated manipulation videos in a physics simulator.Source: §7.4.2 · p. 59

7.4.3 Foundational and Comprehensive Benchmarks

• Comprehensive suites vary in protocol, testing controllability, dynamics, interaction, physical faithfulness, and utility; shared tooling can hold data, planners, and environments fixed.Evidence for world models should match the transition protocol: controlled models must change the world where an action acts while preserving what it leaves untouched.Source: §7.4.3 · pp. 59–60

8.1 Long-Horizon World Consistency

• Predictive memory must update hidden state during occlusion, while evaluation separates state loss from rendering drift across rollout lengths and revisit distances.Such memory complements rollout training that targets error growth; interactive memory must also fit a budget, and reports should include memory size, latency, and action-update rate.Source: §8.1 · pp. 60–61

8.2 Physical and Causal Grounding

• Physical grounding needs intervention-paired data that tests changed outcomes and invariance of unrelated state, while diagnosing where violations first appear.Existing physics methods depend on external sources such as teacher coverage, recovered state and material parameters, or feedback and retrieved evidence.Source: §8.2 · p. 61

8.3 Measurable Synergy in Unified Video Models

• Unified models need matched systems differing only in cross-direction information flow, with transfer measured on temporal order, motion, identity, or physical dynamics.Current video evidence is partial or one-directional, and a round trip between inferred trajectories and rendered video can serve as both training signal and test.Source: §8.3 · pp. 61–62

8.4 Standardized Evaluation of World Properties

• World-model scores are rarely comparable across suites, so evaluation should report protocol, horizon, control frequency, and judge, and connect predictions to decisions.Video predictions need visual and trajectory checks, feature predictions need state probes, and both need decision tests; latent probes and fixed-candidate action ranking are also needed.Source: §8.4 · p. 62

8.5 Scaling Control from Unlabeled Video

• Reconstruction accuracy and codebook usage do not establish control: latent actions should predict controllable effects and remain stable across camera and background changes.Limited action supervision can improve recovered factors, while pseudo-action training pairs need validation in controlled environments or real systems to avoid reinforcing shortcuts.Source: §8.5 · p. 63

8.6 Generalization across Scenes and Embodiments

• Generalization claims depend on held-out factors: scene transfer differs from embodiment transfer, and tests should include unseen scene–body pairs and effect equivalence.Reports should state whether transfer is zero-shot, calibrated, or fine-tuned, along with target-data use; evaluation should cross scenes, embodiments, sensors, and action spaces.Source: §8.6 · pp. 63–64

8.7 Stochastic Worlds and Rare Events

• Stochastic models must assign calibrated probabilities to valid futures, not merely generate diversity; evaluation should assess frequency, mode coverage, and invalid versus unreadable samples.Generated rare cases also require checks of physical feasibility, causal relevance, and downstream utility before entering training data.Source: §8.7 · p. 64, Figure 17 · p. 64

9 Conclusion

• A video world model is defined by its learned transition, which predicts future video or video-derived features and may run passively, under fixed actions, or interactively.The survey distinguishes the quality of rendered video from evidence that a model tracks the world state behind it.Source: §9 · p. 65
• World-model evidence should include state surviving occlusion and long horizons, valid intervention responses, and latency within the control budget for closed-loop use.Stochastic futures require coverage and calibration, unified models require matched evidence of information transfer, and decision-oriented systems must improve choices in controlled or real tasks.Source: §9 · p. 65