PaperlayerView exact revision on arXiv

Source-linked AI summary

Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models

Ziqi Ma, Mengzhan Liufu, Georgia Gkioxari

arXiv:2603.13215v1cs.CV

TL;DR

Video world models are expected to evolve state even when it is not observed, but existing benchmarks do not test this capability. StEvo-Bench interrupts observation across diverse processes and uses automatic verifiers to assess outcomes. The benchmark finds that models struggle to evolve state under interruption, with differing failure patterns across model types.

  • Problem

    Existing benchmarks cover physical plausibility and coherence but do not test whether state evolution continues when observation is interrupted.

    Source: §1 · p. 2

  • Method

    StEvo-Bench evaluates 225 tasks across six evolution categories by interrupting observation with occlusions, darkness, or camera lookaway and applying specialist verifiers.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    General-purpose video models often stop evolving or become incoherent under observation control, while camera-controlled models strongly favor static scenes and have difficulty controlling the camera when state evolves.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark exposes distinct failure modes and potential data and architecture biases in current video world models.

    Source: §1 · p. 2

    Some models are evaluated only qualitatively because of their short, 41-frame context.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Evolutions in the world, such as water pouring or ice melting, happen regardless of being observed. Video world models generate "worlds" via 2D frame observations. Can these generated "worlds" evolve regardless of observation? To probe this question, we design a benchmark to evaluate whether video world models can decouple state evolution from observation. Our benchmark, STEVO-Bench, applies observation control to evolving processes via instructions of occluder insertion, turning off the light, or specifying camera "lookaway" trajectories. By evaluating video models with and without camera control for a diverse set of naturally-occurring evolutions, we expose their limitations in decoupling state evolution from observation. STEVO-Bench proposes an evaluation protocol to automatically detect and disentangle failure modes of video world models across key aspects of natural state evolution. Analysis of STEVO-Bench results provide new insight into potential data and architecture bias of present-day video world models. Project website: https://glab-caltech.github.io/STEVOBench/. Blog: https://ziqi-ma.github.io/blog/2026/outofsight/

1. Introduction

The paper asks whether video world models can keep evolving when observation is interrupted, a capability needed as generated worlds extend beyond what a viewer sees. StEvo-Bench tests this gap and reveals distinct failure patterns across video and camera-controlled models.

1. Introduction

  • Decoupling requires plausible physical evolution, coherence of non-evolving aspects, and state progress during periods when the process is unseen.For example, water should continue to accumulate while a camera is turned away from pouring.Source: §1 · pp. 1–2
  • Current benchmarks cover physical plausibility and coherence, but not whether processes continue evolving while unobserved.The gap matters for larger worlds and longer horizons, where most of the generated world is unseen.Source: §1 · pp. 1–2
  • StEvo-Bench evaluates 225 tasks across six evolution categories by interrupting observation with occlusion, darkness, or camera lookaway.The benchmark also introduces specialist VLM-based verifiers to detect and disentangle failure modes.Source: §1 · p. 2, Figure 2 · p. 2
  • General-purpose video models often stop evolving or become incoherent under observation control; camera-controlled models strongly favor static scenes and struggle more with camera control when evolving state.Memory-based architectures further exacerbate the static-scene bias and do not help decouple evolution from observation.Source: §1 · p. 2

2. Related Works

Related work spans video world models, their evaluation, and approaches that explicitly represent world state. StEvo-Bench focuses on testing evolution during interrupted observation, which prior evaluations and stateful-model settings do not comprehensively address.

2. Related Works

  • Video world models include general-purpose image- or text-prompted generators [11] [28] [33] and action-conditioned generators, often using camera navigation [5] [25] [26].Other action spaces include robotic end-effector motion [1] and time plus camera control [29].Source: §2 · pp. 2–3
  • Evaluation work covers video quality, physics, and consistency, while StEvo-Bench tests whether processes evolve without full observation.Examples include physics benchmarks VideoPhy [3] and VideoPhy2 [4], and consistency and memory benchmarks World Consistency Score [23], Stable World [12], and MIND.Source: §2 · p. 3
  • Stateful models represent state as memory [32] [14] [31], 2.5D/3D [17, 37, 9], or latent state [40] [2] [22] [18]; prior work is generally constrained to simple, synthetic environments.Memory- and 3D-based states are described as biased toward static scenes; FloWM discusses latent-state evolution in synthetic block worlds.Source: §2 · p. 3

3. Benchmarking State Evolution in Video World Models

StEvo-Bench tests naturally occurring processes as observation is interrupted, then uses staged, specialist verification to assess control and evolution. Its criteria cover both successful interruption and correct, coherent state change.

3. Benchmarking State Evolution in Video World Models

  • The benchmark contains 225 tasks across six categories of everyday physical evolutions, including continuous processes, kinematics, relational and causal change, transformations, and expected human or animal behavior.Tasks are grounded in interactions such as pouring water, heating a pan, or activating a sprinkler.Source: §3 · p. 3

3.1. Task Construction

  • Each task pairs an initial image with a prompt specifying an action that starts an evolution and an observation interruption mid-process; the main object's state is assessed before and after interruption.Video models use in-scene occluders or darkness, while camera-controlled models move the object out of view and return the camera.Source: §3.1 · pp. 3–4

3.2. Evaluation Criteria and Automatic Verifiers

  • Evaluation first checks whether action and observation controls succeeded, then requires state progress, physical plausibility, and temporal coherence for task success.Videos that fail either control check are excluded from evolution evaluation because the process was not properly initiated or partially observed.Source: §3.2 · p. 4
  • Unlike prior benchmarks that cover only subsets of the criteria, StEvo-Bench evaluates them together to quantify evolution success and disentangle failure modes.The five verifier roles divide into two control checks and three evolution checks.Source: §3.2 · p. 4, Figure 3 · p. 4
  • Specialist verifiers assess separate binary criteria, enabling diagnosis of entangled failures and narrowing each VLM judgment to a focused yes/no question.State progress checks for continued change during the unobserved interval; physics checks active violations, while coherence checks for unexplained disappearance, teleportation, abrupt objects, or scene resets.Source: §3.2 · pp. 4–5

4. Do Video World Models Evolve State Successfully?

Across video and camera-controlled models, interrupted observation exposes difficulty decoupling state evolution from what the camera sees. Video models often stall or lose coherence, while camera-controlled models tend to freeze scenes or neglect camera motion when evolution occurs.

4.1. Model Candidates

  • The evaluation includes general-purpose video generators and camera-controlled models, with VMem and AETHER assessed qualitatively because of their artifacts or short context window.The model set includes Veo 3, Sora 2 Pro, WAN 2.2, CogVideoX 1.5, HunyuanVideo 1.5, Genie 3, HunyuanWorld 1.0, LingBot-World, GEN3C, VMem, and AETHER.Source: §4.1 · p. 5

4.2. Findings

  • Both model groups have under 10% task success under interrupted observation; Veo 3 and Sora 2 Pro reach 17% and 13% state progress, while camera-controlled models remain below 5%.The reported state-progress rates for Veo 3 and Sora 2 Pro are occasional exceptions, not task-success rates.Source: §4.2 · p. 5, Table 2 · p. 6
  • With observation control, video models commonly stall evolution or lose object coherence, despite succeeding on the showcased processes when fully observed.Open-source models are especially prone to low state progress; state-of-the-art closed-source models achieve only about 60% coherence success.Source: §4.2 · p. 5, Figure 4 · p. 6
  • The observation-control comparison reports significantly lower state-progress and task-success rates than full observation for Veo 3 and Sora 2 Pro.The metrics are averaged over those two models.Source: §4.2 · p. 6, Table 3 · p. 6
  • Camera-controlled models largely keep scenes static as the camera turns, failing to evolve processes; this pattern appears across models and is especially evident in open-source systems.GEN3C strongly favors static-scene generation, and AETHER also fails to evolve state while its short context window limits object recall.Source: §4.2 · p. 7, Figure 5 · p. 7
  • When camera-controlled models do generate dynamics, they often ignore the requested camera trajectory, indicating difficulty achieving evolution and camera motion concurrently.Examples include a camera following a rocket upward instead of turning right, and another camera remaining static despite the input trajectory.Source: §4.2 · p. 7, Figure 6 · p. 8
  • The findings suggest that training data may contribute to static-scene bias and the trade-off between dynamics and camera control; the authors point to different synthetic-data strategies or preference optimization as possible responses.The data discussion mentions rendered static scenes, general video datasets dominated by static objects, and gaming data lacking rich natural physical evolutions.Source: §4.2 · pp. 7–8
  • VMem recalls the initial frame accurately but exhibits static-scene bias rather than better state evolution, alongside artifacts and scene incoherence.The authors argue that its memory design favors memorizing object appearance over modeling evolution.Source: §4.2 · p. 8, Figure 7 · p. 8

5. Control and Verifier Analysis

The section checks whether benchmark models can apply observation and action controls, then tests whether automatic verifiers agree with human judgments. Verifiers match or exceed human agreement on nearly all criteria and recover the same model ordering.

5.1. Control success rate

  • Models can correctly apply both observation and action control, supporting the validity of the evaluation protocol.The control success rates are reported in Table 5.Source: §5.1 · pp. 8–9, Table 5 · p. 9

5.2. Evaluation of verifiers against human annotators

  • Verifier validation used binary labels from three annotators on a random sample of 180 generated videos spanning all models.Agreement was evaluated with accuracy, ROC-AUC, and model ranking agreement.Source: §5.2 · p. 9
  • Accuracy measures binary-label matches, ROC-AUC accounts for class imbalance, and model ranking agreement scores raters’ consistency in comparing model pairs on shared tasks.Model ranking agreement gives partial credit when one rater reports a tie and the other does not.Source: §5.2 · p. 9
  • Human agreement is lowest for physical plausibility and observation control, where judgments involve nuanced thresholds and subjective assessments of control completeness.The verifiers nevertheless match or exceed inter-human agreement on nearly all criteria.Source: §5.2 · p. 9
  • Verifier-human model-ranking agreement meets or exceeds human-human agreement on every criterion, and accuracy is higher for state progress, observation control, and action control.Physical plausibility is the only criterion where verifier accuracy lags slightly; it is also the hardest for humans.Source: §5.2 · pp. 9–10, Table 4 · p. 9

6. Conclusions

StEvo-Bench evaluates whether video world models continue scene evolution when observation is interrupted. Its findings include stopped or incoherent evolution in video models and static-scene bias in camera-controlled models.

6. Conclusions

  • StEvo-Bench probes evolution under interrupted observation using occlusions or dimmed illumination for video models and lookaway commands for camera-controlled models.The benchmark’s verifiers separate different failure modes.Source: §6 · p. 10
  • Video models show evolution-stopping and incoherence failures, while camera-controlled models favor static scenes and have greater difficulty controlling the camera during dynamic evolution.Source: §6 · p. 10
  • The findings prompt reflection on potential training-data and architecture biases, including the efficacy of memory-based architectures.The paper frames StEvo-Bench as an initial step toward evaluating non-pixel-based world evolution.Source: §6 · p. 10

A. Benchmark

The benchmark’s example tasks span varied naturally occurring evolutions and are intended to probe models’ internal simulation capacity across physical, chemical, and social dynamics.

A. Benchmark

  • The example tasks cover six task categories, including gradual continuous evolution, few-object kinematics, transformations, causal change, and expected human or animal behavior.Source: A. Benchmark · p. 13
  • The categories capture distinct kinds of state evolution, from gradual processes to irreversible structural or material changes.Source: A. Benchmark · p. 13
  • Together with varied observation-control methods, the tasks are presented as a comprehensive probe of internal simulation capacity across physical, chemical, and social dynamics.Source: A. Benchmark · p. 13

B.1. Qualitative Examples

The appendix illustrates verifier-detected failures in state evolution and control, along with examples of the rationale verifiers provide.

B.1. Qualitative Examples

  • Figures 9 and 10 show failures in state progress, physical plausibility, or coherence, as well as failures to apply observation control or initiate the evolution.Source: §B.1 · p. 13
  • Figure 9 focuses on incorrect evolution, while Figure 10 focuses on adherence to specified observation and action controls.Source: §B.1 · p. 13

B.2. Example Verifier Rationale

  • Figure 11 gives verifier-rationale examples in which the verifier analyzes the violating aspect of a criterion.Source: §B.2 · p. 13

B.3. Further Verifier-Human Agreement Analysis

Verifier–human agreement was consistent across comparison pairs and strongly correlated with human–human agreement across every criterion. The authors report that verifiers agree with annotators at least as well as annotators agree with one another.

B.3. Further Verifier-Human Agreement Analysis

  • Low standard deviations across criteria and agreement metrics indicate consistent behavior across verifier–human and human–human comparison pairs.Source: §B.3 · p. 13
  • Verifier–human agreement strongly correlates with human–human agreement for every criterion across tasks.Source: §B.3 · p. 13
  • Verifiers agree with human annotators at least as well as human annotators agree with each other.Source: §B.3 · p. 13

B.4. Disagreements

Verifier–human disagreements often occur on ambiguous videos, while occasional VLM verifier errors can arise from missed details or prompt noncompliance. The verifier prompts separate control success from evolution checks and distinguish absent evolution from active physical violations and coherence failures.

B.4. Disagreements

  • Disagreements often coincide with videos where human annotators also disagree, and many cases involve inherent ambiguity.Source: §B.4 · p. 13
  • Threshold choices and subjective judgments about consequential visual discrepancies make definitive binary labels difficult for some videos.Source: §B.4 · p. 13
  • Occasional VLM verifier errors include prompt noncompliance or overlooked visual details; repeated judgments and vote aggregation mitigate single-lapse contamination in many tasks.Source: §B.4 · p. 13
  • The verifier examples and rationales illustrate separate failures on state-evolution and control axes, with agreement figures comparing verifier–human and human–human judgments.Source: Figure 9 · p. 15, Figure 10 · p. 16, Figure 11 · p. 17

B.5. Verifier Prompts

  • Control prompts assess whether the requested action occurred and whether the intended occlusion made the subject or process invisible; failed controls are not evaluated further.Source: §B.5 · p. 21
  • The state-progress prompt checks whether the expected process occurred during camera movement or occlusion by comparing the subject’s state before and after it becomes visible again.Source: §B.5 · p. 22
  • Physical-plausibility prompts distinguish active physics violations from a missing intended change, which is assessed separately as a state-evolution failure.Source: §B.5 · p. 23
  • Coherence checks treat intended hiding as distinct from unexplained disappearance and distinguish continuous camera pans from instantaneous scene cuts.Source: §B.5 · p. 24

Related papers

Loading 2603.13215v1…