Source-linked AI summary
Do Video Generators Track the World Across Segments? A Benchmark and Method for World-State Reasoning in Video Continuation
Yingmao Miao, Pengfei Zhang, Chaoran Xu, Meng Yu, Jing Tang, Xiangxiang Chu, Chao Shen, Chenhao Lin
TL;DR
Video continuation systems retain visible history but may fail to infer the current world state when prior actions create hidden or changed conditions. The paper introduces StateBench and StateAgent, which explicitly test, update, ground, and render continuation states; StateAgent raises SCS-All from 45.2 to 69.3 and benefits long story generation.
Problem
Current continuation methods expose historical observations as frames or features rather than reliably representing the updated state required after previous actions and new prompts.
Method
StateAgent builds an entity-state representation, updates it under the continuation prompt, grounds the predicted post-action state as a future end frame, and renders the next video.
Results
45.2 to 69.3: StateAgent raises the all-case StateBench score and also benefits long story generation.
Takeaways & Limitations
StateBench and the experiments indicate that coherent continuation benefits from an explicit state belief that can be updated, grounded to appearance, and carried across generation boundaries.
Takeaways & Limitations
The qualitative story-generation analysis identifies fixed visual memory as limited by memory-frame capacity, which can cause repeated backgrounds, incorrect character binding, and loss of small state-relevant objects.
Abstract
from arXiv · showhide
Video generators build long videos by composing shorter parts, either by generating segments one after another or by autoregressively extending chunks. Each new part usually depends on memories of historical observations, such as recent frames, selected key frames, memory banks, or cached features. These memories preserve visible evidence from the past, but current generators do not reliably turn such evidence into a world-state interface: what holds in the video world after previous actions and how it should change under the next prompt. A past frame remains valid history, but it may not describe the state needed by the next segment; some states must instead be inferred from occluded or implicit changes rather than copied from a directly observed frame. This creates a simple but overlooked question for video continuation: given a previous video, its prompt, and a new prompt, can a model generate a continuation that reflects the state determined by both the historical video and the new prompt? To answer this question, we introduce Statebench, a benchmark that targets this gap by testing continuations over three state categories: past-visible states, occluded-process states, and complex-transition states. We further propose Stateagent, which explicitly maintains an entity-state representation, updates it under the new prompt, grounds the predicted post-action state as a future end frame, and renders the next video. Experiments show that our method improves controlled video continuation by raising the all-case state score (SCS-All) from 45.2 to 69.3, and also benefits story generation at the one-minute scale. Code is avaliable at https://github.com/AMAP-ML/StateAgent.
1 Introduction
Video continuation systems preserve historical visual evidence but do not reliably convert it into the updated world state required after previous actions and new prompts. The paper introduces StateBench to test this gap and StateAgent to explicitly reason about, ground, and render the next state.
- Motivation: Sequential short-segment generation requires conditioning each new unit on compressed or selected history because full-history conditioning is costly.Existing systems use recent frames, key frames, memory banks, or related compressed representations.
- Motivation: Observation memory records visible history, whereas state reasoning determines what should hold after an action, including hidden consequences.An object placed inside a container may remain there even when the boundary frame only shows the container.
- StateBench: StateBench tests whether continuations update the historical world state under a new prompt across past-visible, occluded-process, and complex-transition cases.The benchmark contains 200 tasks with task-specific checklists that evaluate both action completion and resulting-state correctness.
- StateAgent: StateAgent decouples state updating from video rendering by constructing entity states, predicting post-action states, grounding them as future frames, and rendering continuations.A VLM parses and updates states, an image editor grounds the predicted state, and keyframe-to-video generation renders the segment.
- Results: 45.2 to 69.3: the method raises the StateBench score and also benefits long story generation over observation-based memory methods.The reported improvement targets controlled video continuation while the paper also reports gains for longer story generation.
2 Problem Formulation
The paper formulates cross-segment continuation as a state-handoff problem: visual observation memory may preserve appearance while failing to expose what is physically true after prior events. State-aware generation therefore infers and updates an explicit state before rendering the next segment.
- 2.1 Cross-Segment State Handoff: Observation memory exposes frames or features, but the next segment requires an updated state describing what should hold after the previous video and new prompt.Historical evidence may be sparse, compressed, stale, hidden, or ambiguous at the segment boundary.
- 2.1 Cross-Segment State Handoff: The tail frame can hide or ambiguate a task-relevant relation established by the historical event, creating an interface gap between visible evidence and post-history state.
- 2.1 Cross-Segment State Handoff: An explicit state representation can distinguish histories that expose similar evidence but imply different post-history states.This representation conditions continuation on what is currently true rather than only on the exposed observation memory.
- 2.2 State as Causal Belief: A world state is defined as task-relevant physical relations that may persist beyond visibility and be updated by the continuation prompt.The update may reveal, use, or further transform the state established by the previous video and its prompt.
- 2.2 State as Causal Belief: Cross-segment generation infers the historical state St and generates Vt+1 according to the state implied by St and the next prompt.The transition abstracts how the physical event described or implied by the continuation prompt updates the carried state.
- 2.2 State as Causal Belief: StateBench represents state using relations covering visibility, location, containment, transformation, surface and material state, quantity, and agent-object interactions.These variables provide controlled tests of whether continuation preserves physical consequences implied by the previous segment.
- 2.3 Observation Memory Versus State Reasoning: Appearance continuity and state continuity are complementary: visual memory preserves how entities looked and scenes were framed, but does not reliably decide what should now be true after an action.
3 StateBench: Measuring State Discontinuity
StateBench measures whether video continuation models preserve and update a task-relevant physical state across segment boundaries when the terminal appearance is hidden or ambiguous. It combines a three-category benchmark with checklist-based evaluation of state exposure, correctness, hallucination, and end-to-end consistency.
- Benchmark design: StateBench tests continuation from a previous video and two prompts, requiring the model to use the established state rather than only the terminal frame.Each task provides a 5–10s previous segment, its source prompt, and a continuation prompt.
- Benchmark design: The benchmark contains 200 cases spanning 85 past-visible, 65 occluded-process, and 50 complex-transition examples.In every case, the relevant state is hidden or ambiguous at the generation boundary, while the next prompt induces an action involving that state.
- Evaluation setup: All generation methods receive the same triplet (Vt, pt, pt+1) and produce one continuation, while reference frames are reserved for evaluation.The evaluator also receives a task-specific checklist and uses reference frames for appearance-sensitive entities.
- Evaluation metrics: State Exposure Score measures whether the continuation completes the requested state-checking or state-updating action before further state evaluation.The exposure indicator ei records successful exposure, and checklist responses yiq record affirmative judge answers.
- Evaluation metrics: Conditional state consistency combines state correctness and anti-hallucination checklist judgments on samples that successfully expose the state.State Correctness is the yes rate on correctness questions, Hallucination Rate is one minus the yes rate on anti-hallucination questions, and SCS-All counts exposure failures as end-to-end state failures.
- Evaluation setup: Checklist judges evaluate generated continuations using the previous video, prompts, task checklist, and reference frames when appearance matching is required.Tables abbreviate the conditional score SCScond as SCS.
4 StateAgent: Future-Frame State Carrier
StateAgent separates post-action state prediction from video rendering by explicitly updating an entity-state graph and grounding the result as a future end frame. The renderer then generates a continuation constrained by historical continuity and the predicted state.
- State handoff: StateAgent decouples state prediction from rendering, predicting what should hold after the continuation prompt before generating the next segment.The parsed graph represents the previous segment’s state, while the predicted state specifies the target for continuation.
- State representation: Its sparse entity-state graph stores identities, visible references, current conditions, and relations needed for future continuation.State descriptions can represent conditions such as open, broken, hidden, held, or stained.
- Prompt-conditioned state update: The prompt-conditioned updater computes Ŝt+1 = U(St, pt+1), an explicit estimate of the state that should appear in Vt+1.The update can reveal hidden relations, reuse existing relations, or transform the physical state.
- Future-frame grounding: StateAgent grounds the predicted state into a future end frame using historical layout and selected reference appearances, including entities absent at the boundary.The edited frame visualizes a future-state prediction rather than retrieving a memory frame.
- Video rendering: A keyframe-to-video generator renders motion between the historical and future end frames, leaving state inference to the preceding modules.This modular structure also makes failures attributable to parsing, updating, grounding, or rendering.
5 Experiments
Experiments compare continuation interfaces on StateBench and evaluate whether explicit state conditioning improves controlled continuation and longer story generation. StateAgent outperforms observation-memory and text-only alternatives by updating states, grounding them visually, and rendering continuations toward them.
- Experimental Setup: StateBench compares frame-conditioned, key-frame or memory-based, and autoregressive continuation baselines under the same task setting.The evaluated interfaces include final-frame I2V, recent-frame conditioning, StoryMem variants, and MAGI.
- Main StateBench Results: 74.9 versus 58.5 conditional SCS, while SCS-All rises from 45.2 to 69.3 with StateAgent over the strongest baseline.SCS measures conditional state consistency, whereas SCS-All treats exposure failure as failure.
- Main StateBench Results: StateBench failures span stale past-visible evidence, hidden state updates, and weak visual cues for spatial, quantity, color, and physical-state transitions.Examples include incomplete box tilting, an empty box despite the apple’s placement, and ambiguous or mismatched objects.
- Ablation on State Conditioning: State-based prompt enhancement reaches SES 90.0 but SCS 61.9, while the full pipeline achieves the best SCS and SCS-All by adding visual grounding and rendering.The comparison attributes the remaining gap to text’s inability to bind predicted state to entity appearance.
- Multi-shot Story Generation: On one-minute ST-Bench stories, StateAgent improves aesthetic quality, prompt following, and cross-shot consistency over the compared methods.The qualitative comparison reports better preservation of entity states and object appearances during scene-level prompt changes.
- Multi-shot Story Generation: A fixed visual memory bank can cause repeated backgrounds, incorrect character binding, and loss or drift of small state-relevant objects across later generations.StateAgent instead updates explicit entity states under each prompt to preserve appearance and state across scene changes.
6 Related Work
Related work extends video generation through chunkwise continuation and visual memory, while newer studies address hidden states and world dynamics. StateBench focuses specifically on preserving and updating states across video-segment boundaries.
- Video Continuation and Visual Memory: Long-video systems extend short-term video generators through chunkwise continuation, selected keyframes, entity or spatial memories, and KV caches.These approaches provide different interfaces for carrying historical visual information into subsequent generation.
- State Evolution and Video World Models: PAN models long-horizon dynamics with latent features, while LiveWorld and ReMind target out-of-sight dynamics using persistent spatiotemporal memory or dynamic KV-cache mechanisms.The cited systems differ in representation and cost, and are not identical to the continuation setting studied here.
- Benchmarks for Generation, Memory, and State Reasoning: Generation benchmarks have expanded from visual quality and text alignment toward memory, state reasoning, and out-of-sight state evolution.WorldReasonBench is cited as evaluating prediction of future world states from a continuation.
7 Conclusion
The paper frames cross-segment state handoff as distinct from observation memory and introduces StateBench and StateAgent to address it. Results indicate that explicit, updateable state beliefs improve continuation and support story generation.
- StateBench tests whether continuations preserve and update states across past-visible, occluded-process, and complex-transition cases.
- StateAgent predicts the continuation state, grounds it as a future end frame, and renders the next segment.
- Stronger visual memory alone does not reliably let current generators infer world state across segment boundaries.
- Coherent continuation benefits from an explicit state belief that can be updated, grounded to appearance, and carried across generation boundaries.
A Experimental Details
This section outlines the additional experimental details provided for the benchmark and method. It covers metadata, implementation, and baseline settings.
- The section provides task metadata and implementation details for StateBench and StateAgent.
- It specifies the settings used for baseline methods.
- The section supplements the main description with experimental configuration details.
A.1 StateBench Data Construction
StateBench constructs controlled continuation cases whose relevant states are established in prior segments but hidden or ambiguous at the boundary. StateAgent represents and updates those states before grounding and rendering the next segment.
- StateBench Data Construction: StateBench pairs a state-establishing previous prompt with a continuation prompt that exposes, uses, or transforms the resulting state without naming the answer.
- StateBench Data Construction: Retained histories must complete the state-changing event, hide the task-relevant state at the boundary, avoid prompt leakage, and support checklist evaluation.
- StateBench Data Construction: The benchmark contains 200 cases: 85 past-visible, 65 occluded-process, and 50 complex-transition cases.
- Case Schema and State Types: Each item records prompts, state annotations, previous video, post-history state, target state, and evaluation checklists.
- Case Schema and State Types: StateBench uses fine-grained state types as controlled probes rather than a complete ontology of world state.
- StateAgent Implementation Details: StateAgent extracts entities, observes boundary states, predicts post-action states, selects references, and generates future-frame editing instructions before video rendering.
B User Study
The user study compares StateAgent with four baselines using blinded pairwise judgments of aesthetic quality, state reveal success, and state consistency. Human preferences favor StateAgent most clearly on state-related dimensions, especially state consistency.
- Study Design: The double-blind study compares StateAgent with final-frame I2V, LongCat-Video, StoryMem, and MAGI-1.
- Study Design: 15 complete submissions produced 450 pairwise answers and 1,350 dimension-level judgments across three evaluation dimensions.
- Scoring: StateAgent receives score 1 for a preference, 0.5 for a tie, and 0 when the baseline is preferred.
- Results: 85.6 state-consistency score rate, 81.3 raw win rate, and 88.8 win rate excluding ties were obtained by StateAgent.
- Results: The state-consistency dimension also had the highest majority agreement, indicating clearer participant consensus about state matching.
- Results: StateAgent’s advantage was consistent across baselines, smallest against StoryMem and largest against MAGI-1.
C.1 Additional StateBench Qualitative Results
Figures 6–13 add qualitative StateBench comparisons using historical-segment frames and generated-continuation frames.
- Figures 6–13 provide additional qualitative comparisons on StateBench.The historical segment is represented by six uniformly sampled frames, while each method row shows three frames from the generated final five seconds.
D Limitation
StateAgent renders continuations between the historical end frame and a predicted future end frame, which can produce shot transitions when the endpoints differ substantially.
- StateAgent currently renders each continuation from the historical end frame to a predicted future end frame.
- Substantial differences between the historical and predicted end frames may introduce shot transitions.
- Richer temporal constraints or intermediate state anchors may help address this limitation.