Source-linked AI summary
LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models
Zicheng Duan, Jiatong Xia, Zeyu Zhang, Wenbo Zhang, Gengze Zhou, Chenhui Gou, Yefei He, Feng Chen, Xinyu Zhang, Lingqiao Liu
TL;DR
Existing video world models freeze unobserved regions because they represent camera-dependent observations rather than continuously evolving world state. LiveWorld addresses this by decoupling evolution from rendering, maintaining a static 3D background and monitor-driven dynamic entities. Its LiveBench evaluation reports persistent event evolution and long-term scene consistency, bridging static 2D memorization and persistent 4D dynamic simulation.
Problem
Current video world models freeze unobserved regions at their last observed timestamps, preventing representation of continuously evolving world dynamics.
Method
LiveWorld factorizes the world into a static 3D background and dynamic entities, using monitors to autonomously fast-forward unobserved entities while decoupling evolution from rendering.
Results
LiveWorld bridges static 2D memorization and persistent 4D dynamic simulation, with LiveBench experiments reporting significant improvement over existing baselines.
Takeaways & Limitations
The framework supports persistent event evolution and long-term scene consistency by maintaining and updating out-of-sight dynamic entities.
Takeaways & Limitations
The implementation adds system latency through language-model, segmentation, and visual-SLAM modules, and monitor frames may yield incomplete foreground point clouds under viewpoint changes.
Abstract
from arXiv · showhide
Recent generative video world models aim to simulate visual environment evolution, allowing an observer to interactively explore the scene via camera control. However, they implicitly assume that the world only evolves within the observer's field of view. Once an object leaves the observer's view, its state is "frozen" in memory, and revisiting the same region later often fails to reflect events that should have occurred in the meantime. In this work, we identify and formalize this overlooked limitation as the "out-of-sight dynamics" problem, which impedes video world models from representing a continuously evolving world. To address this issue, we propose LiveWorld, a novel framework that extends video world models to support persistent world evolution. Instead of treating the world as static observational memory, LiveWorld models a persistent global state composed of a static 3D background and dynamic entities that continue evolving even when unobserved. To maintain these unseen dynamics, LiveWorld introduces a monitor-based mechanism that autonomously simulates the temporal progression of active entities and synchronizes their evolved states upon revisiting, ensuring spatially coherent rendering. For evaluation, we further introduce LiveBench, a dedicated benchmark for the task of maintaining out-of-sight dynamics. Extensive experiments show that LiveWorld enables persistent event evolution and long-term scene consistency, bridging the gap between existing 2D observation-based memory and true 4D dynamic world simulation. The baseline and benchmark will be publicly available at https://zichengduan.github.io/LiveWorld/index.html.
1 Introduction
Existing video world models freeze active entities once they leave view, because they conflate world evolution with camera-dependent rendering. LiveWorld separates these processes, evolves unobserved entities through monitors, and LiveBench experiments show persistent event evolution and improved scene consistency over existing baselines.
- Motivation: Video world models support interactive exploration through camera control but typically preserve temporal consistency using 2D snapshots or reconstructed 3D spatial memory.These representations condition generation on historical context while focusing on the observer’s explored views.
- Problem: Out-of-sight dynamics occur when active entities freeze at their last observed timestamp after leaving the observer’s field of view.Revisiting a dog that was eating would incorrectly show it mid-bite rather than reflecting that it finished eating.
- Method: LiveWorld decouples autonomous world evolution from observation rendering and factorizes the global state into a static 3D background plus temporally evolving dynamic entities.The factorization avoids maintaining a fully dense 4D state while retaining temporal dimensions for sparse dynamic entities.
- Method: A monitor-driven pipeline registers detected dynamic entities, fast-forwards them while unobserved, and projects their updated states with the static environment onto the observer’s camera trajectory.Monitors and rendering use a shared generative paradigm for temporal progression and continuous view synthesis.
- Contributions and evaluation: LiveWorld formalizes out-of-sight dynamics, introduces the monitor-centric framework and LiveBench benchmark, and experiments show it bridges static 2D memorization and persistent 4D simulation.The reported experiments describe successful evaluation of long-horizon out-of-sight dynamics and event permanence, with significant improvement over existing baselines.
2 Related Works
Related work spans explicit 3D geometric memory, latent state-transition learning, and generative video world models. The paper identifies that these approaches update visible regions while retaining out-of-view content as frozen historical observations, motivating continuous out-of-sight state evolution.
- World modeling paradigms: World modeling approaches include explicit geometrically consistent 3D representations, latent-space state transitions, and scalable generative video models.Generative video world models predict future frames from historical context and control signals.
- World-state representation: Figure 2 approximates the intractable 4D world state with a temporally invariant static 3D environment and 2D video sequences for dynamic entities.The two representations are obtained through T-axis and Z-axis projections, respectively.
- Spatial memory: Explicit 3D spatial-memory methods preserve camera control and long-term geometry by injecting reconstructed point clouds and camera parameters into video generation.These methods provide structural global memory for geometric consistency but do not by themselves address temporal evolution outside the current view.
- Out-of-sight dynamics: Existing video world models update states only within the camera’s visible region, while out-of-view content remains frozen at its last observed timestamp.The paper frames this as a missing temporal progression problem within the video world model paradigm.
3 Methods
LiveWorld separates persistent world evolution from camera-conditioned rendering by maintaining static background geometry and dynamically evolving active entities. Its monitor-driven pipeline updates these components across generation rounds and projects them into coherent video observations.
- Problem formulation: LiveWorld formulates world modeling as separate evolution and rendering processes, rather than a single camera-conditioned video generator.The evolution engine updates the world state, while the renderer maps that state to observations under camera and text conditions.
- Structured world-state approximation: The maintained world state is approximated by a temporally invariant static background and a time-varying dynamic-entity representation.The static component is accumulated as a 3D point cloud, while dynamic entities retain evolving state over time.
- Dynamic evolution: Historical frames containing active entities are processed by the evolution engine to simulate their continuous temporal progression, producing the dynamic state M_dyn,t.This explicit evolution mechanism updates unobserved active regions instead of leaving them at their last observed timestamp.
- Unified state-conditioned video backbone: The unified state-conditioned video backbone synthesizes T-frame chunks from explicit state projections, appearance references, and text prompts.A state adapter injects projected world-state guidance into the video diffusion backbone, while reference frames provide temporal and appearance information.
- World-state maintenance and rendering: The system detects active entities in preceding frames and checks whether they are already registered before maintaining or creating monitor-based state.This detection and registration process supports localized dynamic evolution across repeated generation rounds.
- World-state maintenance and rendering: Each generation round accumulates newly observed static regions and autonomously evolves registered active entities before rendering the updated world state.Static regions are fused through Stream3R, while monitors simulate out-of-sight dynamics and project the resulting representations onto the target camera trajectory.
4 Experiments
Experiments evaluate LiveWorld on LiveBench using controlled revisits, reference-based spatial metrics, and VLM-driven event metrics. Results show stronger long-horizon background maintenance, dynamic-entity preservation, event progression, novel-view revisiting, and multi-event persistence than comparison methods.
- Experimental Setup: LiveBench contains 100 scenes and 400 evaluation sequences pairing diverse images with multi-round camera trajectories and text-driven event scripts.Trajectories alternate between leaving and revisiting viewpoints through same-pose and different-pose variants.
- Experimental Setup: The evaluation measures background spatial memory, foreground identity preservation, out-of-sight event progression, and temporal smoothness using image, point-cloud, feature, VideoQA, and CLIP metrics.Background metrics compare same-pose revisits with the initial frame, while dynamic entities are evaluated against monitor predictions in 3D world space.
- Main Results: LiveWorld maintains static backgrounds, dynamic entities, and text-scripted event progression more reliably than baselines, whose backgrounds and foreground consistency degrade during long-horizon revisits.Explicit 3D point clouds support background maintenance, while decoupled evolution and state projection support dynamic-entity preservation and event progression.
- Main Results: LiveWorld preserves long-horizon out-of-sight dynamics over 260 frames during repeated rightward and backward camera motion, while other methods fail.Different colors represent different evolving event prompts.
- Main Results: Under different-pose revisits, LiveWorld maintains entity identity and event alignment while baselines degrade further because of artifacts and failed camera control from novel angles.The comparison specifically evaluates revisits from viewpoints different from the initial pose.
- Main Results: LiveWorld achieves 92% primary-event presence, 70% secondary-event presence, 42% and 35% event success, and 26% full success, versus 0% baseline full success.Full success requires both concurrent events to succeed simultaneously; secondary-event triggering remains vulnerable to text-to-video randomness.
- Ablation Studies: Ablations show that removing event evolution loses out-of-sight entities and event completion, while removing spatial memory causes camera-control failure and severe spatial artifacts.Omitting historical references destabilizes the background and causes cascading temporal collapse during the second long-horizon revisit.
5 Conclusion
The conclusion frames out-of-sight dynamics as a freezing problem in video world models and presents LiveWorld as a tractable approach to persistent 4D simulation. Its static 3D background, monitor-based entity evolution, and LiveBench together target continuous world-state consistency.
- Conclusion: LiveWorld formalizes the failure in which unobserved regions freeze at their last seen state and decouples continuous world evolution from view-dependent rendering.The framework factorizes the environment into a static 3D background and monitored active entities that are autonomously fast-forwarded.
- Conclusion: LiveWorld and LiveBench address the gap between static 2D memorization and persistent 4D dynamic simulation.The benchmark is dedicated to evaluating long-horizon out-of-sight dynamics and event permanence.
Supplementary Materials
Supplementary materials show typical LiveBench revisit cases, including both same-pose and different-pose revisits.
- Supplementary Materials: Figure 7 illustrates inference results for same-pose and different-pose revisits in LiveBench.The supplementary material points to additional examples beyond the displayed cases.
A.1 Evaluating the evolution engine.
The evolution engine is evaluated against vanilla Wan2.1-I2V-14B on LiveBench scene images, showing stronger long-horizon performance despite weaker first-round results.
- The unified evolution engine compares with vanilla Wan2.1-I2V-14B on VQA-Acc, CLIPF, DINOfg, and PSNRbg using LiveBench scene images.
- Despite outperforming the finetuned Gevoθ in the first round, Wan2.1-I2V-14B is less capable of maintaining performance over subsequent rounds.
- Historical reference frames alleviate long-horizon error accumulation by providing high-quality reference appearances.
- The method maintains a static background more reliably because its design avoids the pixel-position instability caused by Wan’s difficulty maintaining a static camera.
A.2 More visualization results for different revisit cases.
Additional LiveBench visualizations cover same-pose and different-pose revisits, comparing first- and fourth-round generations to assess long-horizon robustness.
- Figure 7 illustrates inference results for both same-pose and different-pose revisits in LiveBench.
- Round 1 and round 4 generations are compared to evaluate script alignment, temporal coherence, identity, and background stability.
- The blue columns identify the fourth-round performance in the comparison.
B Training Dataset Construction
The training dataset combines curated video clips with detected entities, estimated scene geometry, decomposed state projections, and diverse conditioning references.
- Approximately 40k 130-frame clips are curated from MiraData, SpatialVID, and RealEstate10K at 16 FPS and 832×480 resolution.
- Dynamic entities are detected with Qwen3-VL-8B-Instruct, then segmented and propagated bidirectionally across frames using SAM3.
- Stream3R estimates per-frame metric depth, camera intrinsics, and camera-to-world poses for denser point-cloud construction.
- Ground-truth projections decompose the world state into temporally invariant static geometry and per-frame dynamic foreground appearance.
- Separate static and dynamic projection channels provide a 32-channel latent input, but strict separation strengthens background control while weakening foreground adherence.
- Each sample contains 65 target frames, preceding frames, geometrically diverse scene references, and cropped foreground instance references.
C.1 Benchmark Construction
LiveBench constructs inference-ready revisit scenarios from source images by combining scene reconstruction, dynamic-entity storylines, and calibrated camera trajectories.
- LiveBench uses a two-stage pipeline: trajectory-template generation followed by assembly of inference-ready configurations.
- Scene reconstruction estimates metric depth, camera intrinsics, and camera-to-world pose, then forms a point cloud with 0.01 m voxel downsampling.
- Detected dynamic entities receive storylines of up to Smax=4 explicit spatial-displacement steps validated against visible scene elements.
- Each trajectory family spans 4 rounds of 65 frames, totaling 260 frames, with left and right variants.
- Same-pose revisits follow A→B→A→B→A, while different-pose revisits follow A→B→C with novel viewpoints at destinations.
- Screen-uniform calibration targets a consistent on-screen horizontal shift rather than a fixed metric camera displacement.
- For selected lateral-translation cases, translation amplitude is solved so foreground objects exit the camera view at approximately 75% of the specified rounds.
C.2 Benchmark Example
LiveBench uses scripted camera revisits and entity evolutions to test whether dynamic events persist outside the observer’s view. Its setup distinguishes static-scene coherence from out-of-sight foreground progression and positions the benchmark against related evaluations.
- Benchmark instance: A warehouse instance depicts a person whose scripted behavior changes across rounds, including walking toward a crate, moving to the far wall, and leaning near the doorway.These scripts provide concrete temporal events to evaluate during revisits.
- Benchmark instance: LiveBench includes a Same-Pose Revisit trajectory that repeatedly follows A→B→A→B→A.The instance provides per-round camera poses, depth, and intrinsics for constructing the trajectory.
- Benchmark instance: The benchmark separates observer rendering from out-of-sight evolution, using scene descriptions for static backgrounds and scripts for the person’s motion.The static script describes warehouse layout and viewpoint-dependent changes, while the evolution script specifies actions across rounds.
- Related benchmarks: WorldScore and WorldModelBench emphasize visual quality, physical-law adherence, or camera trajectories, whereas MIND targets memory consistency and action control.The cited comparison motivates LiveBench’s focus on maintaining dynamics that occur while entities are unobserved.
- Related work: LiveWorld differs from FloWM by using virtual monitors and state-conditioned video diffusion to simulate semantic, non-rigid, and discrete event progressions.FloWM instead extrapolates latent trajectories under Lie-group flow equivariance, particularly for continuous rigid motion.
- Related work: LiveWorld targets open-world high-fidelity synthesis by combining temporally invariant 3D SLAM backgrounds with autonomously evolving 4D foreground entities.The paper associates this design with complex appearance changes and persistent visual textures during long-horizon exploration.
D.3 Limitations and insights on future works
The paper identifies practical limitations in LiveWorld’s modular inference pipeline, dynamic representations, monitor allocation, and treatment of interactions. Its supplementary prompts separate static-scene descriptions from foreground evolution and support entity summoning and monitor-based continuation.
- Limitations: LiveWorld relies on off-the-shelf language, segmentation, and visual-SLAM models, introducing latency compared with end-to-end video generation.The paper suggests replacing explicit modules with end-to-end models using implicit dynamic memories.
- Limitations: Dynamic foreground rendering may produce incomplete point clouds when entities are revisited from views different from the monitor’s anchor pose.The authors suggest monocular video-to-4D-mesh generation for denser, more view-consistent representations.
- Limitations: Inference requires balancing the number of monitors against generation latency, which affects the precision of the 4D-world approximation.The paper identifies this balance as a limitation for long generation latency.
- Limitations: Interactions between dynamic events in different monitors are not yet designed in the current minimal implementation.This is presented as a future direction for extending the decoupled formulation.
- Inference prompts: The inference pipeline uses separate scene and foreground prompts, excluding dynamic entities from static-environment descriptions while optionally adding foreground actions.The renderer composes scene_text with fg_text when foreground text is non-empty.
- Inference prompts: An Automatic Evolver generates physically plausible continuation descriptions from monitor anchor frames, and detected or summoned entities can receive new monitors.The renderer can introduce a new foreground entity through a user or evolver-provided prompt before detector-based monitor registration.