PaperlayerView exact revision on arXiv

Source-linked AI summary

Momentum: Persistent Dynamics for Video World Models

Nedko Savov, Stefan Maria Ailuro, Danda Pani Paudel, Yilun Du, Xi Wang, Luc Van Gool

arXiv:2610.23423v1cs.CV

TL;DR

Video world models need to represent how scenes evolve beyond the current camera view, but existing observation histories and static spatial memories do not provide temporally updated grounding. Momentum generates synchronized reference-camera videos as an evolving world state and uses them to ground target-camera generation. In generated street scenes, it improves dynamic scene consistency by 126.2% in an unseen town over static-image conditioning.

  • Problem

    Observation histories and stored spatial memories provide past evidence but do not advance scene content through time, leaving video world models without temporally updated grounding beyond the current view.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    Momentum’s WorldUpdate generates synchronized reference-camera videos as an evolving world state, and WorldInteract conditions target-camera video generation on that state, observation history, and camera trajectory.

    Source: §3.4 · p. 6, §3.4.1 · p. 6

  • Results

    126.2%: Momentum improves SCS-dyn in an unseen town over the Dynamically Grounded WM conditioned on static image references.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Synchronized reference-camera videos represent scene appearance and dynamics beyond the target view without explicit entity tracking or a static-dynamic decomposition.

    Source: §5 · p. 9

Abstract

from arXiv · show

Video world models must capture how a scene evolves beyond the current camera view to generate consistent observations when its content reappears. Existing approaches rely on observation history, static point clouds, or image caches constructed from past frames. These representations are often expensive and, more importantly, cannot update dynamic scene content outside the field of view: generated vehicles and pedestrians may be forgotten after leaving the field of view or reappear in physically inaccurate states. We represent an evolving scene through synchronized videos from reference-camera rigs, each comprising eight cameras that jointly cover the surrounding scene. We connect these rigs in a graph and propagate video generation between neighboring rigs to coordinate scene dynamics across locations and time. We introduce Momentum, a two-stage pipeline: WorldUpdate jointly generates reference-camera videos as a shared memory of scene dynamics, and WorldInteract grounds camera-controlled video generation in this evolving memory, allowing moving content to remain represented outside the target view and reappear consistently. To support training and evaluation, we develop a CARLA pipeline that captures synchronized videos from reference camera rigs and a moving target camera, together with an evaluation framework measuring consistency within the generated world state and the target observations.Experiments on generated street scenes show that grounding target videos in scene dynamics generated by Momentum improves dynamic scene consistency by 126.2% in unseen scenes over static image conditioning, while also improving visual fidelity. Our models and source code will be made publicly available.

1 INTRODUCTION

Video world models need a representation that tracks both scene structure and changing dynamics beyond the current view. Momentum uses synchronized reference-camera videos to evolve that representation and ground target-camera generation in it.

1 INTRODUCTION

  • Stored observations and spatial memories retain past evidence but do not advance scene content through time, leaving the challenge of representing both what exists and how it changes.Observation history is also limited by finite context.Source: §1 · pp. 1–2
  • The proposed representation coordinates scene evolution across reference cameras without requiring explicit object trajectories, identity tracking, or a static-dynamic scene decomposition.It extends image-based spatial grounding to synchronized videos that account for appearance and motion at the target location and time.Source: §1 · pp. 2–3
  • Momentum’s WorldUpdate propagates synchronized multi-view video generation across a graph of eight-camera rigs, and WorldInteract uses the resulting rollout to ground target-camera video.Each rig is conditioned on geometrically reprojected video from an already generated neighbor.Source: §1 · pp. 2–3
  • 130.2% in seen towns and 126.2% in an unseen town: Momentum improves dynamic scene consistency over the Dynamically Grounded WM conditioned on static images.The added reference-video generation stage trades computation for a shared representation of dynamics beyond the current view; its cost can be amortized across target-camera trajectories.Source: §1 · p. 3
  • The paper develops a CARLA pipeline for synchronized reference and moving-target observations, plus an evaluation framework for consistency in the generated world state and target videos.It also develops an image-grounded world model based on SWM.Source: §1 · p. 3

2 RELATED WORK

Video world models support controllable visual environments across games, driving, robotics, and exploration. Prior work extends generation through temporal or spatial memory and models of dynamics, while Momentum focuses on persistent scene evolution across reference-camera regions.

2.1 VIDEO WORLD MODELS

  • Video world models support controllable exploration, game-like simulation, driving observations, and robot learning and control across varied settings.The paper builds on Cosmos Predict 2.5 and focuses on persistent generated dynamics across target-camera trajectories.Source: §2.1 · p. 3

2.2 MEMORY AND DYNAMICS IN VIDEO WORLD MODELS

  • Memory approaches retain temporal or spatial evidence, but retention alone does not advance the represented scene through time.Examples include recurrent or compressed frame history, retrieved observations, 3D caches, surfel indexing, and external street-view images.Source: §2.2 · pp. 3–4
  • Dynamic geometric methods fit scene dynamics to recorded videos, while other methods generate environments under supplied object-motion controls.Momentum instead reuses a dynamics model across scenes and spatial regions without scene-specific fitting or prescribed object trajectories.Source: §2.2 · p. 4
  • Momentum organizes evolving scene state by spatial regions, using coordinated reference-camera videos to represent dynamics and ground target observations without explicit entity tracking or static-dynamic decomposition.This contrasts with approaches that retrieve hidden-subject cues, plan object trajectories, or advance active entities against a static background.Source: §2.2 · p. 4

3 METHODOLOGY

Momentum models grounding as a time-indexed world state, then generates that state through synchronized reference-camera videos and uses it to produce camera-controlled observations. Its methodology specifies the CARLA data pipeline, static-grounding baseline, and two stages for advancing and querying the evolving state.

3.1 PROBLEM STATEMENT

  • The world state is a time-indexed sequence of synchronized videos from spatially distributed eight-camera rigs, which evolves during generation rather than remaining a set of static references.WorldUpdate generates the state sequence, and WorldInteract uses it with observation history and camera trajectory to predict future observations.Source: §3.1 · pp. 4–5

3.2 DATA GENERATION FRAMEWORK

  • The CARLA framework provides static and synchronized dynamic grounding data across six towns, with randomized weather, illumination, vehicles, and pedestrians.Dynamic sessions pair synchronized rig and target-camera videos; evaluation includes unseen intersections in training towns and a held-out town.Source: §3.2 · p. 5

3.3 STATIC GROUNDING

  • The static-grounding baseline retrieves nearby reference images and conditions generation through geometric reprojection and semantic references, with a virtual lookahead sink refreshed at each generation step.The baseline uses a Cosmos Predict 2.5-2B-initialized diffusion transformer and autoregressive chunks conditioned on history and camera trajectory.Source: §3.3 · pp. 5–6

3.4 MOMENTUM

  • Momentum separates world-state generation from target-video generation, so completed world rollouts can ground multiple target-camera trajectories through the same interface.The reference videos represent scene dynamics independently of the target trajectory and can be inspected or generated separately from target videos.Source: §3.4 · p. 6

3.4.1 WORLDUPDATE

  • Multi-Rig DGWM advances synchronized reference-camera videos through joint multi-view generation and geometric conditioning between rigs.It extends a seven-view pretrained model to eight cameras; rigs condition on neighboring synchronized video reprojected using depth and relative poses, with breadth-first graph traversal propagating generation.Source: §3.4.1 · pp. 6–7

3.4.2 WORLDINTERACT

  • Dynamically Grounded WM fine-tunes the static-grounding architecture on synchronized dynamic data and uses time-aligned reference-video frames for geometric conditioning of target-camera generation.Semantic references are sampled at the start of each chunk to provide a common temporal anchor across retrieved viewpoints.Source: §3.4.2 · p. 7

4 EXPERIMENTS

Momentum improves dynamic scene consistency and visual fidelity in end-to-end evaluation, with especially large SCS-dyn gains over static image conditioning. Additional evaluations assess baseline quality, reference-source count, generation horizon, and qualitative scene consistency.

Experimental Setup

  • Models were trained on six CARLA towns and evaluated on Test 1 and Test 2 for visual fidelity, camera-trajectory accuracy, and static and dynamic scene consistency.Unless otherwise stated, evaluation uses four-rig sets and two views from the closest rig for reprojection.Source: §4 · p. 8

Baseline Validation

  • On Ann Arbor MARS data, the synthetic-data-trained Statically Grounded WM exceeds reported SWM results on mPSNR and has lower mLPIPS and rotation error, but higher translation error.The comparison follows SWM’s evaluation protocol and uses masked PSNR and LPIPS.Source: §4 · p. 8

WorldInteract Consistency Evaluation

  • Across seen and unseen towns and scenes with one or four rigs, Dynamically Grounded WM achieves the best visual fidelity, trajectory accuracy, and static and dynamic consistency among the tested WorldInteract choices.Giving synchronized video to the statically trained model alone does not yield comparable gains.Source: §4 · p. 8

WorldInteract Reprojection Sources

  • With three source rigs, Dynamically Grounded WM improves all metrics in the unseen town despite training with a single source rig.In seen towns, one source is better for visual and semantic consistency, while three slightly improve camera-trajectory accuracy.Source: §4 · p. 9

WorldInteract Performance Over Generation Horizon

  • Dynamically Grounded WM maintains visual fidelity more consistently over the generation horizon than models using static grounding.The comparison uses per-frame PSNR on both test sets.Source: §4 · p. 9

End-to-End Evaluation

  • SCS-dyn improves over image-conditioned Dynamically Grounded WM by 130.2% in seen towns and 126.2% in the unseen town, while Momentum also improves visual fidelity and static and dynamic consistency.Momentum achieves the best camera-trajectory accuracy in the unseen town; in seen towns, it matches rotation accuracy but increases translation error relative to that baseline.Source: §4 · p. 9

Qualitative Results

  • Qualitative examples show better moving-object preservation by dynamic grounding and more consistent cross-view motion predictions by the multi-rig model.Figure 4 compares SGWM with DGWM for WorldInteract and DGWM with Multi-Rig DGWM for WorldUpdate.Source: §4 · p. 9, Figure 4 · p. 9

5 CONCLUSION

Momentum grounds camera-controlled video generation in an evolving world state represented by synchronized reference-camera videos. Its experiments report improved scene consistency, including large dynamic-consistency gains over static image references.

5 CONCLUSION

  • Momentum is a two-stage pipeline that grounds camera-controlled video generation in an evolving world state.WorldUpdate advances the state through graph-propagated generation, and WorldInteract uses it to ground target-camera observations.Source: §5 · pp. 9–10
  • Synchronized reference-camera videos represent scene appearance and dynamics beyond the target view without explicit entity tracking or static–dynamic decomposition.The work also develops CARLA data generation, an SWM-based statically grounded model, and an evaluation framework.Source: §5 · pp. 9–10
  • Compared with Dynamically Grounded WM using static image references, Momentum improves SCS-dyn by 130.2% in seen towns and 126.2% in the unseen town.Experiments in seen and unseen towns also demonstrate improved visual fidelity, controllability, and scene consistency through dynamic grounding.Source: §5 · p. 10

6 AI USE STATEMENT

The authors report using generative AI for feedback, implementation, figures, code, and paper drafting, while reviewing AI-assisted work and accepting responsibility for the final content.

6 AI USE STATEMENT

  • Generative AI tools provided feedback on methodology or experiments and helped implement methods, but did not design the research methodology or system ideas.The statement also excludes AI use for generating synthetic datasets, developing theoretical models or conceptual frameworks, analyzing qualitative data, interpreting results, or refining hypotheses.Source: §6 · p. 10
  • AI tools were used to create or modify scientific figures or images, create software code, draft parts of the paper, and improve readability.Source: §6 · p. 10
  • The authors reviewed AI-assisted work, including code and text, and take responsibility for the final content and artifacts.Source: §6 · p. 10

A.1 DATA GENERATION IMPLEMENTATION DETAILS

The CARLA data pipeline captures synchronized reference-rig and moving-target-camera observations. Its sessions specify camera and trajectory settings, scene layouts, and synchronized rollout timing.

A.1 DATA GENERATION IMPLEMENTATION DETAILS

  • Observations use 1280 × 720 pinhole cameras with a 90° field of view and store RGB video, per-frame camera parameters, GNSS measurements, and lossless depth.The declarative collection configuration supports resuming interrupted sessions and distribution across CARLA servers.Source: §A.1 · p. 13
  • Static-grounding rigs use eight co-located cameras spaced at 45° yaw intervals, while target trajectories include autopilot vehicles, AI walkers, and flyovers.The pipeline renders 20 sessions per town and trajectory type; flyovers use collision-checked random walks at 2–8 m altitude.Source: §A.1 · p. 13
  • Dynamic-grounding sessions center on intersections with at least three approaches, and the multi-rig setting places up to four rigs within 40 m, at least 12 m apart.The single-rig setting retains the central rig.Source: §A.1 · p. 13
  • Sessions spawn up to 10 vehicles and 25 pedestrians, while the target camera follows a smoothed, collision-checked path that visits rigs in randomized order.The path stays within 30 m of the nearest rig in multi-rig settings or 20 m in single-rig settings.Source: §A.1 · p. 14
  • Each session is a 300-frame synchronous CARLA rollout at 10 fps, with reference RGB and depth streams aligned to the target-camera timeline.Reference rigs are active only within 25 m of the target camera; otherwise the most recent frame is repeated, and test sessions use disjoint random seeds.Source: §A.1 · p. 14

A.2 STATICALLY GROUNDED WORLD MODEL IMPLEMENTATION DETAILS

The implementation adapts a video-generation backbone for reference-conditioned target generation, then specifies chunking, training, and rollout settings.

A.2 STATICALLY GROUNDED WORLD MODEL IMPLEMENTATION DETAILS

  • The 2B Cosmos-Predict2.5 DiT adds widened input embeddings for warped video, separate reference and sink embedders, and temporal-position handling for references and the lookahead sink.The model otherwise inherits the backbone’s remaining blocks unchanged.Source: §A.2 · p. 14
  • Each chunk generates 20 latents from 77 pixel frames, alongside five history latents, five references, and a sink five latent frames ahead.The reference gap is 50 latent positions, with one latent position between references.Source: §A.2 · p. 14
  • Training fine-tunes the public checkpoint for 10K iterations at 704 × 1280 resolution on eight H100 GPUs, with noise on history and independent dropout of reference and warped-video pathways.History noise has standard deviation 0.1 with probability 0.5; each pathway is dropped with probability 0.2.Source: §A.2 · p. 14
  • Inference rolls out chunks sequentially using a 35-step rectified-flow ODE sampler and classifier-free guidance of 3.0, carrying forward only history latents.Camera trajectory, retrieved references, and sink are re-derived for each chunk.Source: §A.2 · p. 14

A.3 MOMENTUM IMPLEMENTATION DETAILS

Momentum’s implementation generates coordinated reference-rig videos and dynamically grounded target videos, with camera trajectory accuracy measured using chunk-relative pose errors.

A.3 MOMENTUM IMPLEMENTATION DETAILS

  • Multi-Rig DGWM jointly generates all eight rig views at 192 × 352 resolution, with eight latent frames per view and one history latent.The initial chunk uses one ground-truth frame per view of each rig.Source: §A.3 · p. 14
  • The rig model is fine-tuned for 10K iterations, dropping warped-latent conditioning with probability 0.2 to train generation without reprojection and provide a classifier-free-guidance branch.The dropout rate approximates how often a rig is the graph source at inference.Source: §A.3 · p. 14
  • WorldInteract fine-tunes the statically grounded checkpoint on dynamic-grounding sessions and generates 704 × 1280 target videos in 77-frame chunks with five history latents.It uses the same 35-step sampler and guidance of 3.0 as the statically grounded model; the sink is read five latent frames beyond the target end.Source: §A.3 · p. 15
  • Reference rigs render within 25 m, defining WorldUpdate graph edges; WorldInteract defaults to reprojecting the nearest in-range rig, while out-of-range rigs hold their last frame and are excluded as sources.Source: §A.3 · p. 15

B.1 CAMERA TRAJECTORY ACCURACY

  • Camera trajectory errors compare generated and ground-truth camera-to-world poses in non-overlapping 32-frame chunks, each scored in its own coordinate frame.RotErr is mean geodesic distance on SO(3); TransErr is mean Euclidean distance between separately normalized translations, with anchor frames excluded.Source: §B.1 · p. 15

B.2 CONSISTENCY METRICS

The consistency metrics compare tracked semantic masks across generated and ground-truth videos, separating persistent scene structure from moving entities.

B.2 CONSISTENCY METRICS

  • Structural Consistency Score measures scene-layout consistency as mean mask IoU between corresponding regions in generated and ground-truth videos.Source: §B.2 · p. 15
  • SAM 3.1 tracks instances from a fixed semantic vocabulary, and category masks union all instances in each frame; scoring excludes categories empty in both videos and subsamples to 2 fps.Source: §B.2 · pp. 15–16
  • SCS-static covers buildings, roads, sidewalks, vegetation, and sky; SCS-dynamic covers vehicles, pedestrians, people, bicycles, and animals.Source: §B.2 · p. 16

B.3 VIDEO FIDELITY METRICS

Video fidelity is evaluated on static scene regions, while measures of change compare tracked dynamic content across consecutive frames and against ground truth.

B.3 VIDEO FIDELITY METRICS

  • mPSNR measures reconstruction fidelity across static pixels and color channels, using peak value 1.A pixel is considered static only when it is static in both generated and ground-truth videos.Source: §B.3 · p. 16
  • For mLPIPS, dynamic regions of the generated frame are filled with ground truth because feature-space LPIPS cannot be restricted to an arbitrary pixel set.Source: §B.3 · p. 16

B.4 MEASURES OF CHANGE

  • Staticness is mean IoU between tracked dynamic-category masks in consecutive frames of one video; its score falls as more agent area moves, appears, or disappears.Videos are subsampled to 2 fps, and category pairs empty in both frames are excluded.Source: §B.4 · p. 16
  • The evaluation also reports the mean per-clip gap between generated and ground-truth staticness, over clips where both scores are defined.This supplements average staticness because matching averages can conceal frozen dynamic clips and animated static ones.Source: §B.4 · p. 16

C ADDITIONAL EXPERIMENTS

The WorldUpdate consistency experiment compares multi-view generation across rigs with dynamically grounded WorldInteract, revealing a trade-off between dynamic accuracy and metrics that favor static content.

C.1 WORLDUPDATE CONSISTENCY

  • The frozen-frame baseline achieves the best PSNR and LPIPS, while WorldUpdate has the lowest staticness gap in every configuration.The results expose a trade-off between metrics that reward freezing and dynamic accuracy.Source: §C.1 · p. 17, Table 5 · p. 17
  • Because rig cameras are stationary, most image content is static, so PSNR, LPIPS, and SCS-dyn favor freezing rather than motion matching.SCS-dyn rewards overlap over one evaluation step, even when a moving object’s predicted motion differs from the recorded future.Source: §C.1 · p. 17, Table 5 · p. 17
  • WorldUpdate moves the scene by roughly the ground-truth amount and preserves static structure best among generative models on SCS-st.It exceeds the frozen baseline on SCS-st in seen towns.Source: §C.1 · p. 17, Table 5 · p. 17
  • WorldInteract remains close to the frozen baseline in staticness and inherits its PSNR, LPIPS, and SCS-dyn results.The comparison uses ground-truth neighbor-view reprojections for model conditioning.Source: §C.1 · pp. 16–17, Table 5 · p. 17

Related papers

Loading 2610.23423v1…