PaperlayerView exact revision on arXiv

Source-linked AI summary

Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution

Feiran Wang, Bin Duan, Junyi Wu, Gaowen Liu, Yan Yan

arXiv:2610.04152v1cs.CV

TL;DR

Video world models seek to preserve scene structure while predicting dynamic objects, but existing approaches may leave future motion to synthesis or require external trajectories. Kepler4D represents scene state and motion explicitly, then uses structured decisions and deterministic rollout to create editable trajectory controls for video generation. Experiments report controllable object motion and plausible future rollout while preserving scene consistency.

  • Problem

    Existing scene-consistency and geometric-control approaches typically leave future motion to video synthesis or require external trajectories, motivating a shared representation linking observed motion and scene context to explicit future trajectories.

    Source: §1 · p. 2

  • Method

    Kepler4D builds a shared 4D scene state and uses Chain-of-Motion decisions with deterministic rollout to produce editable future object trajectories that guide a pretrained video generator.

    Source: §1 · p. 2

  • Results

    Experiments on real-world videos demonstrate controllable object motion and plausible future rollout while preserving scene consistency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The explicit trajectories expose future object motion for inspection and editing before video synthesis.

    Source: §1 · p. 2

    Generated videos are possible continuations, not verified future events; safety-critical or real-world decision-making applications require additional validation and appropriate safeguards.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Video world models aim to preserve scene structure and predict how dynamic objects evolve beyond visual observations. We present Kepler4D, a framework for future video generation through explicit 4D scene state evolution. Given a monocular video, Kepler4D constructs a shared 3D representation of background geometry, object motion histories, coarse spatial supports, and semantic context. Chain-of-Motion summarizes observed motion and uses a vision-language model to select structured speed and heading decisions and decide whether to bound object-center height from below. A deterministic rollout converts these decisions into future object trajectories for inspection and editing before synthesis. We render the evolving proxies into geometric controls for a pretrained video generator, separating coarse object motion from the synthesis of appearance and articulation. Experiments on real-world videos demonstrate that Kepler4D enables controllable object motion and plausible future rollout while preserving scene consistency.

1 INTRODUCTION

Future video world models seek to preserve scene structure while predicting dynamic objects, but existing consistency and geometric-control approaches often defer motion prediction to synthesis or require external trajectories. Kepler4D links observed motion and scene context to explicit, editable future trajectories in a shared 4D state, then evaluates controllable generation and scene consistency on real-world videos.

1 INTRODUCTION

  • Existing scene-consistency and geometric-control approaches typically leave future motion to video synthesis or require external trajectories.The introduction motivates a shared representation connecting observed motion and scene context to explicit trajectories and rendering controls.Source: §1 · p. 2
  • Kepler4D represents background geometry, object motion histories, coarse spatial supports, and semantic context in a shared 4D state for explicit, editable future motion.Static geometry anchors scene structure; predicted trajectories can be inspected or edited before synthesis.Source: §1 · p. 2
  • Experiments on real-world videos assess automatic evolution, language-guided trajectory editing, and static rendering, reporting controllable object motion, plausible rollout, and scene consistency.Ablations examine motion history, semantic context, and rendered trajectory guidance.Source: §1 · p. 2
  • Chain-of-Motion uses observed motion and scene context to infer structured decisions, which deterministic rollout converts into future 3D trajectories for video-generation guidance.The system renders predicted or edited trajectories with background geometry as controls; the pretrained generator synthesizes appearance and local articulation.Source: §1 · p. 2

2 RELATED WORK

Related work covers video world models for scene evolution, geometric control for consistent views, and methods for predicting motion. Kepler4D combines structured motion reasoning with deterministic 3D rollout and renders editable trajectories as controls for future video generation.

Video World Models and Geometric Control

  • Video world models and spatial-memory methods simulate scene evolution or preserve scene structure, while some approaches model dynamic events beyond the current view.The passage surveys spatial memory, hybrid memory, viewpoint control, and joint camera/object control methods.Source: §2 · p. 2
  • Kepler4D renders editable 3D trajectories predicted from observed motion and scene context as geometric controls for future video generation.Its overview depicts scene-state construction, Chain-of-Motion decisions, deterministic rollout, and control rendering before synthesis.Source: §2 · p. 3, Figure 2 · p. 3

Motion Reasoning and Future Prediction

  • Motion-prediction work uses observations, histories, or language; Kepler4D combines structured VLM decisions with deterministic 3D rollout for scene-aware, editable object-center trajectories.The trajectories provide direct object-motion controls before future video synthesis.Source: §2 · p. 3

3 METHODOLOGY

Kepler4D reconstructs a shared 3D proxy scene from an observed video, predicts future object motion, and renders the evolving scene as controls for video synthesis. Its pipeline separates explicit object trajectories from the generator’s synthesis of fine appearance and articulation.

3 METHODOLOGY

  • The pipeline constructs a shared 3D representation, predicts object trajectories with Chain-of-Motion, then renders predicted or edited trajectories with static geometry as controls for future synthesis.The controls guide a pretrained video generator.Source: §3 · p. 3

3.1 4D SCENE STATE AND ITS EVOLUTION

  • The 4D state combines static background geometry, foreground proxies, and semantic context; each proxy stores a label, observed 3D center history, and coarse spatial extent.The center history informs motion prediction, while the extent defines an ellipsoid for rendering; scene descriptions and layout cues support motion prediction.Source: §3.1 · pp. 3–4

3.2 SCENE STATE CONSTRUCTION

  • Scene construction estimates depth and camera geometry, identifies and tracks foreground objects, reconstructs their centers and extents, and aggregates background points into static geometry.The resulting scene state supplies motion histories and semantic context for prediction, plus geometry for rendering.Source: §3.2 · p. 4

3.3 CHAIN-OF-MOTION INFERENCE

  • Chain-of-Motion extracts motion attributes and uses a pretrained VLM to infer speed, direction, and optional lower-height-bound decisions from motion state and scene context.Deterministic rollout applies fixed updates from the latest position and velocity, enforces the height bound when enabled, and smooths successive positions; decisions can be edited to specify alternative motion.Source: §3.3 · pp. 4–5

3.4 FUTURE VIDEO GENERATION

  • Future-video controls render static background and coarse Gaussian ellipsoid proxies under target camera poses, encoding scene layout and object displacement for a pretrained generator.The last observed frame provides an appearance reference, and motion descriptions provide text prompts; the generator synthesizes fine appearance details.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments evaluate static view synthesis, automatic continuation, and language-guided motion editing, using DL3DV and DAVIS sequences. Across these settings, Kepler4D preserves scene consistency while producing plausible, controllable future motion.

4 EXPERIMENTS

  • The evaluation covers static view synthesis on 55 DL3DV scenes and dynamic continuation and motion editing on 90 DAVIS sequences, with shared observed frames and target camera paths within settings.Automatic continuation uses 38 selected sequences split into observed prefixes and held-out continuations.Source: §4 · p. 6

4.2 AUTOMATIC DYNAMIC EVOLUTION

  • Kepler4D obtains favorable motion-fidelity scores with competitive reconstruction and generation quality in automatic continuation, while the runner example shows a forward vault over the railing.The comparison reports lower Flow-EPE and RotErr than VerseCrafter; SPMem has higher image-motion and camera-rotation errors, while VerseCrafter has the lowest LPIPS.Source: Table 2 · p. 7, §4.2 · p. 7

4.1 STATIC NOVEL VIEW SYNTHESIS

  • For static novel views, Kepler4D better preserves layout and fine textures in wide-baseline views and performs better on most reconstruction and VBench metrics.GEN3C has the highest Motion Smoothness; Kepler4D uses 62.46 GB peak GPU memory and has the lowest reported per-clip inference time, 5.2 minutes.Source: §4.1 · p. 7, Table 1 · p. 7

4.3 CONTROLLABLE DYNAMIC GENERATION

  • Language-guided control produces command-dependent motion changes and reports the highest PSNR and imaging quality and lowest TurnErr under the stated protocol.TurnErr does not directly measure future object-heading accuracy.Source: §4.3 · p. 8, Table 3 · p. 9
  • Rendered trajectory guidance adds gains beyond CoM motion descriptions alone, while motion descriptions improve FID and imaging quality over scene prompting.The comparison progresses from scene description to motion description and then to motion description plus rendered trajectories.Source: §4.3 · p. 9, Table 4 · p. 9

4.4 ABLATION STUDY

  • Trajectory perturbations increase AngleErr and SurfErr and lower imaging quality relative to unperturbed ground-truth trajectory control; downward perturbation produces the largest reported directional error.AngleErr rises from 5.78 to 11.37 for a 60° left rotation and to 26.92 for downward perturbation; SurfErr rises from 0.040 to 0.048 and 0.056, respectively.Source: §4.4 · p. 9, Table 5 · p. 9
  • Removing motion history causes the largest increase in Flow-EPE and drop in DynDeg, while removing semantic context also increases Flow-EPE.Removing surface cues has a smaller Flow-EPE effect but increases SurfErr and reduces imaging quality.Source: §4.4 · p. 9, Table 6 · p. 9

5 CONCLUSION

Kepler4D represents future evolution as explicit object-centric 3D trajectories in an editable shared 4D scene state. Chain-of-Motion reasoning and deterministic rollout expose motion controls before geometry-conditioned video synthesis.

5 CONCLUSION

  • Kepler4D separates scene structure, foreground motion, and semantic context in a shared, editable representation of future evolution.Future evolution is represented as object-centric 3D trajectories inferred from monocular observations.Source: §5 · p. 9
  • Chain-of-Motion reasoning and deterministic rollout make future dynamics interpretable control signals before rendering.The rollout exposes trajectories for inspection or editing before synthesis.Source: §5 · p. 9
  • Experiments across static rendering, automatic dynamic evolution, and language-guided control demonstrate the utility of explicit motion representation for future video generation.Under target camera paths, the video generator supplies appearance details and local articulation while supporting controllable future object motion.Source: §5 · p. 9

AI USE STATEMENT

Generative AI assistants were used only for manuscript polishing; the authors developed and verified the research and reviewed all AI-assisted edits.

AI USE STATEMENT

  • Generative AI assistants were used only to polish manuscript clarity, grammar, and presentation.Source: AI USE STATEMENT · p. 10
  • The authors developed and verified the research ideas, methods, experiments, and conclusions.Source: AI USE STATEMENT · p. 10
  • The authors reviewed all AI-assisted edits and take responsibility for the final content.Pretrained vision-language models used in the method are described in the methodology and implementation sections.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The experiments use existing public research datasets, and generated videos are possible continuations rather than verified future events. Safety-critical applications require additional validation and safeguards.

ETHICS STATEMENT

  • Experiments use existing public research datasets and involve no new human-subject data collection.Source: ETHICS STATEMENT · p. 10
  • Generated videos represent possible continuations, not verified future events.Source: ETHICS STATEMENT · p. 10
  • Safety-critical or real-world decision-making applications require additional validation and appropriate safeguards.Source: ETHICS STATEMENT · p. 10

TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL

The appendices document Kepler4D’s pretrained components, geometry and motion interfaces, deterministic trajectory rollout, and geometric controls for synthesis. They also specify evaluation settings, metrics, and supplementary analyses.

TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL

A.1 PRETRAINED COMPONENTS AND INFERENCE CONFIGURATION

  • Depth Anything 3, Qwen2.5-VL-7B-Instruct, SAM 2.1, and Wan2.1-T2V-14B provide geometry, perception and decisions, mask propagation, and video synthesis, respectively.Object trajectories and rendered controls connect the components; the generator need not produce an explicit 3D motion plan.Source: §A.1 · p. 15

A.2 GEOMETRY, FOREGROUND IDENTIFICATION, AND SPATIAL SUPPORT

  • Geometry construction relates image observations to object motion using depth, calibration, and camera pose, with scale-aligned chunks for longer observed videos.Foreground identities and propagated masks associate objects across time; coarse extents guide control without representing detailed meshes or articulation.Source: §A.2 · pp. 15–16

A.3 MOTION SUMMARIES AND STRUCTURED DECISIONS

  • Chain-of-Motion separates measured displacement, speed, acceleration, and turning cues from context-dependent speed, heading, and optional height-bound decisions.The height bound constrains the predicted center, not physical contact; validated defaults address invalid or omitted structured actions.Source: §A.3 · pp. 16–17

A.4 SPEED AND HEADING UPDATES IN DETERMINISTIC ROLLOUT

  • Deterministic rollout applies per-step speed multipliers and planar heading updates, then optionally bounds object-center height and smooths the trajectory.Automatic turn magnitudes may use measured turn rate or category-derived defaults; language-guided instructions can specify the turning angle.Source: §A.4 · p. 17

A.5 GEOMETRIC CONTROL REPRESENTATION

  • Geometric controls use fixed-extent Gaussian proxies whose means follow predicted centers, providing spatial guidance without detailed object-shape prediction.The synthesis interface combines background and foreground depth and control streams; the last observed frame supplies appearance reference, while text specifies semantic motion.Source: §A.5 · pp. 17–18, Table A2 · p. 18
  • The synthesis loader’s temporal-length handling is distinct from the future prediction horizon, and evaluation compares generated continuations with held-out source-video portions.Source: §A.5 · p. 18

B DATASET USAGE AND EVALUATION SETTINGS

The evaluation combines static view synthesis, dynamic continuation, and language-guided generation under specified observation and camera conditions. Its metrics capture complementary aspects of similarity, perceptual quality, motion, and camera rotation, but reference similarity does not determine whether an edited future is correct.

B DATASET USAGE AND EVALUATION SETTINGS

  • Static evaluation uses 49 observed frames from each of 55 DL3DV-Evaluation scenes and held-out views, so it assesses reconstruction rather than Chain-of-Motion accuracy.All methods receive the same observed frames and, when supported, the same target camera path.Source: B DATASET USAGE AND EVALUATION SETTINGS · p. 18
  • Automatic continuation uses 38 DAVIS sequences without external future trajectories or motion commands, while language-guided cases provide a shared command to methods with different geometric conditioning.DAVIS camera-continuation evaluations use a shared future camera path estimated from the full sequence; they test continuation under that supplied path, not camera-motion prediction.Source: B DATASET USAGE AND EVALUATION SETTINGS · p. 19

C.2 REFERENCE SIMILARITY AND PERCEPTUAL QUALITY

  • Reference similarity metrics compare generated frames with the recording, but an edited motion may appropriately differ from that single realized continuation.VBench image quality and temporal-consistency measures do not require matching the recorded motion, and Dynamic Degree measures activity rather than accuracy.Source: B DATASET USAGE AND EVALUATION SETTINGS · p. 19, §C.2 · p. 20

C.1 METRICS BY EVALUATION SETTING

  • The metric suite varies by task, combining image similarity and quality with motion activity, flow, rotation, turning, and trajectory-related measurements.Static evaluation uses PSNR, SSIM, LPIPS, FID, and VBench; dynamic continuation adds Dynamic Degree, Flow-EPE, and RotErr, while language control uses PSNR, TurnErr, Imaging Quality, and Dynamic Degree.Source: §C.1 · p. 19

C.3 IMAGE MOTION AND CAMERA MOTION

  • Flow-EPE reflects combined camera, object, and appearance motion across the image, whereas RotErr measures camera rotation rather than foreground-object heading.RotErr averages geodesic rotation errors and reports degrees; it uses camera rotations estimated from generated and reference videos.Source: §C.3 · p. 20

C.4 CONTROL-RELATED MEASUREMENTS

The control-related measurements assess camera turning, object-direction adherence, and object-center height relative to an estimated reference; each measures a distinct quantity.

C.4 CONTROL-RELATED MEASUREMENTS

  • TurnErr compares a commanded turning magnitude with camera yaw estimated from the generated clip, so it diagnoses camera turning rather than object-level compliance.Its absolute peak does not distinguish left from right or a final angle from a transient maximum.Source: §C.4 · pp. 20–21
  • AngleErr compares a supplied control direction with the direction reconstructed from generated, mask-tracked object centers, not prediction of the reference future.The comparison assumes nonzero directions in a common coordinate frame and is reported in degrees.Source: §C.4 · p. 21
  • SurfErr measures generated object-center height deviation from an estimated reference, not contact with a physical supporting surface.Its interpretation depends on reconstruction, tracked centers, and the chosen local coordinate; lower values indicate closer adherence to the estimated height reference.Source: §C.4 · p. 21

D BASELINE SCOPE AND ABLATION INTERPRETATION

The paper distinguishes baseline capabilities from tests of motion conditioning and scene-state components, then illustrates how Chain-of-Motion turns observed evidence into editable trajectory guidance.

D BASELINE SCOPE AND ABLATION INTERPRETATION

  • Baselines provide complementary forms of spatial memory, geometric guidance, camera redirection, or joint camera-and-object control, while Kepler4D infers future object trajectories through Chain-of-Motion.In language control, Wan Inpaint receives the shared text instruction but not Kepler4D’s inferred trajectories or foreground control maps.Source: D BASELINE SCOPE AND ABLATION… · p. 21

D.1 MOTION DESCRIPTION AND TRAJECTORY GUIDANCE

  • The motion-conditioning study contrasts scene descriptions, future-motion descriptions, and spatial trajectory guidance, assessing pipeline gains rather than standalone VLM forecasting accuracy.Table A4 summarizes the text and trajectory inputs supplied by the three variants.Source: §D.1 · pp. 21–22, Table A4 · p. 22

D.2 TRAJECTORY PERTURBATIONS AND SCENE-STATE COMPONENTS

  • Trajectory perturbations test synthesis response to changed geometric guidance, while scene-state ablations examine motion history, semantic context, and surface-related cues.Changes in Dynamic Degree indicate motion activity changes, not necessarily improved trajectory accuracy.Source: §D.2 · p. 22

E VISUALIZATION OF CHAIN-OF-MOTION TRAJECTORY PREDICTION

  • The Chain-of-Motion schematic links background geometry, object identities, observed centers and extents, and semantic context to structured decisions for deterministic future center rollout.The optional surface-adherence field enables a lower bound on object-center height rather than enforcing contact with a reconstructed road surface.Source: E VISUALIZATION OF CHAIN-OF-MOTION… · p. 22

E.1 PERCEPTION AND MOTION EVIDENCE

  • Perception identifies and tracks foreground objects across endpoint frames, while motion evidence summarizes measured movement and scene context for the Chain-of-Motion query.The schematic inputs and outputs illustrate an interface, not numerical benchmark logs.Source: §E.1 · p. 22, Figure 6 · p. 23, Table A5 · p. 23

E.2 EXAMPLE OF A MOTION DECISION

  • The motion query selects speed and heading actions, an optional center-height lower bound, and a text description; rollout converts those choices into geometric foreground guidance.The trajectory controls coarse object support, while the description conditions synthesis of behavior; appearance and local articulation are generated downstream.Source: §E.2 · pp. 23–24

F LIMITATIONS AND FUTURE WORK

Kepler4D’s motion control is intentionally coarse and relies on estimated monocular scene state, leaving physical detail and uncertainty as limitations.

F LIMITATIONS AND FUTURE WORK

  • Fixed coarse supports do not explicitly model joint motion, shape deformation, or contact dynamics, so geometric guidance alone does not guarantee physically correct interactions.Articulated supports and contact constraints are proposed for applications requiring finer physical control.Source: F LIMITATIONS AND FUTURE WORK · p. 24
  • Monocular reconstruction estimates can shift object centers under occlusion or changing visible surface area, and the current rollout does not propagate calibrated uncertainty.Confidence-aware tracking and joint refinement of geometry and motion are proposed as robustness directions; their downstream effects remain unevaluated.Source: F LIMITATIONS AND FUTURE WORK · p. 24

Related papers

Loading 2610.04152v1…