PaperlayerView exact revision on arXiv

Source-linked AI summary

VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control

Sixiao Zheng, Minghao Yin, Wenbo Hu, Xiaoyu Li, Ying Shan, Yanwei Fu

arXiv:2601.05138v2cs.CV

TL;DR

Video world models lack unified, precise control over camera and multi-object motion because videos encode dynamics in projected 2D. VerseCrafter addresses this with shared-frame 4D Geometric Control, GeoAdapter conditioning, and the VerseControl4D dataset. Experiments report superior visual quality and more accurate joint motion control than prior methods.

  • Problem

    Video-based world models capture dynamics in projected 2D, while existing controls often lack unified, precise 3D-aware control over camera and multi-object motion.

  • Method

    VerseCrafter represents scenes with a static background point cloud and per-object 3D Gaussian trajectories, renders them into 4D control maps, and conditions frozen Wan2.1 through GeoAdapter.

  • Results

    VerseCrafter achieves superior visual quality and more accurate joint camera and object motion than existing controllable video-generation methods.

  • Takeaways & Limitations

    A shared 4D geometric state provides a unified interface for disentangled camera and multi-object motion control in dynamic video generation.

  • Takeaways & Limitations

    A single 3D Gaussian per object provides only ellipsoid-level control, limiting fine-grained pose and part-level articulation.

Abstract

from arXiv · show

Video world models aim to simulate dynamic, real-world environments, yet existing methods struggle to provide unified and precise control over camera and multi-object motion, as videos inherently capture dynamics in the projected 2D image plane. To bridge this gap, we introduce VerseCrafter, a geometry-driven video world model that generates dynamic, realistic videos from a unified 4D geometric world state. Our approach is centered on a novel 4D Geometric Control representation, which encodes the world state as a static background point cloud and per-object 3D Gaussian trajectories. This representation captures each object's motion path and probabilistic 3D occupancy over time, providing a flexible, category-agnostic alternative to rigid bounding boxes and parametric models. We render 4D Geometric Control into 4D control maps for a pretrained video diffusion model, enabling high-fidelity, view-consistent video generation that faithfully follows the specified dynamics. To enable training at scale, we develop an automatic data engine and construct VerseControl4D, a real-world dataset of 35K training samples with automatically derived prompts and rendered 4D control maps. Extensive experiments show that VerseCrafter achieves superior visual quality and more accurate control over camera and multi-object motion than prior methods.

1. Introduction

VerseCrafter addresses the mismatch between 2D video dynamics and unified 4D control by representing camera and multi-object motion in a shared geometric world state. Its geometry-driven model, adapter conditioning, and VerseControl4D dataset support realistic, view-consistent generation.

  • Video world models predict future visual states, but videos capture dynamics in projected 2D rather than the full 4D spatiotemporal space.
  • Existing camera- and object-control methods often lack multi-object control or 3D awareness, limiting robustness under large viewpoint changes.
  • 4D Geometric Control combines a static background point cloud with per-object 3D Gaussian trajectories in a shared world coordinate frame.Gaussian means encode motion paths, while covariances capture spatial extent and orientation.
  • VerseCrafter renders geometric states into multi-channel 4D control maps and injects them into a frozen Wan2.1-14B backbone through GeoAdapter.
  • VerseControl4D provides automatically derived prompts and rendered 4D control maps across 35K real-world training samples.

2. Related Works

Related work spans video world models and 3D world generation. These directions increasingly use transformer, diffusion, geometric, and camera-guided techniques to produce realistic or explorable environments.

  • Video World Models: Video world models predict future environment states for simulation, planning, and control, increasingly using transformer and diffusion backbones conditioned on actions, text, or camera trajectories.
  • 3D World Generation: 3D world generation methods synthesize explorable environments from text, images, or videos using representations including NeRFs, implicit fields, meshes, and 3D Gaussian splats.

3. Method

VerseCrafter represents scenes as editable 4D geometric states and converts them into decoupled control maps for a frozen video diffusion backbone. The pipeline combines background geometry, per-object Gaussian trajectories, camera poses, and lightweight adapter conditioning.

  • 4D Geometric Control: The method defines 4D Geometric Control as a static background point cloud plus per-object 3D Gaussian trajectories in a shared world coordinate frame.
  • Scene Construction: Input depth, camera intrinsics, and object masks are back-projected into per-object point clouds and a static background point cloud.
  • 3D Gaussian Trajectories: Each Gaussian trajectory models object occupancy with a time-varying mean, covariance, spatial extent, and orientation, offering flexible category-agnostic control.
  • Architecture: GeoAdapter encodes the rendered maps and injects them into frozen Wan2.1-14B blocks as residual geometric conditioning alongside text embeddings.
  • 3D Gaussian Trajectories: Gaussian parameters support user editing through ellipsoid visualization, trajectory dragging, and keyframing in world coordinates.
  • 4D Control Maps: The renderer produces background RGB/depth, trajectory RGB/depth, and a soft merged mask, while decoupled channels separate camera and object motion.

4. VerseControl4D Dataset

VerseControl4D is constructed by extracting and filtering real-world clips, automatically annotating them with captions and geometric signals, and rendering those signals into 4D control maps. The resulting data supports both camera-only and joint camera–object control evaluation.

  • Data Construction: VerseControl4D begins with Sekai-Real-HQ and SpatialVID-HQ, extracting 81-frame clips and applying quality filtering.
  • Data Annotation: Qwen2.5-VL-72B, Grounded-SAM2, and MegaSAM automatically provide captions, object masks, depth, and camera trajectories for retained clips.
  • Data Annotation: Annotated geometry is lifted into background and object point clouds, fitted with 3D Gaussian trajectories, and rendered into background, trajectory, and soft-mask control maps.
  • Dataset Composition: The dataset contains 35,000 training samples and 1,000 validation samples, including static scenes for camera-only control assessment.
  • Evaluation: Joint-motion comparisons report that VerseCrafter more faithfully follows camera and multi-object motion while preserving sharp, geometrically consistent backgrounds.
  • Evaluation: Camera-only comparisons report better target-trajectory following with sharp details and globally consistent 3D geometry.

5. Experiments

Experiments evaluate joint and camera-only control, visual quality, and three control-design choices. VerseCrafter achieves strong control and visual results, while ablations support Gaussian trajectories, depth-aware inputs, and decoupled controls.

  • 5.1. Joint Camera and Object Motion Control: VerseCrafter achieves the best overall video quality and most accurate joint camera-object control on VerseControl4D.The evaluation reports VBench-I2V quality and RotErr, TransErr, and ObjMC control metrics.
  • 5.2. Camera-Only Motion Control: VerseCrafter achieves the best VBench-I2V performance in camera-only static scenes while substantially reducing camera pose errors.Motion smoothness remains comparable to prior methods, and the videos preserve straight structures and stable depth relationships.
  • 5.3. Ablation Study: 3D Gaussian trajectories better follow intended object motion than point trajectories or 3D bounding boxes while preserving plausible shapes and background interactions.Point and box controls often produce scale drift and misaligned motion.
  • 5.3. Ablation Study: RGB+depth inputs recover more consistent parallax and occlusion than controls without depth.Without depth, foreground-background ordering and occlusion boundaries often drift over time.
  • 5.3. Ablation Study: Decoupling background and foreground controls preserves static geometry and produces more precise, stable object motion than merging them.The merged variant consistently underperforms the full model, especially for object-motion accuracy.

6. Conclusion

The paper concludes that VerseCrafter combines explicit 4D geometric control with a frozen Wan2.1 backbone and GeoAdapter conditioning. Its dataset and experiments support high-fidelity generation with precise joint camera and object-motion control.

  • 6. Conclusion: VerseCrafter represents scenes with a static background point cloud and per-object 3D Gaussian trajectories in a shared world coordinate frame.The representation is the basis for explicit 4D Geometric Control.
  • 6. Conclusion: GeoAdapter conditions a frozen Wan2.1 backbone on rendered 4D control maps for high-fidelity video generation with disentangled camera and multi-object control.The backbone remains unchanged while the adapter provides geometric conditioning.
  • 6. Conclusion: VerseControl4D contains 35K training samples with automatically derived prompts and rendered 4D control maps.The dataset supports training and evaluation of the proposed model.
  • 6. Conclusion: Experiments and ablations report superior visual quality and more accurate joint camera and object motion than existing controllable video generators and world models.The conclusion presents 4D Geometric Control as a promising interface for dynamic world simulation and editing.

B. Model Architecture Details

VerseCrafter keeps Wan2.1 frozen and adds GeoAdapter to process rendered geometry controls as latent spatio-temporal features. These features are injected into selected Wan-DiT blocks through residual modulation.

  • Backbone and GeoAdapter: The frozen Wan2.1 backbone includes a Wan Encoder, Wan-DiT denoiser, and Wan Decoder, while GeoAdapter supplies geometry-aware conditioning.The architecture leaves the pretrained backbone unchanged.
  • Geometry Encoding and Tokenization: Background RGB/depth maps, Gaussian-trajectory RGB/depth maps, and a soft merged mask are encoded or rearranged into a unified geometry feature.The feature is patchified and linearly projected into geometry tokens.
  • Residual Conditioning: GeoAdapter processes geometry tokens and injects residual modulations into selected Wan-DiT blocks.The adapter is attached after every selected stride of Wan-DiT blocks, with its output projected before addition to backbone tokens.
  • Training Initialization: The adapter uses zero-initialized output projections so VerseCrafter initially behaves like the original Wan2.1 backbone.During fine-tuning, the projection learns to inject geometry information through residual modulation.

C. VerseControl4D Dataset Details

VerseControl4D is a real-world dataset built through automated collection, filtering, prompting, and geometry annotation. It provides rendered background and object controls for both dynamic joint-control and static camera-only evaluation.

  • Dataset Construction: VerseControl4D is constructed through data collection, clip extraction, quality filtering, and data annotation.Annotation produces automatically derived prompts and rendered 4D control maps.
  • 4D Control Maps: The rendered controls include background RGB/depth maps, 3D Gaussian trajectory RGB/depth maps, and a soft merged mask.These signals are automatically derived for retained video clips.
  • Dataset Statistics: VerseControl4D contains 35,000 training samples and 1,000 validation samples.The dataset combines samples from Sekai-Real-HQ and SpatialVID-HQ.
  • Dataset Examples: Figure 10 visualizes input images, target camera trajectories, ground-truth frames, and the corresponding rendered control signals.The signals include background RGB/depth, object Gaussian-trajectory RGB/depth, and a final merged mask.
  • Scene Types: The dataset includes dynamic scenes with camera and foreground-object motion and static scenes with negligible object motion for camera-only evaluation.About 20% of training samples are static, and the validation set includes 250 static-scene samples.

D. Evaluation Metrics

The evaluation measures image-to-video quality, camera alignment, camera-position accuracy, and multi-object 3D motion control using VBench-I2V, CameraCtrl metrics, and ObjMC.

  • VBench-I2V: VBench-I2V reports eight learned dimensions of video-image consistency and perceptual quality, averaged into an Overall Score.Higher Overall Score values indicate better performance.
  • Camera Control: RotErr compares generated and ground-truth camera rotation matrices frame by frame using the relative rotation angle.Lower RotErr indicates better camera-orientation alignment.
  • Camera Control: TransErr sums per-frame Euclidean distances between generated and ground-truth camera translation vectors.Lower TransErr indicates camera positions more closely match the ground truth.
  • Object Motion Control: ObjMC compares estimated and ground-truth 3D Gaussian trajectories after matching objects with the Hungarian algorithm.Trajectory distances average 3D Gaussian-mean errors over time, while unmatched objects receive a penalty λ set to 10.0 m.
  • Object Motion Control: The final ObjMC score averages per-object trajectory errors, with lower values indicating more accurate multi-object 3D motion control.The matching formulation penalizes missed objects through the unmatched-object penalty λ.

E. Additional Qualitative Results

Additional qualitative results compare joint camera-and-object control, camera-only control, and targeted boundary cases. VerseCrafter more faithfully preserves motion, appearance, and geometric consistency, while the analyses identify limits for static backgrounds, coarse articulation, and some orientation cues.

  • Joint Camera and Multi-Object Control: In joint camera and multi-object scenes, VerseCrafter more faithfully follows camera and object motion while maintaining sharp appearance and geometrically consistent backgrounds.Perception-as-Control and Uni3C show object deformation; Yume lacks precise camera control, and Uni3C is limited to a single human.
  • Camera-Only Control: In static camera-only scenes, VerseCrafter better follows target camera trajectories while preserving sharp details and globally consistent 3D geometry.Other methods exhibit distorted facades, drifting structures, or inaccurate camera motion.
  • Targeted Analyses: The qualitative analyses examine orientation controllability, dynamic backgrounds, articulated and non-rigid objects, multi-view input, and monocular-depth robustness.These analyses target the fidelity, scope, and robustness of 4D Geometric Control.
  • Orientation Controllability: Orientation control is reliable for strongly anisotropic rigid objects but can produce ambiguous heading cues for human-like subjects represented by a single ellipsoid.When the minor axes are similar, projected footprint and depth changes may be too subtle, allowing the diffusion prior to cause heading mismatches.
  • Dynamic Background Modeling: Dynamic backgrounds are constrained because the background point cloud is reconstructed from the first frame and does not explicitly model per-frame non-rigid deformation.Fine, texture-dominant, highly non-rigid dynamics such as waterfalls are therefore only weakly constrained after rendering to 2D control maps.
  • Articulated and Non-Rigid Objects: A single 3D Gaussian can guide object-level motion in articulated and non-rigid examples, but it does not provide explicit part-level articulation control.The reported examples include robotic-arm extension and wind-blown clothes; finer articulation remains future work.

F.2. Geometry Coverage and Robustness

Geometry coverage improves novel-view faithfulness, while moderate depth errors remain tolerable because the point cloud serves as a coarse scaffold. Inference analysis identifies diffusion sampling as the practical bottleneck, with geometric state construction reusable across edits.

  • Geometry Coverage: Multi-view reconstruction expands coverage of weakly observed regions and improves faithfulness under larger viewpoint changes.The rear door is recovered only with multi-view input; single-view reconstruction remains plausible but is less faithful for novel views.
  • Robustness: Moderate monocular-depth errors do not critically undermine generation, which remains visually similar and preserves the main building structure.The diffusion prior can compensate for noisy depth because the reconstructed point cloud acts as a coarse geometric scaffold rather than a per-pixel hard constraint.
  • Boundary Cases: VerseCrafter is most effective when 4D geometric cues are informative, with failures concentrated in subtle human orientation changes and highly non-rigid background dynamics.These cases are identified as under-constrained boundary conditions in the additional control-fidelity analysis.
  • Memory Usage: FSDP reduces peak per-GPU memory from 90 GB to 70 GB with negligible runtime change, while CPU offload reaches 57 GB with a small additional time cost.The latter corresponds to a 36.7% memory reduction in the reported 50-step setting.
  • Inference Efficiency: 4D geometric state construction is cacheable across repeated edits, whereas diffusion inference remains the dominant per-edit bottleneck.Subsequent-edit latency is substantially reduced, and fewer denoising steps can lower it further.

H. Limitations and Future Work

VerseCrafter remains limited by coarse object and mostly static background representations, lacks explicit physical constraints, and is computationally expensive at high resolution and long temporal horizons. The proposed future directions target finer articulation, dynamic backgrounds, physics guidance, and more efficient generation.

  • Object Representation: A single 3D Gaussian provides only ellipsoid-level object control, limiting fine-grained pose and part-level articulation for human-like or near-symmetric objects.Multiple Gaussians per object or articulated 3D structures are proposed to improve orientation and pose control.
  • Background Dynamics: The first-frame background point cloud is mostly static, limiting controllability for highly non-rigid and texture-dominant dynamics such as waterfalls.Explicit dynamic background representations or temporally evolving scene geometry are suggested as remedies.
  • Physical Realism: VerseCrafter does not impose explicit physical constraints during generation, leaving complex interactions without collision, contact, or ground guidance.Potential extensions include collision-aware losses, contact constraints, ground constraints, and differentiable physics guidance.
  • Efficiency: High-resolution and long-horizon generation remains computationally expensive because the system uses a large frozen diffusion backbone and renders multi-channel controls for every frame.Future work includes efficient backbones, distilled sampling, cached control encoding, and streaming or long-video synthesis.
Loading 2601.05138v2…