Source-linked AI summary
WorldCraft: From Camera Navigation to Object Manipulation in Interactive Video World Models
Bohai Gu, Taiyi Wu, Yueyang Yuan, Jian Liu, Xiaocheng Lu, Dazhao Du, Jie Zhang, Jinxiang Lai, Shuai Yang, Xiaotong Zhao, Alan Zhao, Song Guo
TL;DR
Video-based world models support camera navigation but lack object-level actions, limiting their interactivity. WorldCraft adds trajectory-based object manipulation while preserving camera control and tracking moved objects across off-camera rollouts. Experiments show accurate object control, maintained camera fidelity, and persistent object state.
Problem
Video-based world models support camera navigation but lack object-level actions, although real interaction requires trajectories attached to specific objects.
Method
WorldCraft combines camera-invariant world trajectories, spatial-pathway adaptation, and persistent trajectory-conditioned memory to control selected objects during camera navigation.
Results
WorldCraft achieves accurate object control, preserves camera fidelity, and maintains moved-object state across long-horizon rollouts with off-camera excursions.
Takeaways & Limitations
WorldCraft extends interactive video world models toward composable camera navigation and object manipulation with scene consistency and object-state preservation.
Takeaways & Limitations
The method does not predict uninstructed off-camera dynamics, relies on monocular depth estimation, and has limited precision for very small objects.
Abstract
from arXiv · showhide
Recent video-based world models have made pixel-space environments interactive at the camera level: users can navigate viewpoints while the model generates coherent visual continuations. Yet their action spaces remain incomplete: users can move the camera, but cannot act on individual objects. Since real-world interaction is inherently object-centric, such models remain closer to passive scene observers than truly manipulable environments. We present WorldCraft, a framework that expands interactive video world models from camera navigation to object-level trajectory actions. Given a user click and a sketched path, WorldCraft generates future frames in which the selected object follows the prescribed trajectory while the camera continues to navigate the scene. WorldCraft achieves this through a trajectory-centric control pipeline: First, Normalized World Trajectory (NWT) represents user-drawn motion in a camera-invariant world coordinate system and dynamically re-projects it under the current camera pose, separating object motion from camera-induced screen-space displacement; Spatial-Pathway LoRA (SP-LoRA) then injects this world-space signal through the model's spatial-control pathway, adding object manipulation capability while preserving the pretrained camera controller; finally, Trajectory-Anchored State Persistence (TASP) treats the world trajectory as a persistent spatial state and refreshes autoregressive memory after trajectory-conditioned generation, allowing moved objects to reappear at their updated positions after leaving the camera view. Experiments show that WorldCraft enables accurate object control, preserves the video-based world model's camera fidelity under camera-only evaluation, and maintains object state across long autoregressive rollouts with off-camera excursions.
1 Introduction
WorldCraft extends interactive video world models beyond camera navigation by enabling selected objects to follow user-specified trajectories while the camera continues navigating. It addresses camera–object motion entanglement and preserves interactive camera control through a trajectory-centric design.
- Motivation: Recent video-based world models generate coherent visual continuations for camera navigation, but their action spaces stop at the camera despite real interaction being object-centric.Users can navigate viewpoints, yet cannot manipulate individual objects.
- Challenges: Object-level actions introduce coupled challenges: disentangling object motion from camera-induced screen-space displacement and adding trajectory control without overwriting camera control.Pixel trajectories entangle object motion with ego-motion when the camera moves.
- WorldCraft: WorldCraft lets users select an object with a click and specify its motion with a sketched path, generating future frames where the object follows that trajectory as the camera navigates.The framework equips an interactive video world model with object-level trajectory actions while preserving camera control.
- Contributions: WorldCraft formulates object trajectory control as a new autoregressive video-world-model action modality, enabling entity manipulation alongside continued camera navigation.This formulation treats object trajectories as actions rather than static-camera trajectory-guided video generation.
- Contributions: Normalized World Trajectory lifts user paths into normalized world space and dynamically re-projects them under the current camera pose to disentangle ego-motion from object motion.The representation is camera-invariant and supports object control during camera movement.
2 Related work
Prior interactive video world models enable pixel-space simulation but largely constrain control to camera motion or discrete game commands. Complementary video-generation methods introduce trajectory-conditioned object control, including selected-target and unified camera/object motion approaches.
- Interactive video world models: Early interactive video world models demonstrated action-driven video generation, but their control spaces remained limited to camera motion or game-style discrete commands.GameNGen and GameGen-X established action-conditioned video generation within this constrained-control setting.
- Interactive video world models: Yume, GameCraft, and Matrix-Game 2.0 advanced long-horizon visual simulation and camera controllability in open or game-like environments.WorldPlay further combined camera-action conditioning with autoregressive modeling and provided a strong open-sourced baseline.
- Trajectory-guided video generation: DragNUWA, DragAnything, and MotionCtrl enabled trajectory-conditioned object motion control in video diffusion models.DragNUWA conditions on motion trajectories with image and text inputs; DragAnything binds trajectories to selected targets through entity representations; MotionCtrl unifies camera and object motion control.
3 Method
WorldCraft extends an autoregressive video world model with object-level trajectory actions while preserving camera control. Its pipeline combines camera-invariant trajectory representation, spatial-pathway adaptation, and persistent trajectory-conditioned state across autoregressive generation.
- Method overview: WorldCraft adds object-level trajectory actions to an autoregressive video world model while preserving its camera-control capabilities.The method comprises Normalized World Trajectory, Spatial-Pathway LoRA, and Trajectory-Anchored State Persistence.
- Trajectory injection: The trajectory condition replaces image-conditioning channels with displaced first-frame latent features, preserving PatchEmbed compatibility and enabling coarse zero-shot trajectory following.The model input remains [zt; ĉtraj; m], with unassigned positions set to zero.
- Normalized World Trajectory: NWT lifts user screen trajectories into a first-frame-anchored normalized world coordinate system and re-projects them under the current camera pose, separating object and camera motion.This allows user-specified object motion and model-driven camera motion to compose without manual compensation.
- Normalized World Trajectory: Iterative anchor refinement updates the visual anchor and object depth after each generated chunk, reducing projection errors during long autoregressive rollouts.The updated anchor-depth pair is used for subsequent trajectory re-projection.
- Spatial-Pathway LoRA: Full-parameter fine-tuning changes the action encoder and ProPE projections by 10-25× more than attention and feed-forward layers, motivating LoRA on the spatial-control pathway.SP-LoRA adapts only the action encoder and ProPE projection layers while freezing all other parameters to preserve pretrained camera behavior.
- Trajectory-Anchored State Persistence: TASP treats the world model as a persistent state predictor so the world continues evolving when the camera looks elsewhere.It resolves this through two coordinated mechanisms, as introduced in the method.
4 Experiments
Experiments evaluate WorldCraft on trajectory control, camera fidelity, composable camera-object actions, and qualitative long-horizon behavior. WorldCraft achieves strong trajectory and visual-quality results, preserves camera control, maintains off-camera object state, and supports extended composable manipulation.
- Evaluation: WorldCraft is evaluated on held-out trajectory, camera-fidelity, and composable camera-plus-trajectory test sets using trajectory, consistency, and relative-pose metrics.The evaluation includes 50 static-camera trajectory clips, 50 dynamic-camera camera-only clips, and 45 composable clips stratified by camera-rotation magnitude.
- Quantitative results: WorldCraft achieves the lowest trajectory error and the best pixel fidelity, semantic consistency, and VBench++ consistency across all 50 static-camera trajectory clips.Comparisons use identical first frames and object-trajectory conditions against DragAnything and Wan-Move.
- Quantitative results: At 61 frames, WorldCraft’s RPErot is 0.131 versus 0.120 for WorldPlay and 0.252 for the next-best external baseline; at 253 frames, it reaches 0.123 versus WorldPlay’s 0.130.The 253-frame evaluation is a horizon 4× longer than the main protocol.
- Qualitative results: Qualitative comparisons show precise object control under sparse trajectory signals and recovery of an object at its correct updated position after off-camera motion.In the off-camera demonstration, WorldCraft uses TASP while the camera pans away and returns; competing models either lose scene consistency or cannot track the updated object state.
- Extended capabilities: WorldCraft supports part-level and simultaneous multi-object control during long-horizon composable camera-object generation extending to 253 frames (∼10.5 s at 24 fps).Examples include independently steering three objects and moving a shield while keeping the body still.
- Ablations: Ablations show that world-space trajectories improve accuracy over pixel-space conditioning, while SP-LoRA and the Static-BI →Dynamic-AR curriculum provide the best joint trajectory-control and camera-fidelity trade-off.Iterative monocular depth refinement performs best across camera-rotation magnitudes, especially for large rotations.
5 Conclusion
WorldCraft extends camera-controlled video world models with precise object-level action control. It combines normalized world-space trajectories with a lightweight pathway-selective LoRA to add trajectory controllability while preserving camera fidelity.
- Framework: WorldCraft extends camera-controlled video world models with precise object-level action control.The framework targets object-level actions beyond camera motion.
- Framework: A shared spatial-control pathway links camera motion and object trajectories.WorldCraft identifies this shared pathway as the basis for adapting the model.
- Control adaptation: A lightweight pathway-selective LoRA adds trajectory controllability while preserving the base model’s camera fidelity.The LoRA adapts the shared pathway without replacing the pretrained camera behavior.
- Trajectory representation: Normalized world-space trajectories decouple object motion from ego-motion and enable composable camera control.The trajectories provide the input-level representation for separating object movement from camera-induced motion.
A Progressive training
WorldCraft uses a three-stage progressive training strategy to avoid catastrophic interference between trajectory control and camera control. Training progresses from real-domain adaptation, through static-camera trajectory learning, to dynamic-camera joint motion.
- Motivation: Trajectory and camera control share a spatial parameter subspace, making naïve trajectory training prone to catastrophic interference with camera capabilities.The shared pathway motivates progressively increasing data complexity while constraining attention mode.
- Stage 0: Stage 0 adapts the pretrained synthetic-data model to the real-data domain using full-parameter fine-tuning at 5 × 10−7 for 2000 steps without trajectory conditioning.Setting trajectory_rate=0 avoids camera–trajectory conflict during domain adaptation.
- Stage 1: Stage 1 learns trajectory control from static-camera data with BI attention and layer-selective LoRA, yielding a clean trajectory-to-object-motion mapping.Static cameras remove screen-space entanglement, while LoRA protects camera parameters in frozen base weights.
- Stage 2: Stage 2 extends training to dynamic-camera data with AR attention, teaching simultaneous camera and object motion.This stage introduces the combined motion setting after static-camera trajectory learning.
B Scalable data curation pipeline
WorldCraft builds a scalable pipeline that converts unlabeled videos into structured (video, camera, mask, trajectory) training tuples using off-the-shelf vision models and physical-plausibility filtering. The pipeline estimates cameras, discovers salient moving subjects, tracks them, and extracts standardized multi-point trajectories.
- Pipeline overview: The pipeline extracts video clips, camera parameters, moving-subject masks, and multi-point trajectories from unlabeled video at scale.Physical-plausibility filtering removes unsuitable samples.
- Camera estimation: Camera parameters come from ViPE for unannotated WISA-80K videos, while SpatialVID-HQ annotations are converted to pixel-space matrices and interpolated to full frame rate.SpatialVID-HQ uses normalized intrinsics and world-to-camera quaternion–translation pairs before conversion and Slerp interpolation.
- Subject discovery: Subject discovery uses VLM-guided open-vocabulary detection on WISA-80K, querying Qwen3-VL-8B and passing its parsed description to GroundingDINO.The method samples 5 frames uniformly from each clip and requires no category-specific priors.
- Tracking and trajectory extraction: SAM2 uses mask logits, a centroid point, and a bounding box to initialize tracking, reducing identity switches in crowded scenes compared with box-only prompting.The compound prompt combines complementary signals for subject identity and spatial localization.
C Additional analysis details
Activation-level analyses show that trajectory control is added as a small perturbation while preserving the camera effect direction. Camera and trajectory updates also occupy a shared, camera-compatible spatial-control subspace, especially in middle layers.
- Activation-level interaction: Trajectory control adds a small perturbation on top of the camera signal across all 54 transformer blocks, rather than overwriting the camera representation.The analysis probes prope_proj outputs using single-step denoising comparisons across baseline, camera-only, trajectory-only, and combined conditions.
- Activation-level interaction: Mean traj/cam energy ratio = 0.42, indicating that the trajectory signal is substantially smaller than the camera signal.The ratio is computed from per-layer activation deltas under the four input conditions.
- Camera preservation: Mean = 0.89 for cos(cα, cβ), confirming that trajectory variation does not alter the camera effect direction.The counterfactual test crosses two camera poses with two trajectories and evaluates camera-effect similarity layer by layer.
- Shared control subspace: Camera- and trajectory-control updates are aligned along shared principal directions, rather than forming orthogonal subspaces, particularly in middle layers.PCA and cosine-based principal-angle analysis supports injection within a camera-compatible spatial-control subspace.
D Limitations.
WorldCraft’s state persistence is limited to entities with user-specified trajectories, while broader off-camera dynamics remain unresolved. Its control also faces projection errors under large camera rotations and limited precision for very small objects.
- State persistence: State persistence covers only entities with user-specified trajectories, leaving uninstructed dynamics in the broader off-camera world unresolved.Predicting those uninstructed off-camera dynamics remains an open problem.
- Camera compensation: Monocular-depth-based camera-trajectory compensation introduces projection error at large camera rotations.The error arises from reliance on monocular depth estimation.
- Control precision: Latent-token control at 16×16 pixels limits precision when manipulating very small objects.The control granularity is 16×16 pixels.
E Broader impact.
WorldCraft advances world models from passive observation toward active object manipulation, with relevance to embodied AI, content creation, and simulation. Its trajectory signal can also communicate persistent world state for globally consistent dynamics when entities are occluded.
- Broader impact: WorldCraft supports active manipulation beyond passive observation, with potential relevance to embodied AI, content creation, and simulation.The framework expands world-model capabilities toward object-level interaction rather than camera navigation alone.
- Broader impact: The largest full-parameter updates concentrate on action_in, prope_proj, and final_layer adaLN modulation, with no attention or MLP parameters in the top 30.prope_proj accounts for 22 of the top 30 rows, confirming concentration on the spatial-control pathway.
- Broader impact: Trajectory becomes a world-state communication channel that maintains globally consistent dynamics without continuous visual observation of every entity.In a self-driving simulator, occluded pedestrians can continue walking through trajectory-based state communication.