Source-linked AI summary
WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
Wangbo Yu, Kunhao Liu, Wenbo Hu, Shenghai Yuan, Chaoran Feng, Haiyang Zhou, Yukun Huang, Yiran Wang, Wang Zhao, Yingmin Luo, Ying Shan
TL;DR
Video world models need to preserve previously observed content across viewpoints and long horizons, beyond what camera control and streaming alone provide. WorldCrafter uses jointly trained, camera-queryable 3D-aware memory, and reports improved revisit consistency and camera control over evaluated baselines.
Problem
Video world models need memory beyond recent context to keep previously observed content consistent when revisited from different viewpoints.
Source: §1 · p. 2, §2.1 · p. 2Method
WorldCrafter jointly trains a pretrained-3D-initialized memory encoder and video generator, using camera-conditioned memory tokens alongside recent history for chunk-wise generation.
Source: §1 · p. 2, §3.3 · p. 4, §3.1 · p. 4Results
Takeaways & Limitations
Experiments show improved revisit consistency and camera-control accuracy over the evaluated baselines.
Source: §5 · p. 9Consistency can still break down along particularly complex or extended trajectories, and re-encoding history at every chunk adds latency.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator's limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.
1. Introduction
WorldCrafter addresses the need for video world models to recall previously observed content across long horizons and viewpoints. It combines learned 3D-aware memory with camera-conditioned generation, reporting stronger revisit consistency and camera control.
- Existing memory approaches face trade-offs among full-history computation, selective-retrieval view coverage, geometry-dependent spatial memory, and appearance fidelity in geometry-oriented implicit memory.Source: §1 · p. 2
- A memory encoder initialized from pretrained 3D representation encoders maps historical latents into a compact memory space jointly optimized with the video DiT and readout module.Source: §1 · p. 2
- Pose-guided readout focuses memory tokens on requested viewpoints, while recent temporal context supports visible-motion continuation and camera-conditioned generation supports minute-scale streaming.The tokens condition the DiT directly through self-attention without reconstructing target-view images; encoder input and token budgets remain fixed.Source: §1 · p. 2
- WorldCrafter achieves leading revisit consistency, a 47.6% improvement relative to the strongest baseline, and camera-control accuracy.Source: §1 · p. 2
2. Related Work
Related work extends interactive video generation with streaming and camera control, while identifying persistent scene recall as a distinct challenge. Memory approaches differ in what they store and how they preserve viewpoint correspondence and scene appearance.
2.1. Interactive Video World Models
- Streaming rollouts and camera-control signals do not by themselves ensure faithful recall of previously observed scenes beyond the context window.Source: §2.1 · p. 2
2.2. Memory Mechanisms in Video World Models
- Persistent video world models are grouped by stored representation into context, spatial, and implicit memory.Source: §2.2 · p. 2
- Context-memory methods reuse or query historical frames and features, whereas WorldCrafter aggregates history into a 3D-aware representation and reads fixed-size tokens before denoising.Source: §2.2 · pp. 2–3
- Spatial-memory methods synthesize target-view frames to provide cross-view correspondence, but strong target-view alignment can constrain modeling of dynamic scenes.Source: §2.2 · p. 3
- Geometry-aware methods use geometry estimators or supervision, while WorldCrafter adapts multi-view representations learned through novel-view reconstruction to preserve geometry and appearance.Its memory encoder and readout inherit a 3D inductive bias and are jointly optimized with the video generator.Source: §2.2 · p. 3
3. Method
WorldCrafter extends chunk-wise video generation with implicit 3D-aware memory, read out as fixed-size tokens conditioned on requested camera views. It jointly trains the memory and generator, then uses pyramid distillation for faster interaction.
3.1. Preliminary
- Video is encoded into VAE latents, and chunk-wise autoregressive generation denoises successive chunks using a fixed recent-history window.Source: §3.1 · pp. 3–4
3.2. Model Architecture
- Each chunk is conditioned on accumulated-history memory and a target camera trajectory; the DiT processes memory, recent context, and noisy latents together, with camera injection applied only to noisy latents.Source: §3.2 · p. 4
3.3. Memory Encoder
- The encoder maps history latents and camera poses into an implicit representation; k-frame coverage selection bounds encoding, while target-pose readout improves camera control over pose-free readout.The representation aggregates geometry and appearance without explicit 3D reconstruction, and readout produces fixed-size memory tokens.Source: §3.3 · pp. 4–5
3.4. Base Model Training
- Training proceeds in four stages: OSP adaptation, camera-branch training, latent-memory warmup, then joint training of memory, readout, DiT, and camera conditioning.Data combine OSP, DL3DV, and synthetic MIND videos; the filtered OSP subset includes camera-following moving subjects.Source: §3.4 · p. 5
3.5. Distillation for Real-time Interaction
- Pyramid distillation uses two denoising steps at each of three resolutions; separate low- and high-noise models balance natural details against subject-following, reaching 16 fps on a 4-GPU machine.The low-noise model performs the final denoising step, while the high-noise model performs preceding steps.Source: §3.5 · p. 5
4. Experiments
Across static and dynamic scenes, WorldCrafter improves long-horizon revisits and camera control while maintaining visual quality. Ablations support the memory design choices, and its memory processing has lower reported latency than depth-based spatial memory.
4.1. Experimental Setup
- The benchmark covers 83 dynamic and 62 static scenes; five trajectories per image yield 725 videos per method, spanning 528–1,648 frames with closed-loop revisits.The comparisons include eight camera-controllable baselines evaluated with identical initial images, text descriptions, and target trajectories.Source: Alaya-EVOKE · pp. 6–7
4.3. Camera Control Evaluation
- WorldCrafter achieves the lowest error on all three camera-control metrics, while WorldCrafter-fast ranks third on each.Trajectories are normalized to their first pose and aligned using Sim(3).Source: Table 2 · p. 6, §4.3 · p. 8
4.4. Visual Quality Evaluation
- WorldCrafter achieves the highest VBench overall score, 81.910, and the two WorldCrafter models lead five of eight visual-quality dimensions.WorldCrafter-fast achieves the best Temporal Flickering score.Source: Table 3 · p. 7, §4.4 · pp. 8–9
4.5. Ablation Study
- Replacing implicit memory with four retrieved history frames worsens revisit consistency and camera control; its revisit-error disadvantage widens at longer intervals.Figure 5(a) plots mean revisit LPIPS against the interval between the initial observation and revisit.Source: Figure 5 · p. 8, Table 4 · p. 8, §4.5 · p. 9
- Memory processing takes 0.062 s per chunk for WorldCrafter versus 1.346 s for depth-based spatial memory, a 21.7× reduction in summed cost.At 640 × 384 resolution, encoding takes 0.049 s and readout 0.013 s; depth estimation and alignment take 0.409 s, with warping taking 0.937 s.Source: Figure 5 · p. 8, §4.5 · p. 9
- Freezing the memory encoder weakens memory and camera control, while pose-guided readout improves both abilities over pose-free readout.At matched iterations, joint optimization reaches lower revisit error earlier and maintains the advantage.Source: Figure 5 · p. 8, Table 4 · p. 8, §4.5 · p. 9
- Max-coverage retrieval improves revisit consistency and camera control over pairwise FoV-similarity ranking while using the same number of history inputs.Both variants retain the latest latent frame and select eight additional frames under the same DiT memory-token budget.Source: Table 4 · p. 8, §4.5 · p. 9
5. Discussion
WorldCrafter combines implicit 3D-aware memory with camera-controllable autoregressive video generation, improving revisit consistency and camera-control accuracy over evaluated baselines. Consistency can still break down on complex or extended trajectories, and re-encoding history at every chunk adds latency.
5. Discussion
- WorldCrafter uses implicit 3D-aware memory in camera-controllable autoregressive video generation, improving revisit consistency and camera-control accuracy over evaluated baselines.A learned encoder aggregates historical latent frames into a compact representation read out as tokens compatible with the video DiT.Source: §5 · p. 9
- Consistency can still break down along particularly complex or extended trajectories.Source: §5 · p. 9
- Re-encoding history at every chunk incurs extra latency; an incremental autoregressive streaming memory encoder is proposed to reduce repeated computation.Source: §5 · p. 9