Source-linked AI summary
Wonder: Video World Model Done Better
Jiacong Xu, Hanwen Jiang, Zhixin Shu, Kalyan Sunkavalli, Vishal M. Patel, Yiqun Mei
TL;DR
Interactive world models must combine camera control, persistent long-horizon coherence, and low-latency response, but achieving these properties simultaneously remains challenging. Wonder jointly redesigns camera conditioning, sparse full-fidelity memory, and distillation, enabling minute-scale, 16 FPS rollouts with coherent revisits and improved camera following in image- and video-conditioned settings.
Problem
Interactive visual world models must simultaneously provide accurate control, low-latency response, long-horizon coherence, and source-content preservation, but these properties remain difficult to achieve together.
Method
Wonder jointly redesigns camera conditioning with dense coordinate fields, sparse full-fidelity memory, and autoregressive distillation for real-time camera-controllable exploration.
Results
Wonder improves visual quality and camera-following accuracy over recent streaming world models while supporting coherent revisits, dynamic-content preservation, and minute-scale rollouts at 16 FPS.
Takeaways & Limitations
Wonder provides a unified approach to interactive world exploration across image- and video-conditioned settings with stable latency as generation history grows.
Takeaways & Limitations
Point-cloud-based camera control depends on external reconstruction quality and is view-limited, making it less suitable for open-ended exploration.
Abstract
from arXiv · showhide
We present Wonder, a general-purpose video world model for real-time, camera-controllable world exploration. Given an image or a conditional video, Wonder constructs a playable world where users can navigate interactively by moving the camera, discovering unseen regions, and revisiting previously observed areas in real time and over a long-term horizon. Achieving this capability requires a system-level co-design of control method, memory mechanism, and training strategy. We introduce a novel camera conditioning with a dense coordinate field whose renderings provide spatially aligned motion and orientation cues, allowing the model to interpret camera motion directly as visual evidence. To support fast and precise memory retrieval over a growing generation context, we propose an efficient sparse attention-based memory mechanism, enabling the model to selectively attend to a small set of relevant context tokens at inference time, regardless of actual context length. We further develop several techniques to rectify the self-forcing-style distillation pipeline, improving the student model's ability to respect control signals, as well as maintaining diverse generation modes and long-term memory from the teacher. Together, these components enable Wonder to synthesize diverse, minute-scale videos at 16 FPS while preserving coherent geometry, appearance, and dynamics across long rollouts. Beyond image-to-video generation, Wonder naturally supports video-conditioned generation, allowing existing dynamic scenes to be re-shot in real time.
1 Introduction
Wonder is introduced as a general-purpose video world model for real-time, camera-controllable exploration from image or video conditions. It co-designs camera control, sparse memory, and distillation to preserve controllability, persistence, efficiency, and long-horizon coherence.
- Motivation: Interactive visual world models construct persistent environments from images, videos, or text and let users or agents explore them through control signals.This shift extends video generation beyond fixed-duration clip synthesis toward interactive world modeling.
- Limitations: Existing methods address only subsets of the requirements: camera-controllable generators are often bidirectional, fixed-length, and expensive, while streaming methods trade off other capabilities.These limitations make prior systems unsuitable for low-latency interaction or minute-scale rollout.
- System design: Wonder jointly redesigns control representation, memory, and distillation as a coupled teacher-to-student pipeline for camera-controlled causal streaming generation.The pipeline adapts a bidirectional diffusion model, converts it into a causal autoregressive generator, and distills it into a few-step streaming model.
- Control: Its control representation renders camera motion as visual evidence through a synthetic 3D scaffold and environment map along the target trajectory.This directly exposes translation, rotation, and scene-motion cues rather than requiring the diffusion model to infer their image-space effects from geometry alone.
- Memory: Wonder uses sparse full-fidelity memory to retrieve recent context for local continuity and distant context for persistent scene memory without latency growing with rollout length.Dense attention to all historical KV states causes per-step latency to increase as the rollout grows.
- Results: Experiments show improvements in visual quality and camera-following accuracy, with minute-scale rollouts at 16 FPS and stable latency as history grows.Wonder also explores beyond observed views, preserves dynamic content when re-rendering source videos, and maintains coherent revisits over long trajectories.
2 Related Work
Related work spans latent video diffusion foundations, causal autoregressive generation, camera-conditioned synthesis, and interactive video world models. These efforts advance controllable and long-horizon generation, while Wonder addresses persistent challenges to enable more stable interactive world exploration.
- Video Diffusion Models: Latent diffusion transformers built on 3D VAEs have become de facto pretrained foundations for downstream video generation, producing multi-second, high-fidelity clips.Examples include CogVideoX, HunyuanVideo, Wan, and LTX-Video.
- Autoregressive Video Generation: Causal autoregressive methods convert bidirectional diffusion models into streaming generators through next-token prediction, distillation, and training on self-generated rollouts.Diffusion Forcing, CausVid, Self-Forcing, Rolling Forcing, and Causal Forcing progressively target causal generation and long-horizon real-time streaming.
- Camera-controllable Video Generation: Camera-controllable video generation commonly uses pose or geometric conditioning, including 6-DoF extrinsics, Plücker embeddings, and point-cloud renderings.MotionCtrl, CameraCtrl, and ViewCrafter exemplify these conditioning strategies for trajectory-controlled synthesis and geometric consistency.
- Interactive Video World Models: Interactive video world models respond to user actions while pursuing real-time streaming, long-horizon memory, richer action interfaces, and dynamic-world re-rendering.Prior systems include Genie 3, GameCraft, Yume, RELIC, LingBot-World, DreamX-World, and Inspatio-World, with Inspatio-World providing I2V and V2V baselines.
- Interactive Video World Models: Despite these advances, prior interactive world models still face persistent challenges, motivating strategies for more stable interactive world exploration.The supplied discussion identifies unresolved challenges before concluding that the proposed strategy alleviates them.
3 Training Data
Wonder constructs a unified training-data curation pipeline because controllable, long-horizon world modeling requires diverse videos, accurate camera motion, and temporal coverage that preserves scene geometry, appearance, and dynamics. The pipeline combines real navigation videos, long paired V2V data, progressive clip durations, trajectory processing, and quality filtering.
- Data requirements: The unified curation pipeline targets diverse visual content, accurate camera motion, and sufficiently long temporal horizons for controllable world-model training.Long-horizon modeling additionally requires preserving scene geometry, appearance, and dynamics over extended rollouts.
- I2V data: DL3DV supplies diverse real-world navigation videos with naturally captured camera trajectories for I2V training, but its motions are often simple and smooth.The passage contrasts these trajectories with interactive controls such as sharp turns, lateral movements, backward motion, and compound camera actions.
- V2V data: V2V training requires paired videos sharing an underlying dynamic scene while differing in camera trajectory or temporal evolution.MultiCamVideo and CamXTime provide useful paired videos, but their clips are typically short, often around 5 seconds, so Wonder renders long paired V2V data.
- Annotation and filtering: Videos are clipped into 5s, 10s, and 20s durations to support progressive training from short to long horizons.Rule-based filters remove corrupted videos, black frames, failed renderings, and trajectories with collision or clearly invalid motion; a VLM generates hierarchical captions for each long clip.
- Trajectory processing: Estimated camera trajectories are Gaussian-smoothed before discrete action conversion, while reverse playback and speed resampling balance the action distribution.The additional VLM-based quality-checking stage removes videos that fail its quality criteria.
4 Wonder World Model
Wonder is a multimodal video world model that supports explorable image- and video-conditioned generation under user-streamed camera trajectories. Its framework combines camera-conditioned teacher distillation with a few-step autoregressive model to target action following and low-latency interactive streaming, while addressing long-horizon rollout challenges.
- Inference modes: Wonder unifies image-to-video and video-to-video inference within one model for multimodal video world modeling.Image-to-video initializes an explorable world from one frame, while video-to-video re-shoots an existing event from a target camera trajectory.
- Training framework: The system distills a few-step autoregressive video diffusion model from a bidirectional video diffusion teacher, with both conditioned on camera-control signals.This two-stage pipeline is designed for low-latency generation, but high-quality, memory-aware long-horizon rollout remains challenging.
- Camera conditioning: A dense pixel-space camera representation lets the teacher interpret camera motion as a pixel-aligned visual signal, yielding strong action-following behavior.The teacher is then distilled into a few-step autoregressive video diffusion model for low-latency interactive streaming.
4.1 Stage 1: Camera-Conditioned Teacher for Long Video Generation
Stage 1 develops a bidirectional camera-conditioned teacher that initializes the autoregressive student and supervises distillation. Its pixel-space coordinate-field representation turns camera translation and rotation into dense visual signals, while unified source-target inputs support both image- and video-conditioned generation.
- Teacher Objective: A strong bidirectional teacher provides both initialization for the autoregressive student and supervision for distillation.The teacher is trained for precise camera following and multimodal input support.
- Camera Representation: Pixel-space camera conditioning exposes motion as spatially aligned visual evidence instead of relying solely on implicit pose embeddings or explicit point-cloud rerendering.The lattice representation is designed to combine the strengths of implicit and explicit camera-control approaches while avoiding their stated limitations.
- Camera Representation: A dense 3D scaffold induces parallax for metric translation, while an environment map at infinity provides orientation-dependent cues for rotation.Together, these components convert translation and rotation into dense, frame-aligned signals interpretable by the video diffusion model.
- Model Architecture: Wan2.1-I2V-14B supplies the bidirectional video-diffusion backbone, combining a spatio-temporal VAE with a diffusion transformer.The VAE uses a 3D causal encoder-decoder and downsamples spatial resolution by 8× and temporal resolution by 4×.
- Unified Conditioning: The unified input format places an optional clean source segment before a target segment, enabling both video-conditioned and image-conditioned generation.Video conditioning uses clean source latents, whereas image conditioning supplies the target video's first frame as a clean, noise-free latent.
4.2 Stage 2: Fast Autoregressive Student for Real-Time Streaming
Stage 2 distills the bidirectional, many-step teacher into a causal few-step student for low-latency streaming, while sparse memory retrieval preserves long-horizon revisits. Sparse context forcing, mixture-of-students, and GAN control regularization improve robustness, capacity, camera following, and detail quality during autoregressive rollout.
- 4.2 Stage 2: Fast Autoregressive Student for Real-Time Streaming: The teacher is distilled into an autoregressive few-step student because bidirectional attention and many denoising steps make direct interactive control prohibitively expensive.The student targets low-latency streaming generation.
- 4.2 Stage 2: Fast Autoregressive Student for Real-Time Streaming: Full-fidelity KV history is retained, while pooled key summaries select a small set of relevant memory chunks through content-aware sparse attention.This avoids discarding persistent memory needed for consistent revisits while reducing retrieval cost.
- 4.2 Stage 2: Fast Autoregressive Student for Real-Time Streaming: The active memory combines the initial chunk, the most recent r chunks, and the top-k middle-history chunks by query-key similarity, with r = 2 preserving local motion smoothness.The initial chunk acts as an attention sink, recent chunks preserve short-term continuity, and retrieval uses lightweight pooled summaries.
- 4.2 Stage 2: Fast Autoregressive Student for Real-Time Streaming: Sparse context forcing warms the student during ODE initialization, and random dropping of optional historical edges aligns training with sparse inference while preserving anchors and recent context.Required self-attention, first-frame anchors, and recent context frames remain intact; longer horizons use stronger optional-edge drop rates.
- 4.2 Stage 2: Fast Autoregressive Student for Real-Time Streaming: A mixture-of-students increases capacity to capture diverse teacher modes, addressing mode shrinkage, weakened local realism, and degraded fine details in long-horizon rollouts.The design preserves the same autoregressive few-step inference interface.
- 4.2 Stage 2: Fast Autoregressive Student for Real-Time Streaming: GAN control regularization uses frozen teacher representations and low-frequency feature differences to stabilize distillation and substantially improve camera following without introducing prior regression-based mode shrinkage.Three learnable register tokens aggregate low-frequency feature-difference maps after strided downsampling suppresses texture details.
4.3 Inference-time Optimization
Inference-time optimization targets low latency and long-horizon generation through compiled GPU execution, multi-GPU parallelism, and rolling KV-cache management. These techniques reduce repeated computation and distribute memory and computation while extending caching beyond the training horizon.
- Low-latency execution: Compiled GPU kernels and replayed CUDA execution patterns reduce launch overhead during repeated prediction and cache updates.The model operations are compiled once, then reused across runs to reduce GPU memory traffic, kernel-launch overhead, and repeated transformer computation.
- Multi-GPU parallelism: Multi-GPU parallelism distributes computation and memory, using sequence parallelism for FFN and cross-attention and tensor parallelism for self-attention.NCCL All-to-All collectives convert between sequence- and head-partitioned tensor layouts, while the KV cache is partitioned across attention heads.
- Long-horizon caching: A rolling/sliding-window cache supports minutes-long generation by combining sink, dynamically selected top-k, and recent-context chunks.Frame indices are remapped by relative distance to remain within the training horizon and avoid unseen positional embeddings.
4.4 Training Infrastructure and Model Details
Training the bidirectional teacher and three 14B student generators over a 20-second horizon is highly memory-intensive, motivating distributed training techniques that shard model states and video-token sequences across GPUs.
- Training Infrastructure: The 20-second training horizon jointly includes a bidirectional teacher and three 14B student generators, creating substantial memory demands.The system uses a teacher alongside three student generators in the MoS setup.
- Training Infrastructure: FSDP2 shards parameters, gradients, and optimizer states across GPUs, while sequence parallelism distributes the video-token sequence across devices.These techniques are combined to improve training efficiency.
5 Experiments
Wonder is evaluated as an interactive video world model for image- and video-conditioned generation across diverse benchmarks and camera trajectories. It achieves strong visual quality, camera following, dynamic preservation, and long-horizon consistency in both settings.
- Experimental Setup: The experiments evaluate Wonder in image-to-video and video-to-video settings against recent interactive world models using diverse scenes and user-specified camera trajectories.The image-to-video benchmark contains 1,000 images, while the video-to-video benchmark contains 500 dynamic-scene videos.
- Image-to-Video Results: Wonder achieves the best image-to-video overall visual quality, with an average score of 0.8558 and the highest imaging-quality score of 0.7113.The results report stronger scene synthesis and fewer visual artifacts, although some baselines score higher on individual temporal metrics.
- Image-to-Video Results: Wonder accurately follows camera instructions while preserving scene structure, generating semantically consistent unseen regions and more faithful revisits over long-horizon exploration.Qualitative comparisons indicate that most prior methods fail to reconstruct previously observed regions faithfully or suffer substantial visual degradation.
- Video-to-Video Results: Wonder improves video-to-video average visual quality from 0.8374 to 0.8527 compared with Inspatio-World while preserving dynamic input content under controlled camera motion.Improvements are reported across imaging quality, aesthetic quality, motion smoothness, and temporal flickering.
- Video-to-Video Results: Wonder reduces translational RPE from 0.0436 to 0.0187 and rotational RPE from 0.2470 to 0.1119, demonstrating stronger camera-following accuracy than Inspatio-World.The video-to-video results also report better dynamic preservation, streaming capability, and stable long-horizon visual consistency.
6 Conclusion
Wonder is a real-time, camera-controllable video world model that turns a single image into an explorable world and re-renders observed videos along user-specified camera trajectories. Its unified system design addresses controllability, persistence, and efficiency through spatially aligned camera cues and sparse full-fidelity memory.
- Conclusion: Wonder transforms a single image into an explorable world and re-renders observed video along user-specified camera trajectories.The system preserves the observed video's appearance and dynamics.
- Conclusion: A unified system design jointly addresses controllability, persistence, and efficiency.The conclusion identifies these as the central capabilities addressed by Wonder.
- Conclusion: Wonder supports navigation into videos through camera-controllable re-rendering.Figure 11 depicts navigation into videos.