Source-linked AI summary
Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation
Tianyu Huang, Wangguandong Zheng, Tengfei Wang, Yuhao Liu, Zhenwei Wang, Junta Wu, Jie Jiang, Hui Li, Rynson W. H. Lau, Wangmeng Zuo, Chunchao Guo
TL;DR
Generating long-range, 3D-consistent scenes that users can explore remains difficult because existing methods struggle with complex worlds, spatial consistency, and hallucinated visual content. Voyager generates world-consistent RGB-D videos and point-cloud sequences from one image and a user-defined camera path, achieving high visual fidelity and spatial coherence while supporting direct reconstruction and world expansion.
Problem
Existing methods have limited ability to scale to complex, user-navigable scenes while preserving 3D consistency over long-range camera trajectories.
Method
Voyager jointly generates RGB-D videos conditioned on camera trajectories and cached point clouds, enabling autoregressive, world-consistent scene extension from a single image.
Results
Voyager achieves high visual fidelity and spatial coherence in generated videos and point clouds, with the highest score on the evaluated world-generation benchmark.
Takeaways & Limitations
Voyager supports direct 3D reconstruction and consistent expansion of explorable worlds along user-defined camera trajectories.
Abstract
from arXiv · showhide
Real-world applications like video gaming and virtual reality often demand the ability to model 3D scenes that users can explore along custom camera trajectories. While significant progress has been made in generating 3D objects from text or images, creating long-range, 3D-consistent, explorable 3D scenes remains a complex and challenging problem. In this work, we present Voyager, a novel video diffusion framework that generates world-consistent 3D point-cloud sequences from a single image with user-defined camera path. Unlike existing approaches, Voyager achieves end-to-end scene generation and reconstruction with inherent consistency across frames, eliminating the need for 3D reconstruction pipelines (e.g., structure-from-motion or multi-view stereo). Our method integrates three key components: 1) World-Consistent Video Diffusion: A unified architecture that jointly generates aligned RGB and depth video sequences, conditioned on existing world observation to ensure global coherence 2) Long-Range World Exploration: An efficient world cache with point culling and an auto-regressive inference with smooth video sampling for iterative scene extension with context-aware consistency, and 3) Scalable Data Engine: A video reconstruction pipeline that automates camera pose estimation and metric depth prediction for arbitrary videos, enabling large-scale, diverse training data curation without manual 3D annotations. Collectively, these designs result in a clear improvement over existing methods in visual quality and geometric accuracy, with versatile applications.
1 Introduction
Voyager addresses the challenges of constructing long-range, explorable, world-consistent 3D scenes by jointly generating RGB-D videos from a single image and user-specified camera trajectories. Its world cache, depth-aware diffusion, and scalable video reconstruction data engine support coherent scene expansion and direct 3D reconstruction.
- Motivation: Existing novel-view synthesis and video-generation methods produce visually appealing content but face long-range spatial inconsistency challenges.These methods are presented as alternative paradigms for world modeling.
- Voyager framework: Voyager jointly generates long-range, world-consistent RGB-D videos from a single image and user-specified camera trajectories.Its world-consistent video diffusion uses expandable world caching to maintain spatial consistency and avoid visual hallucination.
- World caching: The closed-loop world cache unprojects the input image into 3D, projects partial RGB-D observations into target views, and updates from generated frames.This mechanism supports arbitrary camera trajectories while maintaining geometric coherence.
- Depth-aware generation: Depth-aware joint RGB and depth generation improves 3D consistency and enables direct scene reconstruction without structure-from-motion.Depth serves as a spatial prior, while aligned RGB-depth sequences support reconstruction without additional 3D reconstruction steps.
- Scalable training and exploration: Voyager combines auto-regressive video sampling, efficient world caching, and a scalable data engine that estimates camera poses and metric depth for arbitrary videos.The contributions include infinite world exploration and a training-data pipeline with over 100,000 training examples.
- Results: Experiments show more coherent geometry, direct 3D reconstruction, and infinite world expansion while preserving the original spatial layout.The paper also explores 3D generation, video transfer, and depth estimation applications.
2 Related Work
Related work spans camera-controllable generation and long-video extension, but existing approaches face limitations with single-view inputs, scalable video length, or memory retention. Camera-controllable methods include novel-view reconstruction and models that implicitly incorporate camera parameters.
- Camera-Controllable Generation: Novel-view synthesis generates new viewpoints through multi-view reconstruction but relies on dense viewpoints and struggles with single-view inputs.These methods are cited as one category of camera-controllable generation models.
- Camera-Controllable Generation: A second category implicitly incorporates camera parameters into the model during training.The supplied passage introduces this category but does not provide further details.
- Long-Video Generation: Long-video generation uses training-free, hierarchical, or auto-regressive strategies, but training-free and hierarchical approaches cannot scale to infinitely long videos.Auto-regressive approaches instead rely on memory caches that struggle to retain information.
3 Preliminaries of Video Diffusion Models
Video diffusion models denoise noisy latent representations while modeling temporal dynamics through 3D convolutions and attention. Voyager’s Hunyuan-Video-based model uses image-conditioned latent denoising and 3D-VAE reconstruction for video generation.
- Diffusion fundamentals: Diffusion models learn to recover clean data by reversing a process that gradually adds noise to samples from the data distribution.The reverse process denoises x_t toward x_0.
- Video diffusion: Video diffusion extends diffusion modeling to temporal dynamics using 3D convolutions and attention mechanisms.These architectures model dependencies across video frames.
- Video diffusion: Latent diffusion reduces computation by compressing videos into a low-dimensional latent space.This compression is widely used for video generation.
- Voyager’s video model: Voyager’s video model is based on Hunyuan-Video and encodes ground-truth video sequences into latents with a 3D-VAE.The input consists of a text prompt and video sequence [I_0, ..., I_T−1].
- Voyager’s video model: A full-attention DiT denoises noisy video latents, incorporates image features channel-wise, predicts velocity with mean squared error, and decodes the result through a 3D-VAE.The DiT uses dual-stream and single-stream Transformer blocks; a first-order Euler ODE solver recovers z_0 before decoding.
4 Methodology: Voyager
Voyager generates explorable, world-consistent 3D point-cloud sequences from a single image and user-defined camera trajectory by jointly modeling RGB-depth video, caching scene history, and scaling training-data annotation.
- Overview: Voyager targets long-range explorable-world generation by addressing inconsistent video extension, visual-condition hallucination, and failure to reconstruct 3D worlds from video outputs.The method is designed around a single input image and user-defined camera trajectory.
- World-Consistent Video Diffusion: Voyager conditions generation on aligned partial RGB and depth maps, then generates RGB-depth frames jointly to complete invisible regions while preserving spatial information.Depth is obtained by projecting the initial depth map and camera parameters into a point cloud; the model directly supports 3D-world reconstruction.
- World-Consistent Video Diffusion: Context-based Control blocks inject geometric conditions throughout Transformer blocks, strengthening pixel-level controllability beyond input-only concatenation.The control modules replicate the first double-stream and single-stream blocks, using zero-initialized linear layers to integrate early-stage latent features.
- Long-Range World Exploration: Voyager enables long-range or infinite generation by storing full scene history in a scalable point-cloud world cache with point culling and using smooth video sampling for continuity.This addresses the information loss caused by autoregressive methods that condition on only a few preceding frames or clips.
- Scalable Data Engine: A scalable data engine curates over 100,000 real-captured and rendered video clips and automatically annotates camera and depth information for arbitrary scene videos.Metric3D estimates metric depth ranges so source-specific depth estimates can be aligned to a standard scale.
5 Experiments
Voyager is evaluated against camera-controllable video-generation baselines for visual quality, scene reconstruction, and world generation, with additional ablations of its depth-conditioned diffusion and long-range exploration designs. Across these evaluations, Voyager produces more consistent and detailed results, while smooth sampling enables seamless transitions between extended video segments.
- Video Generation Quality: Qualitatively, Voyager generates more reasonable unseen regions while preserving content from the input view.The comparison covers SEVA, ViewCrafter, See3D, and FlexWorld.
- Video Generation Quality: Voyager outperforms all compared baselines on PSNR, SSIM, and LPIPS video-generation evaluation.Camera parameters and depth maps are estimated with the data-engine pipeline when ground-truth cameras are unavailable.
- Scene Reconstruction: Voyager’s scene reconstructions outperform baselines with VGGT post-hoc initialization, and improve further when initialized with Voyager’s own depth output.Voyager’s RGB-D outputs can be used directly for 3DGS reconstruction, unlike the RGB-only baseline outputs.
- World Generation: On the WorldScore benchmark, Voyager is compared with six existing methods across diverse indoor, outdoor, photorealistic, and stylized worlds.WorldScore contains 2,000 static test examples with an input image and camera trajectory, measuring controllability and generation quality.
- Ablation Studies: Ablations evaluate RGB-only training, RGB-D training, and additional control blocks for world-consistent video diffusion, alongside storage and smooth-sampling behavior for long-range exploration.Storing all points improves visual performance but uses more storage, while smooth sampling ensures seamless transitions between consecutive segments.
6 Application
Voyager supports 3D-related applications through depth-fused video generation, including long-range video generation with controllable camera trajectories and spatial consistency. It also generates correct object content with more realistic visual effects in a car-and-tent composition.
- Long Video Generation: Voyager generates long videos using efficient world caching and smooth video sampling, supporting different camera trajectories across clips.The example demonstrates camera controllability and spatial consistency in generated video.
- 3D Object Composition: In a car-and-tent composition, Voyager generates the correct content with more realistic visual effects than the compared methods.The tent is visible through the car window in the side view.
- 3D Object Composition: For the car-and-tent example, Rodin fails to generate the tent, Trellis produces a tent with missing parts, and Hunyuan generates complete objects with inaccurate spatial relationships.Hunyuan places the tent too far from the car.
7 Conclusion
Voyager is a world-consistent video generation framework for long-range world exploration that generates spatially consistent RGB-D videos aligned with input camera trajectories. This enables direct 3D scene reconstruction and autoregressive, consistent world expansion, with high visual fidelity and strong spatial coherence.
- Contributions: Voyager presents a world-consistent video generation framework for long-range world exploration.The framework is based on an RGB-D video diffusion model.
- Contributions: The RGB-D diffusion model generates spatially consistent videos aligned with input camera trajectories, enabling direct 3D scene reconstruction.The generated videos also support autoregressive and consistent world expansion.
- Results: Voyager achieves high visual fidelity and strong spatial coherence in both generated videos and point clouds.These results are reported in experiments on the generated videos and reconstructed point clouds.
- Applications: The framework supports long-range video generation, image-to-3D generation, world-consistent video style transfer, and monocular video depth estimation.These applications are illustrated in Fig. 6.
results in Sec. D. · A Implementation Details · B World-Consistent Video Diffusion
The implementation uses a staged world-consistent video diffusion model that conditions on noisy, reference, condition, and mask latents while progressively incorporating RGB and depth information. Its final stage adds two Transformer blocks whose aggregated features are returned pixel-wise.
- results in Sec. D.: Figure 9 presents the architecture details of the world-consistent diffusion model.
- B World-Consistent Video Diffusion: The diffusion input comprises noisy latents z_t, input-image latents z_r0, condition latents ẑ_0, and a down-sampled mask m.
- B World-Consistent Video Diffusion: The reference latents z_r0 are padded with zero latents to align the temporal dimension.
- A Implementation Details: During the first training stage, only RGB-related latents are concatenated along the channel dimension before entering the diffusion model.
- A Implementation Details: Depth-related latents are injected during the second training stage, extending the model’s staged conditioning process.
- A Implementation Details: The original diffusion structure is fine-tuned during the first two stages, while two additional Transformer blocks are trained in the final stage.
- B World-Consistent Video Diffusion: Aggregated features from the final Transformer blocks are added back on a pixel-wise basis.
C Scalable Video Data Engine
Voyager’s scalable video data engine estimates camera parameters and depth for every video frame, refines depth through disparity alignment, and normalizes depths to a metric range across datasets. Compared with Flexworld, its more consistent depth estimation produces more precise warped images.
- Data quality: Compared with Flexworld, Voyager’s more consistent and accurate depth estimation produces more precise warped images.The comparison attributes the improvement to Voyager’s depth estimation versus depth rendered by 3DGS.
- Video reconstruction pipeline: VGGT estimates camera parameters and depth for all video frames, providing camera-aligned depth despite limited depth accuracy.The camera alignment enables subsequent depth refinement.
- Depth refinement: MoGE refines VGGT depth by converting depths to disparity and minimizing disparity differences with least-squares optimization.A valid non-sky-region mask is used during optimization, with scale and shift factors aligning the estimates.
- Metric normalization: Metric3D estimates the metric depth range and maps estimated depths into that range to ensure scale uniformity across datasets.The resulting metric depth is defined relative to camera extrinsics and quantiles of the depth vector.
D More Results
Additional visualizations show Voyager’s stronger 3D reconstruction initialization compared with VGGT, alongside further generation and visualization results.
- 3D Reconstruction Initialization: Voyager produces substantially better point clouds than VGGT for initializing 3D reconstruction, indicating more accurate depth estimation.The comparison is shown in Figure 12.
- Additional Results: Additional generation results are presented in Figures 13 and 14.Figure 13 provides more results, while Figure 14 provides more visualization results.