Source-linked AI summary
WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories
Yisu Zhang, Chenjie Cao, Tengfei Wang, Xuhui Zuo, Junta Wu, Jianke Zhu, Chunchao Guo
TL;DR
Camera-guided video diffusion still struggles to produce consistent 3D reconstructions across distinct camera trajectories. WorldStereo addresses this with global-geometric and spatial-stereo memories, and experiments report strong performance across camera-guided generation and 3D reconstruction benchmarks. The framework also generalizes to perspective- and panorama-based scene-generation tasks.
Problem
Camera-guided VDMs have limited camera controllability and inconsistent content across trajectories, making consistent 3D scene reconstruction challenging.
Method
WorldStereo augments a camera-guided VDM with incrementally updated point-cloud global-geometric memory and correspondence-guided spatial-stereo memory.
Results
WorldStereo demonstrates strong performance across camera-guided video-generation and 3D-reconstruction benchmarks, including superior camera-motion accuracy and higher-quality video generation.
Takeaways & Limitations
WorldStereo supports multi-trajectory-consistent video generation and diverse perspective- and panorama-based scene-generation tasks with high-fidelity 3D results.
Takeaways & Limitations
Evaluations use methods at their predefined resolutions and frame numbers, with Uni3C and WorldStereo matched at 512p and 81 frames for fairness.
Abstract
from arXiv · showhide
Recent advances in foundational Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo, a novel framework that bridges camera-guided video generation and 3D reconstruction via two dedicated geometric memory modules. Formally, the global-geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo memory constrains the model's attention receptive fields with 3D correspondence to focus on fine-grained details from the memory bank. These components enable WorldStereo to generate multi-view-consistent videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided video generation and 3D reconstruction benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released.
1. Introduction
WorldStereo addresses inconsistent 3D reconstruction from camera-guided video diffusion by adding geometry-aware memories that support coherent generation across trajectories. It combines global structure and fine detail preservation while retaining pretrained-model usability and evaluates both video generation and 3D reconstruction.
- WorldStereo bridges camera-guided video generation and 3D reconstruction by enabling consistent multi-trajectory video generation with geometry-aware memories.
- Global-geometric and spatial-stereo memories respectively memorize coarse structures and fine-grained details for coherent synthesis across diverse trajectories.
- The framework largely preserves pretrained VDM generalization and usability while producing impressive 3D reconstruction results.
- WorldStereo extends Uni3C with an incrementally updated Global-Geometric Memory and a Spatial-Stereo Memory that establishes explicit geometric correspondence between generated and retrieved views.
- Experiments cover in-domain and out-of-distribution benchmarks, with reported gains in camera-motion accuracy and video-generation quality.
- The paper introduces a customized 3D reconstruction benchmark for evaluating camera-guided VDM outputs.
2. Related Work
Related work spans camera-guided video diffusion, feed-forward 3D reconstruction, and iterative scene-generation workflows. These approaches motivate WorldStereo’s focus on controllable viewpoints, geometric reconstruction, and consistency beyond observed views.
- Camera-Guided Video Generation: Camera-guided VDMs use implicit tuning, action or discrete controls, explicit camera poses, Plücker rays, dynamic representations, and conditional priors.
- Camera-Guided Video Generation: Feed-forward 3D Gaussian Splatting enables fast optimization-free representations, but reconstruction quality degrades when viewpoints extend beyond observed images.
- 3D Scene Generation: Iterative warp-and-inpaint is identified as a popular scene-generation workflow combining depth estimation with image synthesis.
3. Method
WorldStereo generates consistent multi-view videos from image-conditioned camera trajectories using a memory bank, a point-cloud cache, and two geometry-aware memory branches. SSM adds explicit 3D correspondence for detail preservation, while modified DMD accelerates inference.
- Overview: WorldStereo follows a generate-first-then-reconstruct pipeline that produces multiple video sequences with consistent 3D geometry and textures.
- Preliminaries: Uni3C provides camera and point-cloud guidance, with point clouds derived from reference images through back-projected monocular depth.
- Memory Bank & 3D Cache: The 2D memory bank stores temporally downsampled generated frames and initial views for retrieving spatially similar references.
- Memory Bank & 3D Cache: The 3D cache incrementally reconstructs point clouds from memory-bank images and supplies global geometric information for subsequent generation.
- Global-Geometric Memory: GGM iteratively updates point-cloud conditions as global 3D priors for consistent videos, continuation, and panorama-based generation.
- Spatial-Stereo Memory: SSM retrieves reference views, stitches target-reference latent features, and adds pointmap latents encoding their 3D correspondence.
- Spatial-Stereo Memory: SSM addresses GGM’s limited fine-grained detail preservation, while its training uses temporally misaligned multiview pairs with shuffled and masked references.
- Acceleration via DMD: Modified DMD distills WorldStereo into a four-step DiT, using a frozen real score and trainable generator and fake score functions.
4. Experiments
Experiments evaluate WorldStereo for camera control, visual quality, single-view 3D reconstruction, and panorama generation. The results show that its geometric memories improve consistency and reconstruction quality, while the distilled variant retains strong performance with faster inference.
- Evaluation Settings: The OOD benchmark combines 100 WorldScore images with complex translation, rotation, and panning trajectories to test camera precision.WorldMirror extracts predicted camera trajectories for evaluation.
- Camera Control and Visual Quality: WorldStereo∗ outperforms competitors in camera control and visual quality, while memory-based training preserves generalization and visual quality.GGM improves overall video quality; SSM slightly reduces performance but recovers fine-grained details.
- Single-View Reconstruction Benchmark: The 3D reconstruction benchmark uses single images, four predefined trajectories, reconstructed point clouds, and novel-view generation for evaluation.It includes Tanks-and-Temples and MipNeRF360, with point-cloud precision, recall, and AUC metrics.
- Quantitative Results: WorldStereo’s full model substantially improves reconstruction and camera precision over the memory-free baseline, while WorldStereo-DMD retains strong reconstruction consistency with acceleration.The benchmark evaluates the quality and consistency of generated multi-trajectory videos through 3D reconstruction.
- Qualitative Comparison: Qualitatively, WorldStereo produces complete, precise point clouds and consistent novel views, whereas SEVA and Gen3C produce distorted, blurry, ambiguous, or incomplete reconstructions.The benchmark emphasizes foreground structures and requires consistency across predefined trajectories.
- Memory Ablations and Panorama Generation: Ablations show that no-memory generation hallucinates objects, GGM restores coarse structural consistency, and SSM with pointmap guidance recovers fine-grained correspondence.The pointmap helps SSM focus on correct matching regions, while panorama generation uses a 27-frame memory bank and a 3D cache.
5. Conclusion
WorldStereo is a camera-guided video generation framework for 3D reconstruction that uses complementary geometric memories to maintain coherent structures and fine-grained details. It also supports accelerated inference and diverse scene-generation settings, including 3D panoramas.
- Conclusion: WorldStereo generates multiple consistent videos through Global-Geometry Memory and Spatial-Stereo Memory for camera-guided 3D reconstruction.GGM updates a point-cloud 3D cache, while SSM learns coherence between generated and retrieved views through 3D correspondence.
- Conclusion: The framework combines coarse structural consistency from GGM with fine-grained detail preservation from SSM.These mechanisms are integrated into a camera-guided video generation framework tailored for reconstruction.
- Conclusion: WorldStereo uses DMD to accelerate inference with negligible performance drop and generalizes to object-centric, face-forward, and 3D panorama generation.The framework is reported to maintain strong performance across these scene-generation tasks.
A. Datasets
WorldStereo’s datasets span multiple camera-control and scene-generation sources, with dynamically sampled subsets used across training epochs. The dataset summary covers both training and evaluation settings.
- Datasets: The work summarizes training and evaluation datasets, with most training data publicly available and subsets sampled dynamically for each epoch.Sampling follows dataset diversity and video or trajectory quality.
- Datasets: The listed training datasets include DL3DV, Re10K, Tartainair, Map-Free-Reloc, WildRGBD, and UCo3D.Table 4 identifies these datasets as inputs to the work’s various training processes.
B. More Quantitative Ablation Studies
WorldStereo is evaluated through memory-component, camera-control, and high-resolution studies, including a benchmark designed to measure memory capabilities and image fidelity. The method improves camera-control precision and video quality, while its distilled variant substantially accelerates inference without significant qualitative degradation.
- Memory-component benchmark: The memory-component benchmark contains 100 diverse real and virtual scenes with varying environments and camera-motion complexity.It includes data from DL3DV, Map-Free-Reloc, WildRGBD, Tartanair, and UE5-rendered scenes.
- Memory-component benchmark: PSNR, SSIM, and LPIPS are used to evaluate image fidelity because the benchmark provides ground-truth video views.The benchmark also tests memory banks with temporal overlap and randomized reference-frame dropping.
- Camera control: WorldStereo achieves higher camera-control precision and improved video quality than Uni3C across the camera-control evaluation.The comparison uses samples containing depth and point clouds extracted from the first frame, without additional views or memory components.
- Camera control: 20x faster inference is achieved by WorldStereo-DMD without a significant qualitative or camera-control drop.WorldStereo-DMD is compared with Uni3C and WorldStereo on camera trajectory accuracy and resulting image quality.
- High-resolution inference: WorldStereo supports 720 × 1280 inference despite training exclusively on 480p data.Direct high-resolution inference produces images with enhanced detail, as illustrated by the resolution comparison.
C. Trajectory Settings
The trajectory study evaluates how generation order and resolution affect panoramic 3D reconstruction. It selects an orbit-first sequence because the updated memory bank provides more reliable references, while higher resolution yields sharper, more detailed images.
- Trajectory ablation: The trajectory ablation evaluates 10 panoramic scenes using a memory bank initialized with 24 panorama views.Alternative trajectory orders incrementally update the memory bank with generated views.
- Resolution: High-resolution 720×1280 generation produces sharper images with richer details than 480×768 generation.The difference is demonstrated in red-boxed regions.
- Trajectory ablation: Updated memory banks provide more reliable references than the baseline using only panoramic images.The comparison is reported through overlapping FoV scores, where higher scores indicate more relevant retrieved frames.
- Trajectory order: The selected default trajectory is orbit→up→right→left, with orbit prioritized because its viewing angles are information-rich.The study states that left and right rotations are more critical to 3D reconstruction than other trajectories.
- Trajectory parameters: The upward, leftward, and rightward rotations use angles of 45◦, 90◦, and 90◦, respectively.The rotation-center distance is the scene’s median depth, and the orbital radius is 0.3 times that depth; face-forwarding scenes may use reduced angles.
D. Details of Data Curation
Training data for the geometric memories is curated from aligned multi-view point clouds and temporally overlapping reference-target pairs. Random reference dropout broadens training conditions toward unordered and partially missing reference sets.
- GGM curation: Global-Geometric Memory training starts from an initial-frame point cloud and adds 1 to 4 randomly sampled aligned frames.Additional point clouds are generated from depth information and aligned to the initial coordinate system.
- SSM curation: Spatial-Stereo Memory training pairs use reference and target videos with 30% to 90% temporal overlap and equal frame counts.This pairing procedure follows the SSM training setup described in the paper.
- SSM curation: SSM training omits the entire reference condition with 10% probability and otherwise drops each target-associated reference frame with 30% probability.The resulting unordered reference set is intended to more closely match real-world application scenarios.