Source-linked AI summary
SpatialCrafter: Single Image World Modeling with Generative 3D Proxies
Chuan Fang, Lingteng Qiu, Yixun Liang, Rui Chen, Kunming Luo, Zhaohua Zheng, Tongyuan Bai, Feipeng Tian, Zilong Dong, Zihan Zhou, Ping Tan
TL;DR
Existing image-to-scene methods use incomplete spatial conditioning, limiting long-range consistency during exploration. SpatialCrafter instead generates a global 3D proxy and refines it with a video diffusion model, achieving state-of-the-art visual quality and geometric consistency under extreme viewpoint changes.
Problem
Existing image-to-scene methods rely on incomplete 3D proxies or 2D images, making them susceptible to long-term drift and stochastic hallucinations during exploration.
Method
SpatialCrafter uses a two-stage global 3D proxy generator and Generative Deferred Refiner, with PaSS, geometry injection, proxy-aware corruption, and a 115K-scene dataset.
Results
SpatialCrafter achieves state-of-the-art visual quality and geometric consistency on synthetic and real-world benchmarks, remaining robust under extreme camera motion and viewpoint changes.
Takeaways & Limitations
The framework delivers superior spatial coherence for explorable image-to-scene generation under rapid camera motion and extreme viewpoint changes.
Takeaways & Limitations
Incremental scene-exploration methods incur considerable computational overhead from expanding 2D memory banks and real-time point-cloud stitching.
Abstract
from arXiv · showhide
Explorable image-to-scene generation is essential for applications in gaming, robotics, and virtual reality. Existing methods based on video diffusion model (VDM) commonly rely on incomplete conditioning signals such as sparse point clouds or 2D panoramas, leading to stochastic hallucinations, long-term drifts and suboptimal 3D consistency. We present SpatialCrafter, a novel two-stage framework that addresses these issues by introducing a global 3D proxy for high-fidelity image-to-scene generation. Specifically, we decompose the generation process into global proxy generation and appearance refinement. For proxy generation, we propose a Point-anchored Sparse Structure~(PaSS) Flow module that predicts a spatially aligned and geometrically consistent 3D proxy. For appearance refinement, we re-frame the VDM as a Generative Deferred Refiner which synthesizes high-frequency photorealistic details upon proxy-defined scene geometry. To better integrate the proxy with the pre-trained VDM, we introduce Parallel Geometry Injection and Proxy-Aware Corruption training strategies, which improve robustness to proxy artifacts without disrupting the pretrained generative manifold. Furthermore, as no suitable dataset exists for this explorable scene generation task, we construct a new large-scale dataset of 115K scenes. To the best of our knowledge, it is the first hybrid dataset for image-to-scene generation. Extensive experiments on both synthetic and real-world datasets show that SpatialCrafter outperforms state-of-the-art methods, mitigates long-term drift, and remains robust and consistent under rapid camera motion and extreme viewpoint changes. Code, models, and the newly constructed dataset will be publicly released. See more at https://fangchuan.github.io/SpatialCrafter/.
1 Introduction
SpatialCrafter addresses the spatial inconsistency and hallucination caused by incomplete 3D conditioning in video-diffusion world models. It introduces a two-stage global-proxy pipeline, alignment and refinement mechanisms, and a 115K-scene hybrid dataset.
- Motivation and limitations: Prior video diffusion world models lack explicit 3D supervision, while reconstructive proxies remain incomplete and force stochastic hallucination in unseen regions.Images-as-memory methods [Song et al. 2025; Xiao et al. 2025; Yu et al. 2025a] and explicit-proxy methods [Li et al. 2025a; Ren et al. 2025; Wu et al. 2025c; Yang et al. 2025a; Yu et al. 2024b] address parts of this problem but do not eliminate the underlying incompleteness.
- Method: The framework generates a dense, continuous global 3D proxy from one image, then repurposes a video diffusion model for appearance refinement.This two-stage design provides reliable scene guidance beyond the input views.
- Method: PaSS Flow Matching anchors sparse proxy generation to input-view geometry, while Parallel Geometry Injection and Proxy-Aware Corruption improve robust video refinement.These mechanisms address scene-level spatial misalignment and proxy artifacts that can break cross-view coherence.
- Dataset: A scalable data engine combines synthetic indoor renderings with filtered real-world reconstructions to produce approximately 115K high-fidelity scenes.The resulting hybrid dataset is presented as the first large-scale image-to-scene dataset with precise geometric annotations across diverse indoor and outdoor environments.
- Contributions: SpatialCrafter achieves state-of-the-art visual quality and geometric consistency, remaining spatially coherent under extreme camera motion and viewpoint changes where prior methods fail.These results hold across synthetic and real-world benchmarks.
2 Related Work
Prior scene-generation methods optimize 3D representations, generate multi-view imagery, or use memory-augmented video diffusion, but they face consistency, drift, occlusion, and computational limitations. Explicit 3D proxies reduce 2D-memory retrieval overhead, while diffusion refinement enhances imperfect reconstructions.
- Scene-Level 3D Generation: Scene-level SDS methods optimize NeRF or 3DGS representations but often suffer cross-view semantic inconsistency because they lack explicit multi-view constraints.Early approaches include Cohen-Bar et al. [2023], Sargent et al. [2024], and Yang et al. [2024].
- 2D Image Memory: Incremental 2D-memory methods improve short-term temporal consistency but accumulate depth and alignment errors, causing scale drift, loop-closure failures, and high maintenance overhead.Their expanding memory banks and real-time point-cloud stitching also impose considerable computational cost.
- 3D Proxy: Explicit 3D-proxy methods eliminate 2D-memory retrieval overhead, but static reconstruction leaves unseen regions permanently missing while dynamic variants must address proxy incompleteness.This line includes ViewCrafter [Yu et al. 2024b] and GEN3C [Ren et al. 2025].
- Panorama Proxy: Panorama proxies complete a single input view into 360° imagery and lift it into point clouds, yet their single-optical-center representation struggles with occlusion and lacks true 3D parallax.Matrix-3D [Yang et al. 2025a] and One2Scene [Wang et al. 2026] follow this approach.
- 3D-Aware Diffusion Refinement: Diffusion-based refinement methods render coarse 3D representations and enhance them with 2D diffusion priors, including DiFix3D+ [Wu et al. 2025d] and related approaches.This refinement family also includes VideoFrom3D [Kim et al. 2025], GenFusion [Wu et al. 2025b], and Artifixer [De Lutio et al. 2026].
3 Preliminaries
The preliminaries describe latent 3D diffusion for generating global proxies and latent video diffusion for computationally efficient video generation. Both frameworks use conditional flow matching to transport Gaussian noise toward target latent representations.
- Latent 3D Diffusion Models: Latent 3D diffusion builds on TRELLIS [Xiang et al. 2025], combining sparse 3D geometry with multi-view visual features through a two-stage SLAT representation.The coarse stage generates a sparse voxel field, while refinement predicts per-voxel latent features.
- Latent 3D Diffusion Models: A 3D Gaussian Splatting decoder [Kerbl et al. 2023] converts the complete SLAT into a 3D proxy, with Rectified Flow Transformers [Liu et al. 2022] conditioned on DINO image embeddings.The models use conditional flow matching [Lipman et al. 2022] to transport Gaussian noise to the target SLAT.
- Latent Video Diffusion Models: Latent video diffusion models operate in compressed spatio-temporal latents to reduce the computational burden of high-dimensional video generation.A causal VAE encodes videos into compact latents and reconstructs them through a video decoder.
- Latent Video Diffusion Models: Video generation also uses conditional flow matching, with a DiT network conditioned on text guidance predicting the vector field from Gaussian noise to the target video latent.In Wan 2.1 [Wan et al. 2025], the first frame is encoded independently, while later frames use 4× temporal and 8× spatial downsampling into 16-channel latents.
4 Methodology
SpatialCrafter decomposes single-image world modeling into aligned 3D proxy generation and trajectory-conditioned appearance refinement. Its proxy-aware design and approximately 115K-scene data engine support temporally and spatially consistent RGB-D video synthesis.
- 4 Methodology: Given a reference image and camera trajectory, SpatialCrafter synthesizes temporally and spatially consistent RGB-D frames conditioned on each camera pose.Each output frame contains a predicted RGB image and depth map.
- 4 Methodology: The two-stage framework first constructs a global 3D proxy, then refines its trajectory renderings into photorealistic RGB-D video with a Generative Deferred Refiner.The proxy is generated from the input image, rendered along the camera trajectory, and used as an explicit structural and geometric prior.
- 4.1 Global 3D Proxy Generation: PaSS Flow anchors sparse structure from the input image and predicted depth, recasting scene generation as structurally guided sparse voxel completion for geometric alignment.The aligned sparse structure is processed with SLAT Flow and decoded into the global 3D proxy.
- 4.2 Generative Deferred Refiner: Parallel Geometry Injection integrates coarse RGB-depth latents with reference and noisy video latents while freezing pretrained weights and optimizing LoRA parameters.This preserves the video model’s spatio-temporal and photorealistic priors while incorporating geometric guidance.
- 4.2 Generative Deferred Refiner: Proxy-Aware Corruption stochastically perturbs geometric conditions during training to improve robustness to inference-time proxy artifacts, including smoothed textures, floaters, and holes.Training only on curated signals can overfit and degrade robustness to these out-of-distribution artifacts.
- 4.3 Dataset Construction: A scalable data engine constructs approximately 115K 3D scenes by bridging synthetic indoor renderings and real-world video captures for training both stages.Existing benchmarks are described as limited in scale and scene diversity.
5 Experiments
SpatialCrafter is evaluated on synthetic and real-world benchmarks against image-memory and reconstructive-3D baselines, consistently achieving superior performance and robustness under demanding camera motion. Ablations further show that PaSS-Flow, proxy-based RGB-D conditioning, and Proxy-Aware Corruption each improve spatial consistency and novel-view synthesis.
- Experimental Setup: Evaluation uses 81-frame single-image generation with camera trajectories on SpatialGen-Video, RealEstate10K, and DL3DV, measuring visual quality and geometric consistency from ground-truth RGB-D and trajectories.The benchmarks include 104 synthetic scenes and 100 sampled RealEstate10K test scenes, while all methods use identical input conditions and publicly available checkpoints.
- Quantitative Results: SpatialCrafter consistently outperforms baselines across all three benchmark datasets, achieving the lowest FVD (193.54) and RPE (0.093) on SpatialGen-Video.The method also significantly leads on RealEstate10K across all key metrics, while Table 1 reports the full quantitative comparison.
- Quantitative Results: Fine-tuning Voyager on the proposed dataset improves its FVD and PSNR, but Voyager† still trails SpatialCrafter in geometric consistency, especially under extensive camera motion.Its substantially higher RPE and RVE on SpatialGen-Video and DL3DV indicate that improved data quality does not eliminate the limitations of reconstructive proxies.
- Qualitative Results: Under long-range trajectories and extreme viewpoint changes, SpatialCrafter remains structurally consistent while Voyager collapses, GEN3C and ViewCrafter distort content, and DFoT and GeometryForcing produce misaligned or nearly static outputs.These results indicate that reconstructive 3D proxies and 2D image memories are vulnerable to long-term drift and stochastic hallucinations, whereas a global complete proxy supports robust generation.
- Effectiveness of PaSS-Flow: PaSS-Flow produces structurally aligned coarse renderings from point anchors, whereas removing it causes spatial misalignment that degrades downstream refinement.The ablation is conducted on the SpatialGen-Video test set and shows that point anchoring supplies reliable geometric signals for high-fidelity refinement.
- Generative Deferred Refiner Configurations: Refinement ablations show that proxy renderings improve spatial consistency over camera-only conditioning, depth further enhances 3D consistency, and Proxy-Aware Corruption provides an additional gain.The full RGB-D model with PAC achieves the most robust and photorealistic novel-view synthesis across all reported metrics, with qualitative results confirming reduced drift and inconsistency.
6 Conclusion
SpatialCrafter is presented as a two-stage framework for explorable image-to-scene generation that uses a global, complete 3D proxy to address long-term drift and stochastic hallucinations caused by incomplete spatial memory. The work also develops a scalable data engine to address the scarcity of high-quality training data.
- 6 Conclusion: SpatialCrafter introduces a two-stage framework for explorable image-to-scene generation.
- 6 Conclusion: A global, complete 3D proxy addresses long-term drift and stochastic hallucinations associated with incomplete 3D proxies or 2D images as spatial memory.
- 6 Conclusion: A scalable data engine is developed to address the severe scarcity of high-quality training data.
Generated RGB-D Video
This section presents additional visual results generated by SpatialCrafter.
- Generated RGB-D Video: Figure 8 shows more visual results of SpatialCrafter.