Source-linked AI summary
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, Yonghong Tian
TL;DR
ViewCrafter addresses the limited applicability of neural 3D reconstruction when only single or sparse images are available. It combines point-cloud priors with a point-conditioned video diffusion model, iterative view synthesis, and camera trajectory planning to generate controlled, high-fidelity novel views. Experiments across diverse datasets report superior image quality and pose accuracy, with applications in 3D-GS rendering and text-to-3D generation.
Problem
Dense multi-view dependence limits neural 3D reconstruction for generic-scene novel-view synthesis from single or sparse observations.
Method
ViewCrafter conditions video diffusion on point-cloud renders and reference images, then iteratively expands views and reconstructed geometry using adaptive camera trajectories.
Results
Across Tanks-and-Temples, RealEstate10K, and CO3D, ViewCrafter outperforms baselines in zero-shot novel-view image quality and pose accuracy and surpasses prior state of the art in sparse-view 3D-GS reconstruction.
Takeaways & Limitations
The framework supports high-fidelity, consistent novel views with precise camera control and facilitates real-time 3D-GS rendering and scene-level text-to-3D generation.
Takeaways & Limitations
Sparse or single inputs can produce point-cloud missing regions, occlusions, and artifacts, while per-view inpainting can leave occlusion-region content incomplete.
Abstract
from arXiv · showhide
Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. In this work, we propose \textbf{ViewCrafter}, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images with the prior of video diffusion model. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames with precise camera pose control. To further enlarge the generation range of novel views, we tailored an iterative view synthesis strategy together with a camera trajectory planning algorithm to progressively extend the 3D clues and the areas covered by the novel views. With ViewCrafter, we can facilitate various applications, such as immersive experiences with real-time rendering by efficiently optimizing a 3D-GS representation using the reconstructed 3D points and the generated novel views, and scene-level text-to-3D generation for more imaginative content creation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in synthesizing high-fidelity and consistent novel views.
1 INTRODUCTION
ViewCrafter targets high-fidelity, consistent novel-view synthesis for generic scenes from single or sparse images, combining video diffusion generation with explicit point-cloud geometry for camera control. Iterative synthesis and adaptive trajectory planning extend the reconstructed scene coverage and support downstream 3D applications.
- The method combines video diffusion models with point-cloud representations to use generative priors alongside explicit 3D information.Video diffusion supplies generation capabilities, while point clouds provide coarse scene geometry and camera-view control.
- ViewCrafter supports real-time immersive rendering by efficiently optimizing a 3D-GS representation from reconstructed points and synthesized views, and enables scene-level text-to-3D generation.
- Across Tanks-and-Temples, RealEstate10K, and CO3D, the method outperforms baselines in zero-shot novel-view image quality and pose accuracy and surpasses prior state of the art in sparse-view 3D-GS reconstruction.
- ViewCrafter synthesizes high-fidelity novel-view sequences of generic scenes from single or sparse images while maintaining precise camera-pose control.
- Iterative view synthesis and content-adaptive camera trajectory planning progressively expand novel-view coverage and the reconstructed point cloud for long-range, large-area synthesis.
2 RELATED WORK
Prior approaches use regression, optimization, warping, or conditioning signals for novel-view generation but remain limited by domain specificity, scene-specific optimization, artifacts, or incomplete camera control. ViewCrafter uses explicit point clouds to provide precise camera control in video generation.
- Regression-based methods: Regression-based methods generate sparse- or single-image novel views but are often category-specific because of limited representation capabilities.
- Diffusion-guided optimization: Optimization-based approaches supervised by text-to-image diffusion models require scene-specific optimization, compromising their generalization capabilities.
- Warping and inpainting: Depth-warping methods refined with text-to-image diffusion often produce artifacts and unrealistic contexts in inpainted regions.
- Controllable video generation: Video-generation methods support controls such as RGB images, depth, trajectories, and semantic maps, but ViewCrafter introduces explicit point-cloud conditioning for precise camera control.
3 METHOD
ViewCrafter combines point-cloud geometry with video diffusion to synthesize controlled, consistent novel views from single or sparse images. An iterative synthesis and camera-planning procedure expands scene coverage and supports downstream 3D-GS optimization.
- 3.2 Point Cloud Reconstruction from Single or Sparse Images: ViewCrafter reconstructs point clouds and camera parameters from reference images using a dense stereo model such as DUSt3R.With a single image, the input is duplicated to form a pair; multiple inputs can undergo global point-map alignment.
- 3.2 Point Cloud Reconstruction from Single or Sparse Images: Because sparse or single inputs produce incomplete point clouds with occlusions and artifacts, video diffusion supplies higher-fidelity content from imperfect geometric cues.The point cloud provides coarse 3D information, while diffusion-based generation addresses missing regions and reduced visual fidelity.
- 3.3 Rendering High-fidelity Novel Views with Video Diffusion Models: The point-conditioned video diffusion model generates high-quality novel-view sequences from point-cloud renders and reference images along a specified camera trajectory.The model learns a conditional distribution over novel views and incorporates point-cloud renders into its video denoising architecture.
- 3.4 Iterative View Synthesis and Camera Trajectory Planning: Iterative view synthesis progressively extends the reconstructed point cloud and view range to avoid the instability and computational cost of generating long videos directly.The procedure repeatedly generates views, back-projects them into the current point cloud, and updates the camera pose and scene representation.
- 3.4 Iterative View Synthesis and Camera Trajectory Planning: Content-adaptive camera trajectory planning selects candidate poses that reveal missing or occluded regions while avoiding excessively large holes.The algorithm samples candidate poses around the current pose, renders candidate masks, selects the next best view, and interpolates a path for synthesis.
4 EXPERIMENTS
ViewCrafter is evaluated for zero-shot novel view synthesis and sparse-view scene reconstruction across multiple real-world datasets, with additional robustness and ablation studies. It outperforms comparison methods in image quality, pose accuracy, and sparse-view reconstruction, while handling imperfect point-cloud conditions.
- Evaluation Setup: Evaluation covers zero-shot novel view synthesis on CO3D, RealEstate10K, and Tanks-and-Temples, using image-quality and pose-accuracy metrics.Image quality is assessed with PSNR, SSIM, LPIPS, and FID; pose accuracy uses rotation and translation distances.
- Zero-shot Novel View Synthesis: The qualitative comparison places reference images at the left and ground-truth novel views at the right across three datasets.The comparison includes ViewCrafter, LucidDreamer, ZeroNVS, and MotionCtrl.
- Zero-shot Novel View Synthesis: ViewCrafter outperforms zero-shot baselines in image quality and pose accuracy, with lower LPIPS, improved FID, and reduced Rdist and Tdist.These results indicate more perceptually accurate images and more accurate camera-pose control.
- Scene Reconstruction: Sparse-view scene reconstruction uses 2 ground-truth training images per Tanks-and-Temples scene and evaluates rendered novel views with PSNR, SSIM, and LPIPS.The benchmark contains 6 scenes and 12 evaluation views per scene.
- Scene Reconstruction: ViewCrafter consistently outperforms comparison baselines across all scene-reconstruction metrics in the 2-image sparse-view setting.The qualitative comparison evaluates rendered novel views against the ground-truth novel view.
- Robustness and Ablations: ViewCrafter remains robust to occlusions, missing regions, and geometric distortions in point-cloud renders, filling holes and correcting inaccurate geometry in generated views.Its camera trajectory planning also reconstructs occlusion regions more completely than a predefined circular trajectory.
5 CONCLUSION AND LIMITATION
ViewCrafter combines video diffusion models with point-cloud priors for high-fidelity novel view synthesis, while extending views through iterative synthesis and adaptive camera planning. The method also explores real-time 3D-GS rendering and text-to-3D generation, but large view ranges remain challenging with limited 3D clues.
- ViewCrafter combines video diffusion models and point-cloud priors for high-fidelity and accurate novel view synthesis.
- The framework supports varied scene types and both single-image and sparse-image inputs while maintaining consistency and accuracy.
- Iterative view synthesis and adaptive camera trajectory planning facilitate long-range novel view synthesis and automatic trajectory generation.
- Large view ranges remain challenging when limited 3D clues cannot reveal unseen regions, such as producing a front view from only a back view.