Source-linked AI summary
UniWorld-View: Large-Baseline View Synthesis via Video Diffusion Models
Haiyang Zhou, Wangbo Yu, Chaoran Feng, Xunyu Zhou, Yonghong Tian, Li Yuan
TL;DR
Large-baseline novel view synthesis from sparse monocular inputs remains difficult because reconstruction methods degrade under sparse views and generative methods lack sufficiently explicit geometric guidance. UniWorld-View combines occlusion-aware point-cloud rendering with dual-stream video diffusion to improve controllability and geometric consistency. It achieves strong results on WorldScore and zero-shot NVS benchmarks and supports multiview video generation for dynamic 3DGS reconstruction.
Problem
Sparse monocular inputs make photorealistic, geometrically consistent, precisely controlled large-baseline view synthesis difficult, while existing methods provide limited explicit occlusion and geometric handling.
Method
UniWorld-View combines occlusion-aware point-cloud rendering, dual-stream conditioning on renders and source inputs, and hybrid training across dynamic monocular and static multiview data.
Results
UniWorld-View consistently outperforms existing methods in controllability, geometric consistency, and visual quality on WorldScore and zero-shot NVS benchmarks.
Takeaways & Limitations
The framework generates synchronized multiview videos that can provide supervision for downstream dynamic 3DGS reconstruction.
Takeaways & Limitations
Camera-pose-conditioned generative methods remain limited in precise and consistent camera control when they lack explicit 3D modeling.
Abstract
from arXiv · showhide
The abundance of casually captured monocular videos and images on social media provides a valuable source for immersive content creation, where generating novel views from such sparse observations can greatly enhance user experiences. However, producing photorealistic and geometrically consistent views with precise camera control remains challenging when input coverage is extremely limited. Reconstruction-based approaches such as NeRF and 3D Gaussian Splatting (3DGS) deteriorate severely under sparse inputs and fail to explicitly handle occlusions. Generative methods ease data requirements but still struggle with large-baseline view synthesis due to inaccurate or implicit geometric guidance. To overcome these limitations, we introduce UniWorld-View, a unified framework for controllable large-baseline novel view synthesis from monocular inputs. UniWorld-View integrates explicit 3D guidance with generative diffusion modeling to enable precise camera control and geometrically consistent view generation. The geometric guidance is obtained through an occlusion-aware point cloud rendering strategy that resolves visibility ambiguities and provides accurate priors for diffusion-based synthesis. By coupling this rendering strategy with powerful video diffusion backbones, UniWorld-View achieves high-fidelity novel view generation even under extreme camera motions and wide-baseline changes, and can further provide multi-view videos for downstream dynamic 3DGS reconstruction. Experiments on the WorldScore benchmark and zero-shot NVS benchmarks demonstrate the effectiveness of UniWorld-View in controllability, geometric consistency, and visual fidelity.
1 Introduction
UniWorld-View addresses large-baseline novel view synthesis from monocular inputs by combining explicit geometric guidance with video diffusion. Its occlusion-aware rendering, dual-stream conditioning, and hybrid training support controllable, geometrically consistent, high-quality generation.
- Challenges: Sparse-view NeRF and 3DGS pipelines degrade under large-baseline shifts, producing occlusion artifacts and geometric distortions.These reconstruction methods also rely heavily on dense multi-view captures and per-scene optimization.
- Challenges: Camera-pose-conditioned diffusion methods improve generalization but lack explicit 3D modeling, limiting precise and consistent camera control.Their poses function as auxiliary conditions rather than explicit geometric guidance.
- Method: UniWorld-View uses occlusion-aware point cloud rendering with triple reprojection and normal-based visibility checking to provide accurate geometric cues.The method estimates camera-dependent occlusion masks and suppresses back-facing points.
- Method: A dual-stream conditional video diffusion model fuses point cloud renders with source inputs, while hybrid training combines dynamic monocular videos and static multiview datasets.These components target cross-view consistency and robustness across diverse scenes.
- Results: Experiments on WorldScore and zero-shot NVS benchmarks show improvements in controllability, geometric consistency, and visual quality over existing approaches.The framework also generates spatio-temporally consistent multiview videos for dynamic 3DGS reconstruction.
2 Related Work
Related work progresses from explicit neural scene representations and monocular depth-based synthesis toward diffusion models conditioned on camera pose or geometry. UniWorld-View follows this trajectory while combining dynamic point-cloud guidance with video diffusion for broader view synthesis.
- Explicit representations: NeRF and 3DGS have substantially advanced static-scene novel view synthesis through neural scene representations.
- Dynamic reconstruction: Four-dimensional view synthesis commonly reconstructs dynamic representations from synchronized multiview videos, which are difficult for typical users to obtain.Monocular-video approaches evolved from depth-based warping toward learned occlusion reasoning and neural dynamic representations.
- Monocular synthesis: Early monocular depth-and-inpainting methods were limited by category-specific domains and often produced artifacts because of restricted representation capacity.
- Diffusion synthesis: Camera-pose-conditioned diffusion expanded from object-level synthesis to general scenes, while other methods used ray maps or related geometric conditions.
- UniWorld-View: UniWorld-View processes source videos into dynamic point clouds, renders desired camera trajectories, and jointly conditions video diffusion on renders and source video.The framework targets high-fidelity outputs aligned with specified trajectories and consistent with the source video.
- Explicit geometry: Recent methods incorporate tracked points, learned features, depth warping, or point-cloud renders as explicit conditions for diffusion-based view synthesis.However, existing point-cloud rendering strategies can produce ambiguous visibility.
3 Method
UniWorld-View combines explicit point-cloud geometry with video diffusion to synthesize controllable, geometrically consistent views from monocular inputs. Its occlusion-aware rendering removes large-baseline visibility ambiguities before diffusion-based generation.
- Point-cloud conditioning: The pipeline lifts monocular inputs into point clouds and renders target-trajectory views, depth maps, and validity masks as geometric conditioning.Source depth and camera intrinsics are back-projected into a point cloud, which is then projected from target viewpoints.
- Visibility ambiguities: Naive point-cloud renders can produce foreground-background tearing and erroneous back-face projections that mislead large-baseline synthesis.These ambiguities become severe when viewpoint shifts are large.
- Occlusion disambiguation: Triple reprojection identifies occlusion regions by rendering target views, reprojecting them to the source trajectory, and accumulating the resulting visibility masks.The accumulated mask ensures the final render is free from foreground-background tearing.
- Visibility correction: Normal-based visibility correction filters points whose surface normals do not sufficiently align with the target viewing direction.The front-facing mask uses a predefined normal-view alignment threshold α, then combines with cumulative visibility.
- Diffusion conditioning: The combined rendering strategy removes tearing artifacts and spurious back-face projections, providing robust geometric guidance for the diffusion model.UniWorld-View uses dual-stream conditioning to combine point-cloud renders with source appearance and decouples view transformation from content generation for training-data curation.
4 Experiments
UniWorld-View is evaluated on WorldScore and zero-shot novel-view benchmarks using static and dynamic settings. It shows strong controllability, visual quality, geometric consistency, and cross-dataset performance under large viewpoint changes.
- WorldScore Benchmark: UniWorld-View achieves the best overall static score on WorldScore and the highest scores on all three controllability metrics.It also performs strongly on 3D Consistency and Photometric Consistency.
- WorldScore Benchmark: UniWorld-View achieves the second-best WorldScore-Dynamic score in a two-stage dynamic-scene generation evaluation.The evaluation first generates a reference video, then uses it with the prescribed camera trajectory to generate the final video.
- Evaluation Setup: The evaluation compares UniWorld-View with See3D, GEN3C, Uni3C, and SEVA across zero-shot novel-view synthesis benchmarks.The datasets include CO3D, RealEstate10K, and DL3DV, with splits covering small- and large-viewpoint regimes.
- Qualitative Comparison: Under large viewpoint changes, existing methods often produce distorted boundaries, implausible structures, or degraded textures in newly exposed regions.UniWorld-View instead uses occlusion-aware renderings and dual-stream conditioning to preserve geometry, object contours, textures, and image details.
- Quantitative Comparison: UniWorld-View achieves the best PSNR and SSIM on CO3D, RealEstate10K, and DL3DV, plus the best LPIPS on CO3D and second-best LPIPS on DL3DV.On RealEstate10K, GEN3C and Uni3C obtain lower LPIPS, while UniWorld-View provides a more balanced overall trade-off.
5 Conclusion
UniWorld-View combines occlusion-aware point-cloud rendering with dual-stream conditional video diffusion for controllable large-baseline novel-view synthesis. Experiments demonstrate effectiveness and generalization, while synchronized multi-view videos support downstream dynamic Gaussian Splatting reconstruction.
- Core Framework: UniWorld-View resolves point-rendering visibility ambiguities through triple reprojection and normal-based visibility correction.This provides reliable geometric conditions under substantial viewpoint changes.
- Core Framework: The dual-stream conditional video diffusion architecture combines explicit geometric guidance with source-view appearance information.The framework targets precise camera control, consistent scene structure, and high-fidelity visual details.
- Downstream Reconstruction: UniWorld-View generates synchronized multi-view videos from monocular inputs for downstream dynamic Gaussian Splatting reconstruction.The generated views provide multi-view supervision for reconstructing dynamic 3DGS representations.
- Conclusion: Experiments on WorldScore and zero-shot novel-view synthesis benchmarks demonstrate the approach’s effectiveness and generalization ability.The stated scope is controllable novel-view synthesis from monocular images and videos.