Source-linked AI summary
ORCA: Occlusion-Aware Refinement and Completion for Novel View Synthesis
Weronika Jakubowska, Maciej Zięba, Przemysław Spurek
TL;DR
Single-image novel-view synthesis must handle hidden regions and ambiguous depth that become visible as the camera moves, while generated completions can conflict with reconstructed geometry. ORCA adds monocular-depth structure to Gaussian anchors, repairs small disocclusions from existing RGB-D information, and uses generative inpainting only for larger unrecoverable regions. Experiments show improved perceptual quality and cross-view geometric consistency over VistaDream.
Problem
Single-image novel-view synthesis is ambiguous because camera motion reveals hidden regions and gaps, while generated completions may conflict with existing geometry and neighboring viewpoints.
Method
ORCA introduces monocular-depth structure into Gaussian anchors, repairs small disocclusions with existing RGB-D information, and reserves generative completion for larger unrecoverable regions.
Results
ORCA improves perceptual novel-view quality and cross-view geometric consistency over VistaDream on DIV2K and RealmDreamer.
Takeaways & Limitations
Many novel-view artifacts can be repaired by reusing information already present in the reconstructed scene, reducing the need to generate new content.
Takeaways & Limitations
The same-camera original and outpainted observations provide no multi-view parallax and therefore do not constrain scene depth.
Abstract
from arXiv · showhide
Novel-view synthesis from a single image is a fundamentally ambiguous problem. As the camera moves away from the input viewpoint, previously hidden regions become visible, exposing missing geometry and holes in the reconstructed scene. Existing methods often rely on generative models to complete such regions. However, many of these artifacts are small gaps near depth boundaries and do not require generating new scene content. In order to eliminate expensive process of generating image we introduce ORCA, an occlusion-aware method for reconstructing and completing explorable 3D scenes from a single image. ORCA first introduces 3D structure into a Gaussian-anchor representation using monocular depth while preserving the original camera-ray correspondence. During scene exploration, missing regions are handled based on their size and structure. Small disocclusions are repaired using RGB-D information already available in the reconstruction, while generative inpainting is reserved for larger regions that cannot be reliably recovered from the scene. New Gaussian anchors are added and optimized locally without modifying the existing representation. By reducing unnecessary reliance on generative inpainting, ORCA limits generation-induced hallucinations and better preserves the content and structure of the original scene. On DIV2K, ORCA improves novel-view quality over VistaDream across all reported metrics, increasing MUSIQ from 61.60 to 68.71 and CLIP-IQA from 0.474 to 0.574. These results show that many novel-view artifacts can be repaired effectively by reusing information already present in the reconstructed scene.
1. Introduction
ORCA addresses single-image novel-view failures by distinguishing recoverable boundary gaps from genuinely unseen content. It introduces depth-aware Gaussian structure, reuses existing RGB-D information for small disocclusions, and reserves generative completion for regions that cannot be recovered.
- 1. Introduction: Single-image reconstruction is ambiguous because hidden regions and uncertain depth become exposed as incorrect geometry and gaps when the camera moves.Novel viewpoints reveal surfaces, gaps, or regions never observed in the input image.
- 1. Introduction: Generative completion can introduce content that conflicts with reconstructed geometry and neighboring viewpoints.A generated image may appear plausible from one view while becoming inconsistent from another.
- 1. Introduction: Many camera-induced artifacts are small boundary gaps whose missing appearance already exists in the reconstruction, whereas larger occlusions may contain unrecoverable unseen content.This distinction motivates treating small disocclusions differently from genuinely unseen regions.
- 1. Introduction: ORCA repairs small disocclusions with nearby RGB-D information and uses generative completion only when missing regions cannot be reliably recovered from the existing scene.Depth-based background sampling places added geometry behind foreground objects rather than extending foreground surfaces.
- 1. Introduction: ORCA introduces monocular-depth 3D structure into a Gaussian-anchor scene while preserving the original camera-ray correspondence.The method uses depth to deform Gaussian positions along their original rays and then fine-tunes the remaining representation.
- 1. Introduction: New Gaussian anchors are added and optimized locally inside each selected missing region without changing the existing scene representation.The trajectory is rerendered after each repair so later repairs can use previously added geometry.
2. Related Work
Prior work extends single-image scenes through generated views, inpainting, depth guidance, and Gaussian representations. ORCA differs by recovering small disocclusions directly from existing RGB-D information and reserving generation for larger unseen regions.
- 2. Related Work: Single-image novel-view synthesis lacks multi-view geometry and observations of occluded regions, motivating generated views and other completion strategies.The related methods include object-centric multi-view generation, zero-shot real-scene synthesis, and camera-conditioned diffusion.
- 2. Related Work: Recent methods incorporate camera parameters, 3D priors, trajectories, or geometric feedback into multi-view generation.Examples include MVGenMaster, Stable Virtual Camera, 3D-Adapter, and Director3D.
- 2. Related Work: Explorable-scene methods combine image outpainting or generated appearance with geometric initialization, depth guidance, Gaussian scaffolds, and iterative refinement.WonderWorld, ExScene, and One2Scene use different combinations of these components.
- 2. Related Work: VistaDream repeatedly extends RGB-D reconstructions and refines generated observations with multi-view consistency sampling, while RealmDreamer combines Gaussian representations with inpainting and depth guidance.These methods are particularly related because they also target extended 3D scenes from image-based inputs.
- 2. Related Work: ORCA differs by using generation only when the current reconstruction lacks sufficient information, repairing small disocclusions from existing RGB-D data and reserving inpainting for larger unseen regions.This targets consistency between synthesized appearance, existing geometry, and neighboring viewpoints without generating every exposed region.
3. Method
ORCA builds an explorable Gaussian scene by adding monocular-depth structure while preserving camera-ray correspondence, then repairs viewpoint-induced disocclusions according to their size and recoverability.
- 3.1. Initial Scene Representation: ORCA first optimizes a planar Gaussian representation from the input and outpainted images, then introduces monocular-depth structure and fine-tunes the deformed scene.The shared camera viewpoint provides no multi-view depth constraints, so depth is added after initial reconstruction.
- 3.2. Depth-Based Gaussian Deformation: Gaussian means are displaced along their original camera rays according to relative monocular depth, preserving image-space correspondence while deforming scene geometry.Depth is converted to a relative nearness measure and normalized to the Gaussian cloud's spatial scale before deformation.
- 3.3. Disocclusion Repair: During exploration, ORCA renders a camera trajectory, selects the view with the largest disocclusion, and processes connected repair components from local gaps to larger unseen regions.The trajectory is rerendered after each repair so later repairs account for newly added geometry.
- 3.3. Disocclusion Repair: Small or elongated gaps are repaired from farther background RGB-D samples, placing new Gaussian anchors behind foreground objects and optimizing them locally without changing the existing scene.Wider, overlapping Gaussians improve coverage and reduce thin gaps near repair boundaries.
- 3.3. Disocclusion Repair: Large compact regions that cannot be recovered from local background are completed with cropped generative inpainting, with a budget of at most two operations per scene.Generated depth is locally aligned to the reconstruction before new anchors are inserted; failures fall back to local background-based repair.
4. Experiments
ORCA is evaluated against VistaDream on natural and synthetic scenes using perceptual, geometric, and qualitative novel-view assessments. It consistently improves reported image-quality and cross-view consistency metrics while producing more complete renderings under camera motion.
- Quantitative Results: ORCA consistently outperforms VistaDream across reported image-quality metrics on DIV2K and RealmDreamer.On DIV2K, MUSIQ rises from 61.60 to 68.71 and CLIP-IQA from 0.474 to 0.574; on RealmDreamer, MUSIQ rises from 68.66 to 72.85 and CLIP-IQA from 0.378 to 0.457.
- Quantitative Results: ORCA achieves higher MUSIQ and CLIP-IQA than VistaDream on all 11 RealmDreamer scenes.The largest improvements occur in challenging scenes such as bear and piano, although individual LLaVA-IQA criteria sometimes remain unchanged or favor VistaDream.
- Quantitative Results: ORCA improves cross-view geometric consistency, with TSED increasing from 0.8265 to 0.9980 on DIV2K and from 0.9864 to 1.0000 on RealmDreamer.The evaluation measures consistency across rendered views rather than only individual-frame image quality.
- Qualitative Results: Qualitative trajectories show VistaDream producing uncovered regions, Gaussian-like artifacts, and blur, whereas ORCA yields more complete views with more consistent scene structure.ORCA fills small disocclusions without unnecessarily changing surrounding appearance and integrates larger missing regions with less visible degradation.
5. Conclusion
ORCA reconstructs explorable 3D scenes from a single image by adding depth-aware structure and repairing disocclusions according to whether existing scene information can recover them. Experiments support improved perceptual quality and geometric consistency while reducing unnecessary generation of new content.
- Conclusion: ORCA introduces scene depth with monocular estimates while preserving the original camera-ray correspondence in a Gaussian representation.This produces an explorable 3D scene from a single image.
- Conclusion: Small disocclusions are repaired with RGB-D information already in the reconstruction, while generative inpainting is reserved for content that cannot be recovered reliably.New geometry is optimized locally without modifying the existing representation.
- Conclusion: ORCA improves both perceptual novel-view quality and cross-view geometric consistency compared with VistaDream on DIV2K and RealmDreamer.The results support the use of existing reconstructed information to repair many artifacts revealed by camera motion.