Source-linked AI summary
Extreme View Synthesis
Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H. Kim, Jan Kautz
TL;DR
Sparse-view novel view extrapolation is difficult because occlusions and depth uncertainty worsen as viewpoints move farther from the inputs. Extreme View Synthesis uses depth probability volumes, depth-guided refinement, and learned image priors, achieving visually pleasing results up to 30× baseline magnification from as few as two cameras.
Problem
Sparse-view novel view synthesis must handle occlusions and depth uncertainty that become more severe during large viewpoint extrapolation.
Method
The method warps and fuses depth probability volumes, then combines depth-guided patch sampling with learned image priors to refine the initial novel view.
Results
The method produces visually pleasing extreme-view results up to 30× baseline magnification, with refinement improving quality and metrics exceeding SM.
Takeaways & Limitations
Extreme extrapolation is demonstrated from as few as two input cameras across real scenes, camera motions, and different input-camera counts.
Takeaways & Limitations
The refinement network cannot hallucinate pixels outside the input-camera frusta and struggles with naturally appearing regions reconstructed in the wrong location.
Abstract
from arXiv · showhide
We present Extreme View Synthesis, a solution for novel view extrapolation that works even when the number of input images is small--as few as two. In this context, occlusions and depth uncertainty are two of the most pressing issues, and worsen as the degree of extrapolation increases. We follow the traditional paradigm of performing depth-based warping and refinement, with a few key improvements. First, we estimate a depth probability volume, rather than just a single depth value for each pixel of the novel view. This allows us to leverage depth uncertainty in challenging regions, such as depth discontinuities. After using it to get an initial estimate of the novel view, we explicitly combine learned image priors and the depth uncertainty to synthesize a refined image with less artifacts. Our method is the first to show visually pleasing results for baseline magnifications of up to 30X.
1. Introduction
Sparse-view novel view synthesis becomes especially difficult for extrapolated viewpoints, where disocclusions, occlusions, and depth uncertainty increase artifacts. Extreme View Synthesis combines geometric warping with learned image priors to address these challenges using as few as two cameras and extrapolation up to 30×.
- Motivation: Sparse observations make seamless navigation difficult, especially when extrapolated views create disocclusions.Novel view synthesis supports telepresence and remote-environment navigation, but these applications require denser visual sampling than sparse inputs provide.
- Challenges: Depth-based methods provide geometric constraints but suffer from occlusions and uncertain depth, with both problems worsening farther from input cameras.Existing approaches either propagate reliable depth or model uncertainty explicitly, but do not combine depth for refinement with image priors for artifacts.
- Prior approaches: Learning-based methods can fill missing regions using image priors, yet still produce artifacts when few inputs are used and the virtual camera moves substantially.
- Contribution: Stereo Magnification reaches 4.5× baseline extrapolation, whereas this work targets up to 30× with as few as two cameras and freely moving, rotating virtual cameras.
- Contribution: The method uses depth probability volumes and depth-guided patch sampling to combine scene information with learned priors for interpolation and extreme extrapolation.The authors report visually pleasing results even when only two input images are available.
2. Related Work
Related work spans pixel- or ray-space interpolation, geometry-based warping, layered representations, and learned single-image methods. Extreme View Synthesis differs by targeting large viewpoint extrapolation from as few as two cameras.
- Prior approaches: Early methods interpolate corresponding pixels or rays, while geometry-based methods warp and merge input views using scene information.These approaches generally require correspondences, depth, or geometry as additional information.
- Learned representations: Learning-based representations use plane sweep volumes, layered depth images, or learned scene layers to synthesize or interpolate views.The supplied related-work passages describe these as approaches that take images as input and learn how to combine scene representations.
- Depth-based methods: Explicit depth warping handles larger camera displacements geometrically, but depth uncertainty can create noticeable artifacts.Other methods propagate sparse depth or model confidence, addressing uncertainty without matching the present setting.
- Single-image methods: Deep learning priors also support single-image light-field, appearance-flow, stereoscopic, and layered-scene synthesis.
- Distinction: This approach targets extrapolated images under large viewpoint changes using as few as two cameras.That capability distinguishes it from the cited methods in the supplied passages.
3. Overview
The method estimates depth probabilistically, warps and fuses the resulting volumes into a novel-view representation, then refines an initial rendering with a patch-based network.
- Overview: Depth estimation errors around discontinuities cause warping artifacts, which worsen with few inputs or extrapolated virtual-camera positions.
4. Estimating the Depth Probability Volume
The method represents per-ray depth uncertainty as probability distributions, then resamples, accumulates, and normalizes these volumes for the novel view. This preserves multimodal uncertainty near depth discontinuities while enabling efficient view-dependent fusion.
- Volume representation: Each pixel is associated with a probability distribution over nd discretized depth values along its camera ray.The collection of these ray distributions forms a depth probability volume Di ∈ R^h×w×nd.
- Volume representation: The depth volumes are trained as a multi-class classification problem using cross-entropy against one-hot ground-truth depth classes.Disparity is sampled uniformly rather than depth to improve estimation accuracy for closer objects.
- Uncertainty: Most rays produce a single strong disparity peak, while rays near depth discontinuities produce foreground and background peaks representing inherent uncertainty.These discontinuities are especially challenging for view synthesis.
- Implementation: The novel-view volume uses uniformly spaced disparity levels, with scene-specific disparity bounds and optional cross-bilateral filtering guided by the input RGB image.The implementation sets nd = 100 disparity steps between the bottom and top depth percentiles.
- Novel-view volume: The resampling procedure accumulates input volumes on planar slices, and the resulting ray probabilities require additional normalization because they do not initially sum to one.
- Novel-view volume: Input probability volumes are resampled into the novel-view volume, accumulated across contributing views, and normalized along each ray.The representation supports averaging intersecting-ray probabilities while avoiding the cross-view comparability problem of traditional cost volumes.
5. Synthesis of a Novel View
The method renders an initial novel view by backward-warping input pixels using a depth probability volume and sweeping depth planes from far to near. Rejecting uncertain depth estimates preserves geometric constraints but leaves artifacts and holes for refinement.
- The depth probability volume DNV guides backward warping of input pixels into the novel view.Pixels are mapped from input coordinates (xi, yi) to novel-view coordinates (x, y).
- Rendering begins at the farthest plane, d = 0, and merges corresponding input pixels using camera-distance and principal-axis-angle weights.The merge function R combines pixels from the input views.
- As disparity increases toward the camera, pixels are overwritten when DNV(x, y, d) exceeds a threshold.This back-to-front sweep selects increasingly near depth planes.
- The resulting initial estimate generally contains artifacts and holes because uncertain depth estimates are rejected during back-to-front overwriting.At this stage, the image is intended to obey the geometric constraints represented by the depth probability volumes.
6. Image Refinement
The refinement stage addresses artifacts and holes in the warped view by combining the synthesized patch with depth-informed patches from the input images. A variable-input UNet uses independently encoded candidate patches and max-pooled features to produce higher-quality patches.
- Image Refinement: The refinement network targets unrendered regions, edge fattening, and local deformations caused by occlusions, missing depth, and independent pixel rendering.These artifacts affect the initially synthesized image.
- Image Refinement: Weighting RGB values by depth probabilities produced softer edges or ghosting that were harder for the refinement network to correct.The authors speculate that these artifacts appear more plausible to the network than abrupt, incoherent RGB changes.
- Image Refinement: The method combines generic image priors with explicit scene information by selecting input patches according to the depth probability volume.This avoids relying only on generic priors during refinement.
- Image Refinement: Patches are warped with homographies induced by candidate depth planes to reduce view-dependent differences without local deformations.The selected patches form an educated subset of a plane-sweep volume.
- Image Refinement: The network accepts a variable number of input patches because occlusions, field-of-view limits, and multimodal depth distributions change patch availability.Each input view may contribute a different number of patches.
- Refinement Network: A UNet encodes each available patch independently, max-pools their features, and concatenates them with synthesized-patch features.Skip connections combine synthesized-patch features with pooled candidate-patch features at corresponding layers.
- Refinement Network: The refinement network is trained on MVS-Synth with perceptual loss and ADAM.Further network and training details are reported in the Supplementary material.
7. Evaluation and Results
The evaluation combines quantitative comparisons on synthetic scenes with qualitative tests on real scenes, including extreme extrapolation from two inputs and refinement-network ablations. Results show improved image quality, sharper outputs, fewer artifacts, and effective use of additional input patches, while limitations remain for unseen regions and depth quantization.
- Execution Time: 40s for depth probability volumes, 30s for view synthesis, and 28s for refinement were the reported average execution times using two input views.The timing covers depth-volume computation, novel-view rendering, and the refinement network.
- Synthetic Scenes: Higher PSNR and SSIM values were obtained for the refined method output than for both the warped output and Stereo Magnification.Table 1 compares Ours warped, Ours refined, and SM; the refinement network improves quality, and the method output scores higher than SM.
- Real Scenes: At 30× stereo-baseline magnification, the method produced sharper results with fewer and less noticeable artifacts than Stereo Magnification.The comparison includes scenes with complicated structure and occlusions.
- Real Scenes: Using only two input cameras, the method synthesized views displaced several times the input baseline, including a difficult dolly-in trajectory.The evaluation also presents results with four input cameras and varied scenes and camera displacements.
- Refinement Network: Patch-guided refinement reconstructed local structure and synthesized pixels in areas that had previously been occluded, unlike an equal-parameter single-patch network.Figure 8 compares the proposed architecture with a network whose only architectural difference is the absence of additional patches.
- Limitations: The refinement network cannot hallucinate pixels outside the input cameras’ frusta, and discrete depth values can cause quantization artifacts in some novel views.Increasing disparity levels can reduce quantization artifacts but increases memory footprint and execution time.
8. Conclusions
The method targets extreme novel-view synthesis with as few as two input cameras and extrapolation up to 30× for stereo pairs. It combines geometric constraints with learned priors and supports varying numbers of input cameras.
- 8. Conclusions: The method targets extreme cases with as few as two input cameras and up to 30× extrapolation for stereo pairs.Figure 10 compares two-camera and four-camera inputs, while Figure 11 shows a 30× baseline comparison.
- 8. Conclusions: The approach combines traditional geometric constraints with learned priors to synthesize novel views.The conclusion describes the method as operating across real scenes, camera motions, and different numbers of input cameras.
- 8. Conclusions: The method supports different numbers of input cameras, including the two- and four-camera settings illustrated in Figure 10.The figure compares the input-view positions and virtual views for both settings.