Source-linked AI summary
Single-View View Synthesis with Multiplane Images
Richard Tucker, Noah Snavely
TL;DR
Single-view view synthesis must infer scene structure and occluded content from one image. The paper predicts a multiplane image directly from that image and introduces scale-invariant supervision for online-video training, achieving broad view-synthesis applicability, reasonable depth estimates, and limited background inpainting.
Problem
Single-view view synthesis requires inferring 3D structure and occluded content while producing high-quality, spatially consistent novel views from one photograph.
Method
The method predicts a camera-centric multiplane image from one RGB image and uses scale-invariant view-synthesis supervision with sparse point sets and depth-related losses.
Results
The approach applies across several datasets, produces reasonable depth maps, and improves view synthesis over compared representations while requiring neither light-field data nor known-scale training inputs.
Takeaways & Limitations
The predicted background image can inpaint content behind foreground-object edges without an explicit inpainting stage, although the demonstrated inpainting is typically limited to a few pixels.
Abstract
from arXiv · showhide
A recent strand of work in view synthesis uses deep learning to generate multiplane images (a camera-centric, layered 3D representation) given two or more input images at known viewpoints. We apply this representation to single-view view synthesis, a problem which is more challenging but has potentially much wider application. Our method learns to predict a multiplane image directly from a single image input, and we introduce scale-invariant view synthesis for supervision, enabling us to train on online video. We show this approach is applicable to several different datasets, that it additionally generates reasonable depth maps, and that it learns to fill in content behind the edges of foreground objects in background layers. Project page at https://single-view-mpi.github.io/.
1. Introduction
The paper predicts a multiplane image from one RGB image to synthesize spatially consistent novel views, using scale-invariant supervision that supports training on online video.
- Single-view view synthesis requires inferring 3D structure, occlusions, hidden content, and spatially consistent real-time rendering.
- The network generates an MPI from a single image and renders novel viewpoints plus a disparity map.The MPI representation can model disocclusions and non-Lambertian effects while enabling efficient rendering.
- The approach is presented as the first to generate MPIs directly from one input image rather than multiple viewpoints.Prior MPI methods used inputs ranging from stereo pairs to twelve-image camera arrays.
- Scale-invariant view synthesis addresses global scale ambiguity when supervising view synthesis on the authors’ training data.The method uses sparse point sets produced during training-data generation to determine scale.
- The method is trained and evaluated on online videos and compared with prior view-synthesis methods using different representations.The comparison includes a complete 4D light-field method and a layered-depth-image method.
2. Related work
Related work spans single-view depth prediction and learned view synthesis, but prior methods did not predict an MPI from one image or learn single-view view synthesis from in-the-wild video.
- Single-view depth maps alone cannot represent content occluded in the reference view, while accurate depth is not strictly necessary for high-quality view synthesis.
- Prior depth-learning work used videos in the wild, including pseudo-depth from optical flow and sparse structure-from-motion measurements.
- The paper is the first, to the authors’ knowledge, to learn single-view view synthesis from videos in the wild.
- Earlier learned view-synthesis methods predicted views independently or rendered multiple views from a shared scene representation, with layered representations supporting occluded content.MPIs had succeeded for interpolation and extrapolation from multiple input images, but not from a single input.
- Single-view alternatives include full light fields, soft disparity maps, layered depth images, and segment-based 3D planes.The paper applies MPIs to the single-view case and reports higher-quality results than light fields and LDIs.
- A prior depth-based system required dense accurate depth supervision and multiple post-processing stages, whereas this method predicts an MPI in one stage from multiple-view supervision.
3. Approach
The approach predicts a multiplane image from one RGB image, renders novel views by warping and compositing its layers, and addresses training scale ambiguity with sparse-point-based scale selection. Its losses combine view synthesis, edge-aware disparity smoothness, and sparse depth supervision.
- Data and pipeline: At inference, a network maps one source RGB image to an MPI from which views at new camera positions can be generated.Training uses videos with static scenes and moving cameras; SLAM and structure-from-motion provide viewpoints and sparse point clouds.
- Representation and rendering: An MPI represents a scene with D fronto-parallel RGBA layers at fixed depths, equally spaced in inverse depth.The network output uses the source camera as the MPI reference camera.
- Representation and rendering: Novel-view rendering first warps each layer to the target viewpoint using a homography and bilinear sampling, then composites the warped layers.The same warping is applied to color and alpha channels, followed by alpha compositing.
- Scale-invariant synthesis: Scale-invariant synthesis uses sparse point sets to choose σ by minimizing log-squared disparity error, then applies σ when rendering target views.This removes dependence of rendered images on the arbitrary scale of input viewpoints and point sets, making view-synthesis losses usable.
4. Experiments
Experiments evaluate single-view MPI synthesis through RealEstate10K ablations, depth prediction, and comparisons on Flowers and KITTI. Scale invariance provides the largest ablation benefit, while background prediction mainly helps disoccluded regions and perceptual similarity.
- Evaluation scope: Experiments cover RealEstate10K ablations, iBims-1 depth evaluation, and comparisons on Flowers and KITTI.The study reports quantitative and qualitative evaluations across these datasets and benchmarks.
- RealEstate10K ablations: Scale-invariant synthesis improves view-synthesis performance more substantially than sparse direct depth supervision.Removing depth supervision causes a small degradation, whereas removing scale invariance causes a considerably larger one.
- RealEstate10K ablations: At larger camera movements, the full model performs best on LPIPS, while nosmooth and nobackground can score higher on PSNR and SSIM.This metric-dependent pattern shows that perceptual similarity and pixel metrics favor different variants.
- Background prediction: The predicted background reduces foreground-edge artefacts and improves synthesis on disoccluded areas despite not clearly improving whole-image scores.The full model predicts some content behind foreground edges, while nobackground performs worse on disoccluded pixels.
- Depth quality: Edge-aware smoothness produces disparity with smoother regions and sharper boundaries, improving predicted depth-map quality relative to nosmooth.The nosmooth model lacks sharp depth discontinuities and introduces discontinuities where depth should remain smooth.
- Cross-dataset comparisons: On Flowers, the method improves over Srinivasan et al.; on KITTI, both resolution variants improve over Tulsiani et al., with background benefits larger for disoccluded pixels.The MPI representation also supports arbitrary novel viewpoints without further inference steps and does not require complete light-field training data.
5. Conclusion
The paper demonstrates single-image MPI prediction for view synthesis without ground-truth 3D or depth. Its scale-invariant supervision supports training on scale-ambiguous online video, while predicted backgrounds provide limited inpainting behind foreground edges.
- Conclusion: The method predicts multiplane images from single RGB inputs without requiring ground-truth 3D or depth.The system learns the representation directly for view synthesis.
- Conclusion: Scale-invariant view synthesis enables training on scale-ambiguous data such as online video.The approach uses the consistent scale available within each training example.
- Conclusion: Predicted background layers can inpaint content behind foreground edges without an explicit inpainting stage.The paper reports that this inpainting typically extends only a few pixels and may contain artefacts.