Source-linked AI summary
Pushing the Boundaries of View Extrapolation with Multiplane Images
Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, Noah Snavely
TL;DR
The paper studies high-quality view extrapolation from a narrow-baseline stereo pair, especially when target views reveal disoccluded content. It analyzes how MPI disparity sampling limits renderable views and introduces MPI prediction procedures that theoretically allow up to 4× more lateral viewpoint movement than prior work while reducing depth and repeated-texture artifacts.
Problem
View extrapolation from narrow-baseline images is limited by depth discretization artifacts and unrealistic repeated textures in disoccluded regions.
Method
The method combines theoretical MPI analysis with a 3D CNN and randomized-resolution training for higher disparity sampling, plus a hidden-content appearance constraint enforced by two-step MPI prediction.
Results
Up to 4× more lateral viewpoint movement is theoretically enabled than in prior work, while the method reduces depth discretization and repeated-texture artifacts.
Takeaways & Limitations
Renderable-view range grows linearly with MPI disparity sampling frequency, and constrained hidden-content appearance produces more convincing disocclusion effects.
Takeaways & Limitations
The theoretical analysis considers translated, non-rotating cameras and no occlusion, including a locally linear coordinate approximation for a worst-case MPI.
Abstract
from arXiv · showhide
We explore the problem of view synthesis from a narrow baseline pair of images, and focus on generating high-quality view extrapolations with plausible disocclusions. Our method builds upon prior work in predicting a multiplane image (MPI), which represents scene content as a set of RGB$α$ planes within a reference view frustum and renders novel views by projecting this content into the target viewpoints. We present a theoretical analysis showing how the range of views that can be rendered from an MPI increases linearly with the MPI disparity sampling frequency, as well as a novel MPI prediction procedure that theoretically enables view extrapolations of up to $4\times$ the lateral viewpoint movement allowed by prior work. Our method ameliorates two specific issues that limit the range of views renderable by prior methods: 1) We expand the range of novel views that can be rendered without depth discretization artifacts by using a 3D convolutional network architecture along with a randomized-resolution training procedure to allow our model to predict MPIs with increased disparity sampling frequency. 2) We reduce the repeated texture artifacts seen in disocclusions by enforcing a constraint that the appearance of hidden content at any depth must be drawn from visible content at or behind that depth. Please see our results video at: https://www.youtube.com/watch?v=aJqAaMNL2m4.
1. Introduction
The paper addresses view extrapolation from a narrow-baseline stereo pair, where distant target viewpoints reveal disoccluded content and degrade rendered-view quality. It analyzes MPI rendering limits and introduces methods to reduce depth discretization and repeated-texture artifacts.
- View synthesis predicts novel scene views from captured images, supporting effects such as 3D parallax, synthetic defocus blur, and free-viewpoint navigation.
- View extrapolation becomes difficult as the target viewpoint moves farther from the inputs because more previously occluded content is revealed.
- An MPI represents a scene with fronto-parallel RGBα planes sampled within a reference camera frustum and renders novel views by projecting those planes.
- Theoretical analysis shows that the renderable-view range is limited by MPI disparity sampling frequency and increases linearly with that frequency.
- A 3D CNN with randomized-resolution training predicts MPIs with increased disparity sampling frequency, reducing depth discretization artifacts in distant rendered views.
- A two-step prediction procedure constrains hidden-content appearance to visible content at or beyond the relevant depth, producing plausible textures and reducing repeated disocclusion artifacts.
2. Related Work
Related view-synthesis methods range from densely sampled light fields to geometry-dependent reprojection and end-to-end deep networks. The paper focuses on MPI-based scene representations because view extrapolation requires consistent prediction of content hidden in all input images.
- Traditional methods range from light-field rendering with many densely sampled images to view-dependent texture mapping based on an accurately estimated global mesh.
- Many modern methods compute local geometry and blend projected textures, but traditionally interpolate between inputs rather than predict content occluded in all input images.
- End-to-end methods that learn geometry and rendering from limited inputs can synthesize distant views, but lacking built-in geometry and rendering knowledge limits them to synthetic non-photorealistic scenarios.
- Other photorealistic pipelines predict local geometry for each novel viewpoint, which prevents guaranteed consistency across predictions and can create temporal artifacts in view sequences.
- Forward-projecting a predicted 3D scene representation supports consistency between rendered views and prediction of hidden content, motivating layered-depth and MPI approaches.
- 2D and multi-view inpainting operate on individual renderings or input collections, so they do not provide scene-consistent content for regions appearing only during view extrapolation.
- Shape-completion methods predict hidden geometry and semantics but not appearance, whereas this work jointly inpaints hidden geometry, color, and opacity for novel-view rendering.
- An MPI consists of fronto-parallel RGBα planes in a reference camera frustum, with novel views rendered by alpha compositing warped plane colors.
3. View Extrapolation for Visible Content
An MPI has a finite renderable viewpoint range even for mutually visible content, determined by its disparity sampling frequency. The paper derives this limit and proposes a 3D-CNN approach that predicts denser MPIs at test time to extend visible-content extrapolation.
- Theoretical signal processing limits: An MPI’s renderable viewpoint range is limited by disparity sampling frequency, even when content is visible from every input and target viewpoint.Rendering beyond this range produces depth discretization artifacts resembling aliasing in volume rendering.
- Theoretical signal processing limits: Fourier analysis models an MPI-rendered view as a slice through the MPI spectrum, with farther viewpoints corresponding to steeper slices and reduced spatial bandwidth outside the valid range.The MPI spectrum is bounded by spatial and disparity sampling frequencies.
- Theoretical signal processing limits: The allowed lateral camera movement increases linearly with axial distance from the MPI and increases linearly as the disparity interval decreases.At the renderable boundary, adjacent MPI planes shift by one pixel relative to each other before compositing.
- Increasing disparity sampling frequency: Prior MPI prediction uses a 2D CNN with 32 fixed planes at 1024×576 resolution, making denser fixed-plane training computationally infeasible under GPU memory limits.Because additional planes increase extrapolation ability linearly, this fixed-plane design restricts the renderable range.
- Increasing disparity sampling frequency: A fully 3D convolutional network with randomized-resolution training predicts full-resolution MPIs with up to 128 planes at test time, theoretically increasing visible-content extrapolation by 4×.Training spans different spatial and disparity sampling frequencies that fit within GPU memory.
4. View Extrapolation for Hidden Content
The method addresses repeated textures in hidden content by constraining each hidden voxel to reuse visible content at or behind its depth. A two-step MPI pipeline first estimates visible scene content, then predicts hidden-content geometry and flow-based colors.
- Motivation: Repeated RGB content across MPI layers causes repeated occluder textures in extrapolated disocclusions.This behavior occurs with both 2D and 3D CNN architectures.
- Appearance constraint: The appearance of hidden content is constrained to reuse visible scene content at or behind the same depth.This reduces output uncertainty and copies textures from nearby visible background content.
- First step: The two-step procedure first predicts an initial MPI from input plane-sweep volumes using a 3D CNN.The initial MPI contains RGB and α values and estimates visible and hidden scene content.
- Second step: Occluded RGB content is softly removed using transmittance, producing a reference-visible MPI for the second prediction stage.The transmittance describes how much an MPI voxel contributes to the rendered reference view.
- Second step: A second CNN predicts final opacities and a 2D flow vector for every MPI voxel.The flow gathers colors from visible-content renderings at or behind each plane, with bilinear interpolation for differentiability.
5. Training Loss
The MPI pipeline is trained with view synthesis supervision using reconstruction losses for both initial and final rendered views, including a deep VGG-19 feature-matching loss.
- View synthesis supervision: Training uses held-out novel-view reconstruction losses for both the initial and final predicted MPIs.The initial and final MPIs are predicted from two input images and rendered at a target camera pose.
- Feature loss: The total loss includes a deep feature-matching loss computed from selected VGG-19 layers.The feature loss compares the final rendered view with the held-out target view.
6. Results
Experiments on YouTube Real Estate 10K evaluate overall rendering quality, disocclusion accuracy, and perceptual plausibility. The method outperforms the original MPI approach and alternative hidden-content strategies qualitatively and quantitatively.
- Experiment details: Approximately 87% of training examples require view extrapolation because the target lies outside the interval between the source frames.Each example samples two source frames and a target frame from a randomly chosen video.
- Evaluation metrics: The metrics measure overall image quality, disocclusion accuracy, and perceptual plausibility of hidden content.SSIMfov evaluates the target region viewing all MPI planes, SSIMocc evaluates disoccluded pixels, and NATocc uses a Wasserstein-1 image prior.
- Baseline comparison: Our initial and final MPIs outperform the original MPI model, with final renderings producing more accurate and plausible disocclusions.The method also produces fewer depth discretization artifacts and repeated textures.
- Hidden-content prediction: The full method improves over its initial-MPI ablation in disocclusion accuracy and perceptual plausibility, with fewer repeated-texture artifacts.This comparison evaluates the flow-based hidden-content appearance constraint.
- Hidden-content prediction: Image-based disocclusion inpainting reduces overall quality and creates distracting temporal artifacts along rendered camera paths.The inpainted disocclusions are less accurate and less plausible than those from the full method.
7. Conclusion
The paper combines theoretical MPI-limit analysis with a practical prediction method that theoretically permits 4× more lateral viewpoint movement than prior work. The authors identify increased disparity sampling and flow-based hidden-content constraints as the sources of this improvement.
- Conclusion: 4× more lateral viewpoint movement is theoretically enabled than in prior work.The method predicts MPIs with increased disparity sampling frequency and uses a flow-based hidden-content appearance constraint.
- Conclusion: The authors state that photorealistic view synthesis with convincing occluded 3D content remains an open area for improvement.They present the work as a direction for future progress in deep-learning-compatible scene representations.
A. Supplementary Video
The supplementary video compares renderings from the proposed model with the original MPI method and a disocclusion-inpainting baseline.
- The comparisons provide qualitative evidence of improved renderings with fewer depth discretization and repeated texture artifacts.
B. Section 3.2 Derivation Details
The supplementary derivation details the camera geometry, Fourier representation, and rendering operation used to analyze MPI view limits.
- Camera setup geometry: Figure S1 relates reference and novel camera positions through lateral and axial translations (u, s).The MPI lies in the reference-camera frame, while the novel camera views it after translation.
- Camera setup geometry: MPI plane coordinates scale linearly with 1/d because planes contain equal-sized pixel grids sampled within the camera frustum.
- Theoretical setup: The rendering analysis assumes content in the closest planes and locally linearizes the coordinate transformation from (x, d) to (x′, d).
- Fourier derivation: The generalized Fourier slice theorem interprets MPI rendering as a 2D slice through the 3D Fourier transform of a sheared and dilated MPI.The slicing operator sets the last N−M dimensions to zero.
- Fourier derivation: The rendered view is computed by taking the MPI Fourier-transform slice at k_d = 0 and applying the inverse Fourier transform.The omitted 1/|B−T| term is treated as a scaling factor absorbed into C.
C. Network Architecture and Training Details
The supplementary material specifies the 3D CNN implementation and randomized-resolution training used for MPI prediction, along with disocclusion-mask computation for evaluation.
- Training: The system is implemented in TensorFlow and trained with Adam for 300,000 iterations at learning rate 2 × 10^-4 and batch size 1.
- Training: Randomized-resolution training samples PSV tensors across multiple spatial resolutions and 24–27 plane depths.At test time, the network receives tensors of size [576,1024,27,6].
- Evaluation: The method computes disocclusion masks between reference and target viewpoints for quantitative evaluations.
E. “rinit + Adversarial Disocclusions” Details
This baseline uses a two-step MPI prediction strategy with an adversarial loss applied to final rendered target images to encourage realistic disocclusions.
- Baseline strategy: “rinit + Adversarial Disocclusions” uses an identical Φ1 for initial MPI prediction, while Φ2 directly predicts RGBα layers instead of flows.
- Adversarial training: The baseline applies an adversarial loss to each final rendered target image using SN-PatchGAN with spectral normalization and hinge loss.
- Adversarial training: The adversarial loss is added to the main objective with weight λ = 5.0.