Source-linked AI summary
DeepView: View Synthesis with Learned Gradient Descent
John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, Richard Tucker
TL;DR
Sparse-view MPI reconstruction is an ill-posed inverse problem prone to overfitting and artifacts. DeepView uses learned gradient descent to infer an MPI with visibility-aware updates, achieving state-of-the-art view synthesis on two datasets, including Spaces.
Problem
Sparse-view MPI reconstruction is ill-posed because the MPI has more parameters than the effective measurements, making overfitting and visual artifacts a central challenge.
Method
DeepView uses learned gradient descent to iteratively update an MPI, with a network learning priors, adaptive steps, and visibility-aware occlusion reasoning.
Results
DeepView achieves significantly higher SSIM than Soft3D and the Zhou et al.-based model across experiments, improving Spaces average SSIM by 39% (0.9584 vs. 0.9319).
Takeaways & Limitations
The learned-gradient-descent MPI produces state-of-the-art view synthesis and supports real-time rendering of novel viewpoints.
Takeaways & Limitations
Occluding specular surfaces remain difficult for MPIs, and experiments beyond four iterations were unavailable because of memory constraints.
Abstract
from arXiv · showhide
We present a novel approach to view synthesis using multiplane images (MPIs). Building on recent advances in learned gradient descent, our algorithm generates an MPI from a set of sparse camera viewpoints. The resulting method incorporates occlusion reasoning, improving performance on challenging scene features such as object boundaries, lighting reflections, thin structures, and scenes with high depth complexity. We show that our method achieves high-quality, state-of-the-art results on two datasets: the Kalantari light field dataset, and a new camera array dataset, Spaces, which we make publicly available.
1. Introduction
DeepView estimates a multiplane image from sparse views using learned gradient descent, which learns priors and visibility-aware updates for challenging view synthesis. It achieves high-quality results on the Kalantari and Spaces datasets.
- 1. Introduction: MPIs represent light fields as stacks of semi-transparent colored layers and can model effects such as transparency and alpha matting.This representation supports real-time synthesis of new views of real scenes.
- 1. Introduction: DeepView estimates a multiplane image from sparse input views using learned gradient descent.A convolutional network predicts an initial MPI, then learned updates iteratively refine it using the current estimate.
- 1. Introduction: The learned update rule acts as a prior on natural-scene parameters and uses parameter-specific steps to converge in fewer iterations than standard gradient descent.The update network manipulates gradients to keep representations on the manifold of natural scenes.
- 1. Introduction: The method uses gradients that encode visibility information between input views and MPI layers, supporting occlusion reasoning.This visibility information is incorporated while iteratively improving the MPI.
- 1. Introduction: DeepView produces high-quality view synthesis results that outperform recent approaches on the Kalantari dataset and the newly introduced Spaces dataset.Spaces is presented as a large, challenging dataset for training and testing view synthesis methods.
2. Background and related work
Prior view-synthesis methods either impose appearance assumptions or reconstruct explicit geometry, each struggling with aspects of complex scenes. DeepView combines MPI estimation with learned gradient descent to reduce overfitting while reasoning about visibility and occlusions.
- 2. Background and related work: Sparse light-field capture makes view synthesis an under-sampled reconstruction problem because dense viewpoint measurements are rarely practical.Light fields represent all rays through a volume, but real scenes are commonly recorded from limited viewpoints.
- 2. Background and related work: Methods that impose light-field priors, such as Lambertian reflectance or transform sparsity, can confuse scene geometry with reflectance.These approaches do not explicitly model scene geometry.
- 2. Background and related work: Explicit 3D reconstruction methods struggle with specular highlights, transparency, and semi-transparent or thin objects.Their representations include global reconstructions and collections of per-input-view depth maps.
- 2. Background and related work: MPIs offer a more flexible scene representation than depth maps or triangle meshes, but MPI prediction must model visibility between the MPI and input views.Correct representation requires handling regions occluded in some input views.
- 2. Background and related work: Simple gradient-descent optimization can model visibility but overfits underdetermined MPI parameters, requiring difficult-to-design priors or regularization.The limited input views provide fewer effective measurements than the number of MPI variables.
- 2. Background and related work: Unlike methods that run a network for every desired output view, MPI-based approaches can support real-time view synthesis after predicting the representation.DeepView iteratively refines the MPI while propagating visibility information through geometric operations.
- 2. Background and related work: DeepView combines inverse-problem estimation with learning by processing per-iteration gradients through a CNN to generate MPI updates.The update CNN is designed to avoid overfitting, take large steps, and reason about occlusions.
3. Method
DeepView reconstructs an MPI from sparse views by replacing conventional gradient-descent updates with learned CNN updates that use visibility-aware gradient components. The resulting iterative system combines learned priors, adaptive updates, and occlusion reasoning to produce renderable novel views.
- MPI representation and rendering: The MPI renderer warps each fronto-parallel RGBA plane into the target view and composites the warped planes from back to front.The planes are fixed in inverse-depth order, and premultiplied alpha simplifies the over-compositing operation.
- Learned gradient descent: The method solves MPI reconstruction as an inverse problem in which the MPI has more variables than the sparse input images provide measurements for.This underdetermination makes simple optimization prone to overfitting and motivates learned priors.
- Learned gradient descent: A learned network replaces the standard gradient-descent rule, folding the prior and adaptive parameter-specific step size into its update function.The network processes gradient information to generate updates rather than using a fixed analytical rule.
- Training: The unrolled network is trained by rendering its final MPI to a held-out view and comparing that rendering with the corresponding ground-truth image.After training, the resulting network is applied to new, unseen scenes.
- View synthesis gradients: Per-view gradient components encode rendered-image, accumulated-over, and net-transmittance information, allowing the update network to use visibility between input views and MPI layers.Accumulated over summarizes content behind a depth slice, while net transmittance captures attenuation through front slices.
- Network architecture: The system initializes an MPI from input plane sweep volumes, then iteratively updates it using per-view gradient components and the current MPI estimate.The initialization and update CNNs share a core architecture but use different learned weights.
4. Evaluation
DeepView is evaluated on the Kalantari Lytro dataset and the more challenging Spaces camera-array dataset against Soft3D and a Zhou et al.-based model. It achieves higher SSIM, improves with more input views, and produces detailed reconstructions while retaining limitations around occluding specular surfaces and computational demands.
- Experimental setup: The evaluation covers the Kalantari Lytro dataset and Spaces, a 100-scene dataset captured with a 16-camera rig and jittered rig positions.Spaces supports varied input-camera configurations and is intended as a more challenging shared benchmark for view synthesis.
- Experimental setup: DeepView is compared with Soft3D and an adapted Zhou et al.-based network, with ablations varying model components and LGD iterations.The adapted Zhou et al. model could not be trained with 12 views because of RAM and speed constraints.
- Quantitative results: 18%: DeepView improves average SSIM over Soft3D on the Kalantari dataset, reaching 0.9674 versus 0.9604.SSIM is higher-is-better and is computed against ground-truth images.
- Quantitative results: 39%: DeepView improves average SSIM over Soft3D on Spaces, reaching 0.9584 versus 0.9319.Across experiments, DeepView also yields significantly higher SSIM than Soft3D and the Zhou et al.-based model.
- Quantitative results: Performance improves as input views increase from 4 to 12 and remains good at wider baselines.The Zhou et al.-based model suffers at the largest baseline, where visibility propagation requires more network connections.
- Ablation and iteration studies: Including all gradient components gives the best feature loss, while performance improves as LGD iterations increase from 1 to 4.More than four iterations could not be tested, and four iterations already required too much memory for practical higher-resolution use.
- Qualitative results: DeepView reconstructs challenging edges, reflections, thin railings, wires, foliage, and high-depth-complexity regions with sharper visual and depth results than Soft3D.Its MPI can represent diffuse reflections through transparent alpha values that reveal more distant reflection planes.
- Qualitative results: Occluding specular surfaces remain difficult: placing a reflective table surface on a far plane can make it partially transparent and reveal nearer chair legs.This example exposes a representation limitation shared by the MPI configuration used in the method.
5. Discussion
DeepView achieves strong view synthesis while retaining flexibility in MPI resolution and depth-plane configuration. Its main costs are substantial training resources, depth-plane scaling with disparity, fixed input-view counts, and limitations in depth accuracy and representation efficiency.
- Performance and flexibility: DeepView achieves state-of-the-art view synthesis while allowing MPI resolution, plane count, and plane depths to change after training.A model trained on medium-distance objects can still perform well when nearer scenes require more depth planes.
- Drawbacks and limitations: Training takes several days on multiple GPUs, and the approach requires substantial implementation complexity, RAM, and training speed.
- Drawbacks and limitations: The number of MPI depth planes must increase with maximum disparity, motivating multiple MPIs for larger scenes.
- Drawbacks and limitations: The current implementation trains only models with a fixed number of input views, although max-pooling may enable this restriction to be removed.
- Future work: Depth visualizations are strong around object edges but less accurate in smooth areas because the training objective does not guide depth there.Adding a ground-truth depth loss is proposed as future work.
- Future work: MPIs produce realistic synthesized images but are over-parameterized for scenes containing large areas of empty space.The paper suggests enforcing MPI sparsity or developing a more parsimonious representation.
6. Conclusion
DeepView infers an MPI with learned gradient descent, using gradient components that encode visibility and support occlusion reasoning. The method achieves state-of-the-art performance on a difficult real-world dataset and demonstrates the promise of learned gradient descent for complex nonlinear inverse problems.
- DeepView infers a multiplane image scene representation using learned gradient descent.
- The gradient components encode visibility information, enabling the network to reason about occlusion.
- The resulting method achieves state-of-the-art performance on a difficult, real-world dataset.
- The approach demonstrates the promise of learned gradient descent for complex, nonlinear inverse problems.