Source-linked AI summary
Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines
Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, Abhishek Kar
TL;DR
Real-scene virtual exploration needs high-quality novel views, but dense Nyquist sampling is impractical and sparse-view methods offer little capture guidance. The paper uses MPI-based local light fields blended across sampled views and derives a prescriptive sampling bound, achieving Nyquist-level perceptual quality with up to ≈4000× fewer images while supporting practical capture and realtime viewing.
Problem
Nyquist-rate capture can require intractably dense sampling, while sparse-view methods provide little rigorous guidance for choosing input views.
Method
The method predicts a layered MPI representation for each input view, blends adjacent local light fields, and derives a plenoptic-sampling bound for capture density.
Results
≈4000× fewer input images match the perceptual quality of dense Nyquist-rate view sampling.
Takeaways & Limitations
The pipeline makes view synthesis practical by prescribing capture density and supporting reliable rendering for complex real-world scenes.
Takeaways & Limitations
Ambiguous or repetitive textures and moving scene content can cause floating or blurred patches from incorrect layer assignments.
Abstract
from arXiv · showhide
We present a practical and robust deep learning solution for capturing and rendering novel views of complex real world scenes for virtual exploration. Previous approaches either require intractably dense view sampling or provide little to no guidance for how users should sample views of a scene to reliably render high-quality novel views. Instead, we propose an algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields. We extend traditional plenoptic sampling theory to derive a bound that specifies precisely how densely users should sample views of a given scene when using our algorithm. In practice, we apply this bound to capture and render views of real world scenes that achieve the perceptual quality of Nyquist rate view sampling while using up to 4000x fewer views. We demonstrate our approach's practicality with an augmented reality smartphone app that guides users to capture input images of a scene and viewers that enable realtime virtual exploration on desktop and mobile platforms.
1 INTRODUCTION
The paper targets immersive, close-distance virtual experiences of real scenes, where dense sampling is infeasible and existing sparse-view methods provide little sampling guidance. It introduces a plenoptic-sampling-based deep learning pipeline with a theoretical sampling bound and practical demonstrations.
- Motivation: 2.5 million images per square meter are required for Nyquist sampling of one 0.5-meter scene example, making dense capture intractable.The example assumes a mobile phone camera with a 64° field of view and 1-megapixel rendering.
- Motivation: Sparse-view methods often abandon plenoptic sampling, leaving users to determine acceptable input-view patterns through trial and error.This sacrifices rigorous reasoning about how sampling affects performance.
- Approach: The method promotes each source view into a layered local scene representation, then synthesizes novel views by blending adjacent representations.The layered representation is based on multiplane images and renders only a limited local range from each source view.
- Approach: The paper extends plenoptic sampling theory to prescribe how densely users must capture a scene for reliable rendering.This provides a scene-dependent sampling requirement rather than relying on ad hoc capture patterns.
- Results: 642 ≈4000× fewer images achieve the same perceptual quality as Nyquist view sampling, with the reduction governed quadratically by the number of predicted planes up to field-of-view limits.The theoretical bound is empirically validated and applied to novel-view rendering.
- Practical demonstration: The system is demonstrated through an augmented-reality capture app and realtime desktop and mobile viewers for virtual exploration.The app guides users toward input images that satisfy the derived sampling guidelines.
2 RELATED WORK
Prior image-based rendering methods span light-field sampling, geometry-based reconstruction, and deep learning, but face trade-offs in sampling density, scene assumptions, consistency, or robustness. The paper combines local layered representations with deep learning while retaining prescriptive sampling analysis.
- Plenoptic sampling: Traditional light-field rendering uses regular-grid image samples, while plenoptic theory provides a way to compute sampling density from scene depths and optical effects.Non-Lambertian materials and occlusions increase the light field’s spectral support.
- Plenoptic sampling: Prescriptive sampling is a practical advantage of plenoptic methods because acquisition systems can use calculated view densities for high-quality rendering.The paper extends this principle to a deep-learning-based view synthesis strategy.
- Geometry-based IBR: Geometry-based IBR methods estimate global or local geometry, then reproject and blend nearby views, with local approaches avoiding expensive global mesh estimation.Layered geometry can also represent non-Lambertian reflectance.
- Geometry-based IBR: Smoothing geometry across large viewpoint neighborhoods can prevent accurate rendering of view-dependent specular effects because their apparent depth changes with viewpoint.This limitation is especially relevant for non-Lambertian scenes.
- Deep learning for view synthesis: Several deep view-synthesis methods predict depth or interpolate views, but sparse-view handling or viewpoint consistency remains limited.Per-view local-geometry prediction can produce inconsistent renderings across smoothly varying viewpoints, while some angular methods cannot handle sparse sampling.
- Deep learning for view synthesis: The paper adopts MPIs as local light-field representations, predicts one per input view, and trains end-to-end through blending for larger-baseline interpolation.Its 3D CNN dynamically adjusts depth-plane count with input sampling rate and needs synthetic data plus a small real fine-tuning set.
3 THEORETICAL SAMPLING ANALYSIS
The method interprets per-view MPI layers as depth-range light-field samples and extends plenoptic sampling theory to derive practical view-spacing bounds that handle occlusions and finite fields of view.
- Method: The method promotes each sampled view to a layered scene representation and interprets the resulting set as a specific form of light-field sampling.Each MPI contains D depth layers and supports continuously valued local novel views through alpha compositing.
- Sampling-rate reduction: The required number of input views decreases quadratically with the number of predicted planes, up to limits imposed by camera field-of-view overlap.In two viewing directions, the reduction becomes D^2 relative to traditional Nyquist sampling.
- Without occlusions: Traditional plenoptic sampling permits a D-fold increase in camera spacing when light fields are separately sampled across D depth ranges.The tighter per-range spectral supports allow distinct reconstruction filters for each depth range.
- Occlusions: Occlusions expand Fourier support, so summing separately reconstructed layer spectra cannot recover the full scene light field.The union of the layer supports is smaller than the support of the occluded light field.
- Occlusions: Alpha compositing depth-range light fields from back to front restores the full Fourier support while preserving the D-fold camera-spacing increase.The method uses predicted per-layer opacities to respect occlusions during compositing.
- Image-space interpretation: The practical spacing bound requires adjacent views to keep the closest scene point’s maximum disparity below min(D,W/2) pixels.The overall interval must satisfy both the disparity constraint and the finite-field-of-view overlap constraint.
4 PRACTICAL VIEW SYNTHESIS PIPELINE
The pipeline expands each captured view into a local MPI light field, then blends nearby MPI renderings to synthesize novel views. Accumulated-alpha weighting selects non-occluded content and improves rendering of complex reflectance.
- Local light-field construction: A CNN promotes each captured input image to an MPI, and nearby MPI renderings are blended to reconstruct novel views.Each MPI represents a local range of views; blending expands the effective light field across the captured view region.
- MPI prediction: The prediction network uses a reference view and four nearest neighbors, reprojected into D disparity-sampled planes forming five plane-sweep volumes.The volumes have size H × W × D × 3 and are processed jointly by the prediction network.
- MPI prediction: A fully 3D-convolutional architecture supports variable plane counts D, allowing view and disparity sampling densities to satisfy the sampling bound.The architecture is convolutional along height, width, and depth.
- Continuous view reconstruction: Accumulated-alpha weighting normalizes blending weights so MPI renderings contribute non-occluded content and fill regions occluded in other views.The resulting output is fully opaque, while simple RGB averaging retains stretching or ghosting artifacts.
- Continuous view reconstruction: Blending multiple MPIs approximates non-Lambertian light fields by combining local representations whose specularities occupy different virtual depths.This models curved specularity trajectories more effectively than a single MPI.
5 TRAINING OUR VIEW SYNTHESIS PIPELINE
The pipeline is trained on synthetic and real natural-scene imagery so it can learn view and depth distributions while adapting to real-world reflectance and pose errors. Training first uses single-MPI rendering, then full accumulated-alpha blending and real data fine-tuning.
- Training data: Training combines synthetic renderings with real images to model expected view and depth distributions and generalize to real lighting, reflectance, and pose errors.Synthetic data provides scale and controlled distributions; real data supplies real-world effects and estimation imperfections.
- Training data: Synthetic training views use random target baselines up to 128 pixels of disparity followed by random 3D camera perturbations.The synthetic datasets are SUNCG and UnrealCV, spanning simple interiors and detailed non-Lambertian environments.
- Training procedure: Each training step predicts two MPIs from separate five-view inputs, renders a held-out target from both, and blends the renderings using accumulated alpha.The target-view image reconstruction loss supervises the complete blending pipeline.
- Training procedure: The network is pretrained for 500k iterations with one MPI, trained for 100k iterations with two-MPI blending, and fine-tuned on real data for 10k iterations.Training uses a VGG perceptual loss and Adam optimization.
6 EXPERIMENTAL EVALUATION
Experiments validate the sampling bound and show high-quality novel-view rendering under substantial undersampling, including complex non-Lambertian scenes. The method outperforms light-field, mesh-based, and layered-view baselines, while occlusions limit the most extreme sampling regime.
- Sampling theory validation: The method renders Nyquist-level perceptual quality with up to dmax = 64 pixels of disparity between input views when MPI plane count matches the maximum disparity.Figure 8 evaluates plane counts D = 8, 16, 32, 64, and 128 across sampling rates.
- Sampling theory validation: Occlusions limit Nyquist-quality reconstruction at 128 pixels of disparity because sparse views reduce depth evidence and require hallucination of unseen occluded content.This is presented as the authors’ explanation for the failure to match Nyquist quality at that sampling rate.
- Sampling theory validation: Adding planes beyond the maximum pixel disparity provides limited value once the sampling bound is satisfied.At 32 pixels of disparity, LPIPS error decreases through 32 planes but remains constant from 32 to 128 planes.
- Comparison to prior methods: The method produces superior renderings, particularly for non-Lambertian effects, without artifacts reported for competing methods.Qualitative comparisons include complex geometry and highly non-Lambertian scenes.
- Ablation studies: Blending multiple MPIs outperforms a single MPI and unweighted average blending, whose limitations include restricted field of view, depth discretization, and ghosting in disoccluded regions.Accumulated-alpha blending prevents artifacts by suppressing content from occluded regions.
7 PRACTICAL USAGE
The method provides practical capture guidelines, real-time rendering, and apps for sampling and exploring real-world scenes. Its main constraints include preprocessing time, ambiguous textures or motion, and poor scaling to higher image resolutions.
- 7.1 Sampling Guidelines: The sampling guideline lets users choose rendering resolution W and sampled-view count N for Nyquist-level perceptual quality given the view-plane extent and closest scene depth.The guideline is expressed through camera field of view, view-plane side length, and maximum disparity.
- 7.1 Sampling Guidelines: The empirical maximum disparity for the deep-learning pipeline is 64 pixels.This value is substituted into the theoretical sampling bound to obtain practical capture requirements.
- 7.2 Asymptotic Rendering Time and Space Complexity: Users can trade capture time, rendering time, and storage by selecting image resolution and sampled-view count within the sampling guideline.Capture time is O(N), while increasing N allows fewer planes per MPI and reduces per-MPI rendering time.
- 7.3 Capture App: The iOS ARKit app uses scene depth and a selected view-plane size to guide users through automatic image capture at prescribed sampling locations.With W fixed at 500, the app overlays the next sampling point and records an image when the phone reaches it.
- 7.4 Preprocessing: The capture pipeline requires COLMAP for camera poses because ARKit poses are not accurate enough for acceptable results.COLMAP takes about 2–6 minutes for sets of 20–30 input images.
- 7.4 Preprocessing: About 10 minutes of preprocessing produces MPIs and camera poses before rendering 1-megapixel novel views.MPI prediction takes approximately 0.5 seconds for a small MPI or 12 seconds for a larger MPI on an Nvidia GTX 1080Ti GPU.
- 7.6 Limitations: Ambiguous or repetitive textures and moving scene content can produce floating or blurred patches when the MPI network assigns opacity to incorrect layers.More input views or stronger learned global geometry priors could potentially reduce these artifacts.
- 7.6 Limitations: Higher image resolutions are difficult because layered representations have cubic image-width complexity and require CNNs with larger receptive fields.The paper suggests multiresolution CNNs, hierarchical volume representations, or more compact layered scene representations as possible directions.
8 CONCLUSION
The paper concludes that its layered local light-field method provides practical, reliable view synthesis for complex real-world scenes while matching dense Nyquist-quality sampling with far fewer inputs.
- 8 CONCLUSION: The method works reliably for complex real-world scenes, including non-Lambertian materials, by blending nearby layered local light-field representations in real time.It combines learned layered representations with a theoretical sampling bound for high-fidelity rendering.
- 8 CONCLUSION: ≈4000× fewer input images achieve the perceptual quality of dense Nyquist-rate view sampling.The paper reports results across more than 60 diverse and complex real-world scenes.
A BASELINE METHODS IMPLEMENTATION DETAILS
The baseline implementations adapt several view-synthesis methods for comparison under matched input and depth-plane conditions.
- Baseline Methods: Soft3D is implemented from its paper description and receives the same five input images used to compute each MPI.Its vote volumes are aggregated across 25 viewpoints before rendering novel views.
- Baseline Methods: BW Deep uses a larger CNN matching the proposed method, accepts five images, and varies plane count instead of fixing D = 100.For quantitative results, D is set to dmax.
- Baseline Methods: ULR estimates dense depths and a global triangle mesh from all input images, then blends reprojected input images using heuristic weights.The implementation uses COLMAP multiview stereo and screened Poisson surface reconstruction.
- Baseline Methods: LFI reconstructs novel views from bilinearly interpolated samples on regular grids and blends five nearby reprojected views for unstructured data.The reprojected views use the same blending weights as the proposed method.
- Baseline Methods: Plane-sweep methods use dmax depth planes for Soft3D, BW Deep, and all versions of the proposed method.This provides a shared depth-plane setting across those comparisons.
B NETWORK ARCHITECTURE
The network outputs opacity and blending information for each voxel, forming the layered scene representation used by the method.
- Network Architecture: The final network layer outputs five channels: one sigmoid opacity channel and five softmax blending weights derived from the other four channels plus an all-zero channel.The blending weights generate the MPI color channels for each voxel.