Source-linked AI summary
NeX: Real-time View Synthesis with Neural Basis Expansion
Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, Supasorn Suwajanakorn
TL;DR
Novel view synthesis needs representations that handle view-dependent effects while remaining practical for real-time rendering. NeX augments MPI with neural basis expansion and hybrid implicit-explicit modeling, achieving strong quality across benchmark and challenging datasets. The method is fast enough for real-time rendering, while retaining limitations on extreme viewpoints, hardest effects, exposure variation, and training requirements.
Problem
Traditional RGBα MPI represents diffuse, view-independent surfaces and struggles with reflections, refractions, and other view-dependent effects.
Method
NeX represents MPI pixel colors as neural linear combinations of basis functions and explicitly optimizes k0 to preserve fine detail.
Results
NeX achieves the best overall scores across major metrics on benchmark and Shiny datasets, with more than 1000× faster rendering than NeRF.
Takeaways & Limitations
Neural basis expansion enables NeX to capture complex view-dependent effects while supporting efficient real-time rendering on standard graphics hardware.
Takeaways & Limitations
NeX shows stack-of-cards artifacts at far viewpoints, cannot fully reproduce the hardest Shiny effects, and may require many input views and long training.
Abstract
from arXiv · showhide
We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects -- in real time. Unlike traditional MPI that uses a set of simple RGB$α$ planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forward-facing datasets as well as our newly-introduced dataset designed to test the limit of view-dependent modeling with significantly more challenging effects such as rainbow reflections on a CD. Our method achieves the best overall scores across all major metrics on these datasets with more than 1000$\times$ faster rendering time than the state of the art. For real-time demos, visit https://nex-mpi.github.io/
1. Introduction
Novel view synthesis must construct scenes from sparse images and render them from unseen perspectives, ideally with photorealistic real-time performance. NeX extends MPI to represent view-dependent effects and fine detail.
- Sparse-view synthesis must infer a scene representation and render it from unseen perspectives.
- Real-time, photorealistic rendering is needed to make applications such as virtual visits and 3D product viewing practical.
- Standard RGBα MPIs use parallel semi-transparent planes but are limited to diffuse surfaces with view-independent appearance.
- NeX models each MPI color as a viewing-angle function approximated by neural spherical-basis expansion instead of static color values.
- A hybrid implicit-explicit strategy stores high-frequency detail explicitly, improving fine detail and producing sharper results in fewer training iterations.
- NeX is evaluated on benchmark forward-facing datasets and the Shiny dataset, which contains challenging effects including rainbow reflections on a CD.
2. Related Work
Related work spans MPI representations, neural view synthesis, and light-field factorization. NeX combines MPI structure with neural basis-function factorization for view-dependent rendering.
- Learning MPIs: MPI methods represent scenes with parallel semi-transparent planes and have been extended to multiple images, single images, and learned gradient updates.
- Learning MPIs: Blending multiple view-independent MPIs can support time-varying appearance but limits applicable objects and scenes.
- View synthesis and interpolation: View synthesis methods address different sampling regimes, from dense light-field interpolation to sparse-view geometry and disocclusion inference.
- View synthesis and interpolation: Neural approaches include angle-specific pixel prediction, neural textures on meshes, latent codes in geometric structures, and neural bidirectional texture functions.
- View synthesis and interpolation: NeRF represents a 5D radiance field with an MLP that directly regresses volume density and RGB colors while incorporating viewing direction.
- Light field factorization: Light-field factorization approximates radiance as sums of products, whereas NeX models the factors with neural networks and learns the factorization through training.
- NeX overview: NeX’s overview uses one MLP for alpha and view-dependent coefficients and another for basis functions, then composites RGB values across MPI planes.
3. Approach
NeX extends multiplane images with view-dependent pixel colors represented by learned basis functions and combines implicit neural modeling with explicit detail parameters. Differentiable rendering and GPU shader implementation support real-time novel-view synthesis.
- 3.2. View-Dependent Pixel Representation: NeX represents each pixel color as a viewing-direction function approximated by a linear combination of learnable basis functions.The basis functions are shared globally, while each pixel stores RGB coefficients and alpha.
- 3.1. Original MPI Representation: MPI represents scenes as equidistant planar images along a reference viewing frustum, rendered after homography warping and alpha compositing.The planes may be placed in depth or inverse-depth space depending on the scene.
- 3.2. View-Dependent Pixel Representation: MPI’s standard RGBα representation is limited to diffuse or Lambertian appearance, excluding many reflection and refraction effects.A single MPI can simulate planar reflections to some extent by placing reflected content on one plane.
- 3.3. Modeling MPI with Neural Networks: NeX uses MLPs as deep priors to regress MPI parameters from pixel coordinates, avoiding noisy direct optimization that overfits training views.The paper reports superior results versus direct optimization and standard regularizers such as total variation.
- 3.4. Implicit-Explicit Modeling Strategy: Explicitly optimizing the base-color coefficient k0 reduces the burden on the neural network and produces sharper detail in fewer iterations.Other parameters remain predicted by MLPs, while k0 is optimized explicitly with a total-variation regularizer.
- 3.4. Implicit-Explicit Modeling Strategy: Sharing coefficient sets across every M planes improves speed and model compactness without significant visual-quality degradation when N and M are properly chosen.The shared coefficients exclude alpha values, which remain separate across planes.
4. Experiments
NeX is evaluated against MPI-based and neural view-synthesis methods across forward-facing datasets, including the challenging Shiny dataset, with ablations of basis functions and modeling choices. It achieves strong quality and real-time rendering while modeling view-dependent effects and fine detail.
- Runtime: 300 frames per second are achieved on an NVIDIA RTX 2080Ti, versus NeRF’s 55 seconds per frame on the same machine.For one rendered pixel, NeX uses 0.16 MFLOPs while NeRF uses 226 MFLOPs; training the example scene took around 18 hours on one NVIDIA V100.
- Evaluation setup: NeX is compared with DeepView, LLFF, NeRF, SRN, and other methods on Real Forward-Facing, Shiny, and Spaces datasets.The evaluation includes quantitative and qualitative comparisons across MPI-based and non-MPI-based approaches.
- Benchmark results: NeX produces the highest average scores across all 3 metrics on the Real Forward-Facing dataset.The metrics are PSNR, SSIM, and LPIPS; higher is better for PSNR and SSIM, while lower is better for LPIPS.
- Evaluation setup: The Shiny dataset contains challenging effects including rainbow reflections, refraction, metallic and ceramic reflections, sharp specular highlights, and thin structures.It was introduced to test view-dependent modeling beyond mostly diffuse benchmark scenes.
- Benchmark results: NeX outperforms NeRF on all 3 metrics on Shiny and produces higher average scores than DeepView on all metrics in the 12-view Spaces setup.On Shiny, qualitative results include rainbow reflections and reflected images, while some extremely sharp highlights remain failure cases.
- Ablations: Adding view-dependent modeling increases PSNR on all test scenes, while learned basis functions outperform fixed bases at the same coefficient count.Learned-basis scores peak around 6-9 coefficients and show signs of overfitting afterward.
5. Limitations & Failure Cases
NeX inherits MPI limitations and fails on the hardest view-dependent scenes, while training remains costly and may require more input views.
- NeX produces “stack of cards” artifacts when viewed too far from the center angle.Individual MPI planes can become visible from sufficiently oblique viewpoints.
- NeX cannot fully reproduce Shiny scenes containing light sparkles, extremely sharp highlights, or refraction through test tubes.The paper identifies these effects among its hardest failure cases.
- Exposure differences in training images that are not properly compensated may cause flickering in rendered output.
- Training the MPI still takes a long time and may require more input views to reproduce view-dependent effects.Learning with fewer input images is proposed as an interesting direction.
6. Conclusion
The paper concludes that neural basis expansion makes MPI effective for complex view-dependent effects while retaining efficient rendering and state-of-the-art quality.
- NeX combines MPI with neural basis expansion to capture complex view-dependent effects and compute efficiently on standard graphics hardware.
- Extensive evaluations on public datasets and the more challenging Shiny dataset demonstrate state-of-the-art quality.
- Neural basis expansion may apply to general light field factorization and efficient rendering for scene representations beyond MPI.
- Explicit optimization of some reflectance parameters and high-frequency texture can help recover fine detail in implicit neural representations.
A. Additional Implementation Settings
The supplementary implementation settings describe camera calibration, memory-aware pixel sampling during training, and the three metrics used for evaluation.
- Input images are calibrated with COLMAP using a simple-radial camera model, shared intrinsics, exhaustive SIFT matching, and bundle-adjustment refinement.Accurate camera poses and intrinsic parameters are described as crucial for the pipeline.
- Training samples a subset of image pixels each iteration to avoid exceeding GPU memory limits.Neighboring pixels are also sampled to compute image gradients in both directions.
- Evaluation uses PSNR, SSIM, and LPIPS, with PSNR and SSIM computed using scikit-image and LPIPS using a VGG variant.
B.1. Comparison on Real Forward-Facing Dataset
On the Real Forward-Facing dataset, supplementary results report per-scene comparisons and training times for scenes with different numbers of input photos.
- The Real Forward-Facing dataset contains 8 scenes, with per-scene results reported against the main-paper aggregate results.NeRF scores are computed from undistorted outputs using the estimated radial distortion parameter.
- Qualitative comparisons show that NeX achieves sharper fine detail on the Real Forward-Facing dataset.The comparison appears in the main-paper figure and supplementary video.
- 18 hours of training were required for Fern with 17 input photos, while Flower required 27 hours with 30 input photos.Both measurements used a single NVIDIA V100 and a 20-core Intel Xeon Gold 6248.
B.2. Comparison on Shiny Dataset
The Shiny evaluation uses the authors’ eight-scene dataset of challenging view-dependent effects and compares NeX with per-scene results reported in Table B.4.
- Results: The Shiny results provide a per-scene analysis of NeX’s performance on view-dependent effects that are more challenging than those in standard forward-facing datasets.The paper introduces Shiny specifically because benchmark datasets contain mostly diffuse scenes and simpler view-dependent effects.
- Results: NeX achieves better performance than NeRF on all metrics in every Shiny scene.The per-scene breakdown is provided in Table B.4, with full visual comparisons available in the supplementary webpage.
B.3. Comparison on Spaces Dataset
The Spaces comparison evaluates NeX against DeepView using the released DeepView outputs because its authors did not make the implementation publicly available.
- Evaluation setup: NeX is evaluated on the Spaces dataset and compared with DeepView using the latter’s released output results.The authors run NeX on Spaces and report a per-scene breakdown corresponding to Table 3 in the main paper.
- Evaluation setup: DeepView’s unavailable source code means the comparison uses its published outputs rather than a reproduced implementation.This constrains the comparison to the released results provided by the DeepView authors.
- Results: Table B.5 reports the per-scene Spaces comparison, while the supplementary webpage provides a full visual comparison.The passage identifies the table and supplementary material as the locations for quantitative and visual results.
B.4. Details for Types of Basis Ablation Study
The basis ablation examines alternative basis-function families for view-dependent modeling, including forward-facing parameterizations and spherical or hemispherical constructions, alongside qualitative and metric comparisons.
- Basis ablation: The ablation replaces NeX’s neural basis H⃗φ with alternative basis sets including Fourier, Taylor, and spherical harmonics.The experiment changes the basis functions in Algorithm 1 and compares their effects on reconstruction metrics.
- Forward-facing parameterization: For forward-facing scenes, the viewing angle is parameterized using vx and vy because vz is determined by them over a hemisphere.The relation is given as vz = 1 − vx^2 − vy^2, allowing the Fourier-style basis to use two parameters.
- Qualitative comparisons: Qualitative comparisons show NeX against NeRF on Shiny and against NSVF on the CD scene.The NSVF comparison is contextualized by its focus on object captures with tightly bounded volumes, unlike the CD setup.
- Datasets and results: The Shiny dataset contains eight scenes with varying image resolutions and input counts, and complex view-dependent scenes such as CD require more images.Per-scene results are reported in Table B.4, with Spaces results reported separately in Table B.5.
- Ablation metrics: The supplementary evaluation reports SSIM and LPIPS results for the same basis-function experiment shown with PSNR in the main paper.SSIM is higher-is-better, while LPIPS is lower-is-better.