Source-linked AI summary
IBRNet: Learning Multi-View Image-Based Rendering
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, Thomas Funkhouser
TL;DR
Novel view synthesis must produce realistic images from limited posed views while addressing the dense-view, proxy-geometry, and per-scene optimization requirements of earlier approaches. IBRNet generates continuous radiance fields on the fly from multiple source views using an MLP and ray transformer, and volume-renders the result. It improves upon methods that generalize to novel scenes and remains competitive with single-scene neural rendering methods after fine-tuning.
Problem
Novel view synthesis methods often require dense input views, proxy geometry, or lengthy optimization for each new scene, limiting their applicability to sparse-view and real-world settings.
Method
IBRNet learns a generic view interpolation function that predicts continuous colors and densities from multiple source views using an MLP and ray transformer, then renders rays with volume rendering.
Results
IBRNet improves upon state-of-the-art novel view synthesis methods designed to generalize to unseen scenes and is competitive with state-of-the-art neural rendering methods after per-scene fine-tuning.
Takeaways & Limitations
The method combines image-based rendering and neural scene representations to achieve high-quality novel-view synthesis without precomputed geometry or expensive optimization for each new scene.
Takeaways & Limitations
With very sparse training views and complex geometry, local source-view information may be insufficient, allowing globally optimized radiance fields such as NeRF to perform better.
Abstract
from arXiv · showhide
We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. The core of our method is a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations (3D spatial locations and 2D viewing directions), drawing appearance information on the fly from multiple source views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering (IBR), and allows us to render high-resolution imagery. Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes. We render images using classic volume rendering, which is fully differentiable and allows us to train using only multi-view posed images as supervision. Experiments show that our method outperforms recent novel view synthesis methods that also seek to generalize to novel scenes. Further, if fine-tuned on each scene, our method is competitive with state-of-the-art single-scene neural rendering methods. Project page: https://ibrnet.github.io/
1. Introduction
IBRNet combines image-based rendering with neural scene representations to synthesize novel views from multiple nearby source images without scene-specific optimization or proxy geometry. It generalizes to unseen scenes and can be fine-tuned for competitive single-scene rendering.
- IBR methods support high-resolution rendering but typically require dense input views or difficult-to-estimate proxy geometry.These requirements can lead to rendering artifacts.
- Neural scene representations can model complex scenes faithfully but usually require lengthy optimization for each new scene.This per-scene optimization limits their value for many real-world applications.
- IBRNet generates a continuous scene radiance field on the fly from multiple source views while jointly reasoning about density, occlusion, visibility, and color blending.The method therefore avoids scene-specific optimization and precomputed proxy geometry.
- The architecture aggregates source-view features along rays, predicts densities with a ray transformer, and computes view-dependent colors before volume rendering.The MLP extracts and aggregates latent 2D features, while the ray transformer reasons across samples along the ray.
- IBRNet renders high-resolution photo-realistic views for unseen complex scenes and improves upon state-of-the-art methods designed to generalize to new scenes.Its differentiable pipeline is trained end-to-end using multi-view images.
- After per-scene fine-tuning, IBRNet is competitive with state-of-the-art neural rendering methods.The contribution includes a fine-tuning procedure for single-scene quality.
2. Related work
Prior novel view synthesis methods use image-based warping and blending, discrete volumetric representations, or per-scene neural representations. These approaches trade off rendering quality, input requirements, storage, generalization, and scene-specific optimization.
- Image-based rendering synthesizes novel views by warping, resampling, or blending reference views, with weights based on ray-space proximity or proxy geometry.Later work improves proxy geometry, optical-flow correction, and soft blending.
- Discrete volumetric methods store learned representations in voxel grids or multi-plane images and render views through alpha compositing.They can generalize across scenes but must explicitly process and store discretized representations.
- Neural scene representations use neural networks to model scene shape and appearance from multiview observations.NeRF optimizes a continuous 5D radiance field and achieves impressive novel view synthesis results.
- NeRF and related neural scene representations require optimization for each new scene, with NeRF taking hours or days to converge.Concurrent methods seek to generalize NeRF, particularly for very sparse input views.
3. Method
IBRNet synthesizes a target view by selecting nearby source views, predicting colors and densities at continuous 5D locations, and compositing them with volume rendering. Its multi-view aggregation and ray transformer provide visibility reasoning and long-range context without scene-specific optimization.
- The pipeline selects nearby source views, extracts dense features, predicts continuous 5D colors and densities, and composites samples along each ray with volume rendering.The system is trained end-to-end using reconstructed image colors.
- IBRNet aggregates source-view image colors, features, and viewing directions for each sampled 3D point before predicting ray colors and density features.The source-view inputs are projected to query locations and sampled through bilinear interpolation.
- Weighted pooling combines multi-view feature means and variances into density features, improving the network’s ability to handle occlusions compared with direct average or max pooling.
- For color prediction, IBRNet predicts blending weights from multi-view features and relative viewing directions, then uses a weighted combination of source colors.Directly regressing the color performed worse than predicting blending weights.
- A ray transformer lets samples attend to one another along the ray before predicting scalar densities, adding contextual geometric reasoning with only a small computational overhead.The module uses positional encoding and self-attention over samples ordered from near to far.
- The method uses nearby views as a limited working set, which can cause temporally inconsistent densities and flickering when that set changes abruptly along smooth camera paths.Distance-weighted pooling is used to smooth changes and improve synthesis stability and quality.
4. Experiments
Experiments evaluate pretrained and fine-tuned IBRNet across synthetic and real scenes, including qualitative quality, ablations, source-view sparsity, and inference cost. The method generalizes well to novel scenes, remains competitive after fine-tuning, degrades reasonably with sparser inputs, and uses substantially fewer FLOPs than NeRF.
- Experimental Settings: The evaluation uses synthetic objects and real scenes, with 10 training-set source views sampled for each test view and PSNR, SSIM, and LPIPS metrics.Training data combines synthetic Google Scanned Objects with RealEstate10K, Spaces, and cellphone captures; evaluation includes DeepVoxels, NeRF synthetic, and Real Forward-Facing scenes.
- Results: The pretrained model consistently outperforms LLFF on all test scenes, while fine-tuned IBRNet is competitive with state-of-the-art neural rendering methods.After fine-tuning, it outperforms NeRF on Diffuse Synthetic 360° and Real Forward-Facing, but has lower PSNR and SSIM on Realistic Synthetic 360°.
- Results: On Real Forward-Facing data, fine-tuned IBRNet achieves substantially better SSIM and LPIPS than NeRF, producing more photo-realistic synthesized images.Qualitatively, it better recovers fine geometry, appearance details, thin structures, partially occluded origins, and glass reflections than the compared methods.
- Ablations and Analysis: Removing the ray transformer causes inaccurate densities, black-hole artifacts, and blurriness, while removing view directions reduces reproduction of view-dependent effects such as specularities.Direct color regression is modestly worse than blending source-image colors.
- Ablations and Analysis: IBRNet degrades reasonably as source views become sparser, based on PSNR measurements for pretrained and fine-tuned models under progressively subsampled inputs.Views are subsampled by factors of 2, 4, 6, 8, and 10 from the original 479-view training sets.
- Ablations and Analysis: IBRNet requires less than 20% of NeRF’s FLOPs while achieving comparable performance, with computation depending on the local source-view working set.Its FLOPs scale roughly linearly with the number of source views and do not grow with overall scene scale.
5. Conclusion
The paper presents a multi-view image-based rendering framework that blends nearby image pixels using network-inferred weights and volume densities. It combines IBR and NeRF-style rendering to achieve high-quality novel views without precomputed geometry, discretized volumes, or expensive per-scene training.
- Conclusion: IBRNet synthesizes novel views by blending pixels from nearby images with weights and volume densities inferred by an MLP and ray transformer.The framework uses volume rendering to produce the final image.
- Conclusion: The approach produces state-of-the-art rendering quality on complex scenes without requiring precomputed geometry, expensive discretized volumes, or expensive training for each new scene.These properties combine advantages associated with image-based rendering and neural radiance fields.
A. Additional implementation details
IBRNet combines shared-weight feature extraction with ray-based density and color prediction, while supporting practical training and rendering constraints. Its training pipeline handles invalid projections and uses scene-specific fine-tuning when needed.
- Feature extraction: The feature extractor is U-Net-like, adapts ResNet34, replaces Batch Normalization with Instance Normalization, and uses strided convolutions instead of max-pooling.
- Feature extraction: The 64-dimensional output feature map is split into two 32-dimensional maps for the coarse and fine IBRNet stages.
- IBRNet architecture: IBRNet aggregates multi-view features to estimate density and visibility along rays, then blends source-view colors using predicted weights.
- IBRNet architecture: The model is invariant to source-view permutations and supports variable numbers of source views and ray samples.
- Training details: During training, invalid projections discount source views, points unseen by every source view receive zero density, and rays with fewer than three valid density samples are ignored.
- Training details: Pre-training uses eight V100 GPUs and takes about a day, while reported Real Forward-Facing fine-tuning takes about six hours per scene on one V100 GPU.
B. Additional qualitative results
Qualitative results show that the ray transformer improves challenging-region synthesis, fine-tuning improves both images and proxy geometry, and sparse views or complex geometry can still cause failures.
- Ray transformer: The ray transformer significantly improves synthesis quality, especially near occlusion boundaries.
- Geometry visualization: The pretrained model produces reasonable proxy geometry, while per-scene fine-tuning improves both synthesized images and the underlying geometry.
- Realistic Synthetic 360°: On Realistic Synthetic 360°, comparisons show pretrained and fine-tuned results alongside ground truth, with a failure case involving sparse source views and complex geometry.
- Architecture overview: The architecture diagram organizes computation over N source views and M samples along each ray, including shared MLP processing and weighted pooling.
- Ray transformer: Without the ray transformer, severe black-hole artifacts appear near occlusion boundaries; the module eliminates these artifacts by improving surface-location inference.