PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Rays for Occlusion-aware Image-based Rendering

Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, Wenping Wang

arXiv:2107.13421v2cs.CVcs.GR

TL;DR

Generalizable radiance-field methods can mix inconsistent features from input views that are occluded from a 3D point. NeuRay predicts point visibility for occlusion-aware feature aggregation, refines it with a consistency loss during finetuning, and reports strong results on unseen scenes and after finetuning. Its NeRF-like density formulation is computationally impractical because visibility requires repeated density evaluations along each input ray.

  • Problem

    Generalizable radiance-field methods struggle when occlusions make multi-view image features inconsistent, because invisible-view features interfere with radiance-field construction.

  • Method

    NeuRay uses pixel-aligned features on input rays to predict 3D-point visibility, initializes them from multi-view stereo geometry, and refines them with a scene-specific consistency loss.

  • Results

    NeuRay outperforms other generalization methods without scene-specific optimization and achieves superior finetuned results to other generalization models, with better rendering than NeRF reported overall.

  • Takeaways & Limitations

    Visibility-aware feature aggregation supports high-quality novel-view rendering on unseen scenes and with only a few scene-specific finetuning steps.

  • Takeaways & Limitations

    A NeRF-like density parameterization is computationally impractical because computing visibility requires Kr density evaluations for each input view-point pair.

Abstract

from arXiv · show

We present a new neural representation, called Neural Ray (NeuRay), for the novel view synthesis task. Recent works construct radiance fields from image features of input views to render novel view images, which enables the generalization to new scenes. However, due to occlusions, a 3D point may be invisible to some input views. On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction. To solve this problem, we predict the visibility of 3D points to input views within our NeuRay representation. This visibility enables the radiance field construction to focus on visible image features, which significantly improves its rendering quality. Meanwhile, a novel consistency loss is proposed to refine the visibility in NeuRay when finetuning on a specific scene. Experiments demonstrate that our approach achieves state-of-the-art performance on the novel view synthesis task when generalizing to unseen scenes and outperforms per-scene optimization methods after finetuning.

1. Introduction

Generalizable radiance-field methods struggle with inconsistent features caused by occlusions, while NeuRay predicts visibility to enable occlusion-aware feature aggregation and improves rendering across generalization and finetuning settings.

  • Motivation: Generalizable methods construct radiance fields from multi-view image features but struggle to distinguish occlusion-induced inconsistency from non-surface points.This ambiguity can lead to rendering artifacts and incorrect density estimation.
  • NeuRay: NeuRay represents each input ray with pixel-aligned features that predict whether 3D points at specific depths are visible.The predicted visibility separates occlusion-caused inconsistency from non-surface-caused inconsistency.
  • NeuRay: Visibility estimation in unseen scenes uses multi-view stereo geometry to initialize NeuRay features before visibility guides radiance-field construction.The design addresses a chicken-and-egg dependency between surface estimation and visibility estimation.
  • Finetuning: A consistency loss lets NeuRay and the constructed radiance field refine one another during scene-specific finetuning.NeuRay memorizes geometry predicted by the radiance field, which then improves radiance-field construction.
  • Results: Experiments show strong unseen-scene rendering without scene-specific optimization, superior finetuned results to other generalization models, and approximately 3-second rendering for 800 × 600 images.The reported evaluation covers the NeRF synthetic, DTU, and LLFF datasets.

2. Related works

NeuRay belongs to image-based and generalizable rendering approaches that construct radiance fields from input views, while using a ray-based representation to model occlusion-aware visibility.

  • Image-based rendering: Image-based rendering methods blend input images with geometry proxies, while light-field methods offer photorealism but typically have limited renderable ranges.Multi-view stereo can provide 3D proxy geometry to extend the renderable range.
  • Image-based rendering: Deep-learning image-based rendering replaces handcrafted components with CNNs, but common methods are sensitive to estimated depth-map quality.NeuRay also uses cost volumes or estimated MVS depth, while supporting training from scratch and scene finetuning.
  • Generalization volume rendering: Generalizable rendering methods construct radiance fields on-the-fly from input views instead of optimizing a separate NeRF network for each scene.NeuRay differs by using a ray-based representation for occlusion inference.
  • Generalization volume rendering: NeuRay predicts visibility for sampled points during volume rendering, enabling occlusion-aware aggregation of input-view features before computing output colors.The pipeline samples points on the test ray, estimates sample alpha values and colors, and accumulates them by volume rendering.

3. Method

NeuRay renders novel views by predicting input-view visibility and using it to guide feature aggregation before volume rendering. Its visibility representation is initialized from MVS geometry and can be finetuned through trainable intermediate features.

  • 3.1. Volume rendering: NeuRay samples points along test rays and uses volume rendering to accumulate their predicted colors and alpha values into novel-view pixels.The rendered color is formed from sample-point colors weighted by hitting probabilities.
  • 3.2. Occlusion-aware radiance field construction: Generalization methods aggregate local image features projected from input views to each 3D point, then decode the aggregate into alpha and color.NeuRay augments this aggregation with visibility values for the input views.
  • 3.3. NeuRay representation: NeuRay predicts an input ray’s visibility at depth z from a pixel-aligned feature vector in a visibility feature map G.The visibility function is non-increasing and bounded between 0 and 1.
  • 3.3. NeuRay representation: Visibility is parameterized as one minus a CDF represented by a mixture of logistics distributions whose parameters are decoded from G by an MLP.The mixture improves performance for semi-transparent surfaces or surface edges.
  • 3.4. NeuRay initialization: For unseen scenes, MVS cost volumes or patch-match depth maps are processed by a CNN to produce G, which supplies visibility for occlusion-aware aggregation and rendering.The pipeline estimates geometry, predicts visibility features, aggregates local features with visibility, and volume-renders the result.
  • 3.5. Finetune with NeuRay: During finetuning, the intermediate feature map G′ is initialized by an MVS-based network, made trainable, and optimized with the remaining NeuRay parameters after the initialization network is discarded.This trainable representation supports scene-specific memorization of geometry and improves radiance-field construction.

4. Experiment

Experiments evaluate NeuRay across object and forward-facing datasets, generalization and finetuning settings, ablations, convergence, and rendering speed. NeuRay consistently improves occlusion-aware rendering, with especially strong gains under sparse views and scene-specific optimization.

  • Experiment settings: The evaluation uses NeRF synthetic, DTU, and LLFF datasets with PSNR, SSIM, and LPIPS under generalization and finetuning settings.Generalization methods are pretrained on training scenes and tested on unseen scenes; finetuning uses each test scene’s input views, while NeRF is trained from scratch.
  • Comparison with baselines: NeuRay outperforms other generalization models on unseen scenes and surpasses baselines on object datasets after finetuning, while matching IBRNet on LLFF.On LLFF, dense input views reduce the impact of occlusion; sparse 360° views in the Synthetic NeRF dataset create more severe inconsistency for IBRNet.
  • Ablation studies: Visibility initialized by cost volumes significantly improves rendering quality, whereas adding estimated depth alone provides only slight improvement.The ablation compares IBRNet, image-feature aggregation, depth augmentation, and NeuRay visibility.
  • Ablation studies: Mixture logistics distributions improve rendering near abrupt depth changes, and the consistency loss further improves NeuRay’s per-scene optimization quality.The consistency loss enables geometry memorization between NeuRay and the constructed radiance field.
  • Analysis: NeuRay produces higher-quality renderings than baselines at equal training steps and already yields high-quality images after 10k steps.NeRF and MVSNeRF remain blurred, while IBRNet produces occlusion artifacts at 10k steps.

5. Conclusion

The conclusion presents NeuRay as an occlusion-probability representation for efficient visibility estimation and visibility-aware radiance-field construction. Experiments show high-quality novel-view rendering without scene training or with only a few finetuning steps.

  • 5. Conclusion: NeuRay represents scenes with occlusion probabilities on input rays and efficiently estimates visibility from arbitrary 3D points to input views.Visibility guides multi-view feature aggregation during radiance-field construction.
  • 5. Conclusion: Experiments on DTU, NeRF synthetic, and LLFF demonstrate high-quality rendering without scene training or with only a few finetuning steps.

A.1. Comparison between NeRF [33] and NeuRay

NeRF parameterizes ray probabilities through point densities, whereas NeuRay parameterizes occlusion probabilities and derives visibility, hitting probability, and alpha values from them.

  • NeRF’s parameterization: NeRF computes point densities with an MLP, derives alpha values from density and interval length, and bases ray probabilities on those densities.Visibility is accumulated from the ray origin to each sampled point.
  • NeuRay’s parameterization: NeuRay predicts occlusion probability on input rays, then computes hitting probability and visibility from that parameterization.This reverses the direction of NeRF’s density-based parameterization.
  • NeuRay’s parameterization: NeuRay derives each input-ray alpha value from hitting probability divided by visibility, using the relationship between these probabilities.The resulting alpha is computed from changes in occlusion probability and visibility.
  • Test-ray computation: For test rays, NeuRay forms alpha values by visibility-weighted aggregation of alpha values from input rays before computing hitting probabilities.Only visible input rays affect the test-point alpha values.

A.2. Discussion about visibility from density

Computing visibility from volume density is impractical because it requires evaluating density samples along every input ray from each view to each 3D point.

  • Discussion about visibility from density: Visibility estimation through volume density requires K_r network forward passes for one input view and one 3D point.With one test ray, 8 neighboring views, 128 test-ray samples, and 32 samples per input ray, this requires 8×128×32 evaluations.

A.3. Memorization interpretation in finetuning

During finetuning, NeuRay memorizes scene geometry and visibility on input views, allowing later rendering steps to infer occlusions more accurately. The consistency loss links geometry predicted by the radiance field back to NeuRay, unlike existing methods that remain occlusion-oblivious.

  • A.3. Memorization interpretation in finetuning: The consistency loss makes NeuRay memorize surfaces predicted by the constructed radiance field during finetuning.This connects the scene geometry represented by the radiance field and NeuRay.
  • A.3. Memorization interpretation in finetuning: NeuRay predicts hitting probabilities for a pseudo test ray and uses the consistency loss to make its visibility features reproduce those predictions.View A is rendered from neighboring views B, C, and D during the illustrated training step.
  • A.3. Memorization interpretation in finetuning: Memorized visibility on one input view lets NeuRay identify points invisible to that view when rendering a later pseudo test view.The figure illustrates visibility memorized on view A being used for occlusion inference when rendering view D.
  • A.3. Memorization interpretation in finetuning: Unlike existing methods, NeuRay refines its scene representation so feature aggregation can account for scene occlusions during finetuning.Existing methods adjust network parameters but remain oblivious to occlusions in the scene.

A.4. Training details and architecture

NeuRay uses a pretrained MVSNet cost volume to initialize its representation, then combines feature aggregation and color blending networks for rendering. Training uses fixed MVSNet weights, Adam optimization, and a depth loss during pretraining.

  • Architecture: The cost volume uses 3 neighboring input views at one-quarter of the original image height and width, with a pretrained MVSNet.MVSNet weights are fixed during renderer training because of GPU memory limitations.
  • Architecture: The feature aggregation architecture follows IBRNet, with alpha and color networks operating on aggregated multi-view features.The alpha network is denoted A and the color blending network is denoted B.
  • Architecture: The alpha network assigns large alpha values when visible input-view features are consistent, while the color network blends colors from visible views.Visibility information determines which projected input colors contribute to each point’s rendered color.
  • Training details: During pretraining, a depth loss encourages the mean of the first logistic distribution to match input depth estimated from the cost volume or depth maps.The input depth is represented by z_in.
  • Training details: Training uses Adam with 512 randomly sampled rays per step, 400k generalization steps, and 200k finetuning steps.The initial learning rates are 2e-4 for generalization and 1e-4 for finetuning, with halving schedules.

A.5. More comparison with baselines

NeuRay remains effective with sparse input or working views and recovers scene details more clearly than NeRF at equal training steps. Its advantage is linked to explicit occlusion awareness when feature inconsistency becomes more severe.

  • A.5. More comparison with baselines: On LLFF, NeuRay achieves performance similar to IBRNet after finetuning because dense forward-facing views already make many points visible to numerous inputs.This dense-view setting reduces the benefit of explicit occlusion handling.
  • A.5. More comparison with baselines: NeuRay performs better than IBRNet with only 2 working views on LLFF after scene-specific finetuning.The table compares PSNR for Nw = 2, 4, and 8 working views.
  • A.5. More comparison with baselines: As working views decrease, occlusion-driven feature inconsistency becomes more severe, causing IBRNet’s performance to degenerate while NeuRay remains occlusion-aware.The authors attribute NeuRay’s sparse-view advantage to its constructed visibility representation.
  • A.5. More comparison with baselines: NeuRay recovers details more clearly than NeRF with the same training steps on the given scene.The comparison includes subtle scene details such as brick surfaces and Lego textures.

A.6. Initialize from estimated depth maps

NeuRay can be initialized from estimated depth maps, whose optimization improves cross-view consistency and corrects erroneous depth values. Its direct rendering is structurally meaningful, but the method remains bounded by visibility coverage and geometry quality.

  • A.6. Initialize from estimated depth maps: Initialization from COLMAP estimated depth maps produces good rendering results, and is better than cost-volume initialization on DTU.The authors attribute the DTU improvement to more accurate COLMAP reconstruction.
  • A.6. Initialize from estimated depth maps: NeuRay performs better than IBRNet as input views become sparser, although performance decreases with fewer views.This experiment evaluates 100, 75, and 50 input views on Lego.
  • A.6. Initialize from estimated depth maps: NeuRay optimization refines coarse depth maps into more consistent depth maps and corrects erroneous depth values rather than merely discarding them.Initialized NeuRay already produces reasonable depth maps, while finetuning rectifies errors.
  • A.6. Initialize from estimated depth maps: Optimizing both the raw visibility features and network parameters is necessary for the best ablation results.Optimizing either component alone leads to inferior results.
  • A.11. Direct rendering from NeuRay: Direct rendering fits each point’s color as a spherical-harmonic function, weighting color differences by hitting probabilities so occluded rays do not interfere.The fitted function also represents anisotropic colors.
  • A.11. Direct rendering from NeuRay: Direct NeuRay rendering produces images with correct structures and details despite having no rendering-loss supervision.The constructed radiance field remains the primary comparison target in the direct-rendering experiment.
  • A.12. Limitations: NeuRay cannot render regions invisible to all selected working views, and feature matching may fail in textureless or cluttered regions.More input or working views, or improved feature matching, may alleviate these issues.
Loading 2107.13421v2…