PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoNeRF: Generalizing NeRF with Geometry Priors

Mohammad Mahdi Johari, Yann Lepoittevin, François Fleuret

arXiv:2111.13539v2cs.CV

TL;DR

GeoNeRF addresses the costly per-scene optimization and limited geometry and occlusion reasoning of generalizable neural view synthesis. It uses cascaded cost volumes and attention-based rendering, outperforming prior generalizable models while enabling efficient fine-tuning and RGBD adaptation.

  • Problem

    NeRF requires lengthy per-scene optimization and dense image capture, while generalizable models have limited geometry and occlusion understanding that can produce artifacts.

  • Method

    GeoNeRF constructs cascaded cost volumes for source views and uses attention-based aggregation with an auto-encoder renderer to synthesize novel views.

  • Results

    GeoNeRF outperforms previous generalizable models, while fine-tuned GeoNeRF reaches competitive results with NeRF after 10k and comparable results after 1k iterations.

  • Takeaways & Limitations

    GeoNeRF renders state-of-the-art-quality images for complex scenes without per-scene optimization and can be adapted to RGBD inputs.

  • Takeaways & Limitations

    Failure cases can occur when stereo reconstruction fails, particularly in textureless areas, because incorrect geometry priors mislead the renderer.

Abstract

from arXiv · show

We present GeoNeRF, a generalizable photorealistic novel view synthesis method based on neural radiance fields. Our approach consists of two main stages: a geometry reasoner and a renderer. To render a novel view, the geometry reasoner first constructs cascaded cost volumes for each nearby source view. Then, using a Transformer-based attention mechanism and the cascaded cost volumes, the renderer infers geometry and appearance, and renders detailed images via classical volume rendering techniques. This architecture, in particular, allows sophisticated occlusion reasoning, gathering information from consistent source views. Moreover, our method can easily be fine-tuned on a single scene, and renders competitive results with per-scene optimized neural rendering methods with a fraction of computational cost. Experiments show that GeoNeRF outperforms state-of-the-art generalizable neural rendering models on various synthetic and real datasets. Lastly, with a slight modification to the geometry reasoner, we also propose an alternative model that adapts to RGBD images. This model directly exploits the depth information often available thanks to depth sensors. The implementation code is available at https://www.idiap.ch/paper/geonerf.

ams OSRAM

GeoNeRF targets the lengthy per-scene optimization and limited geometry and occlusion understanding of existing neural novel-view synthesis methods. It combines cascaded cost volumes with attention-based rendering and supports RGBD inputs.

  • NeRF requires lengthy training from scratch for every scene and densely captured images.
  • Generalizable NeRF models condition rendering on nearby source views but have limited geometry and occlusion understanding, causing artifacts.
  • GeoNeRF constructs cascaded cost volumes to obtain fine, high-resolution geometry priors for conditioning its renderer.
  • Its attention-based model aggregates information from different source views, while an auto-encoder processes information along each ray.
  • GeoNeRF detects and excludes occluded views for each spatial point, processing that point with the remaining views.
  • An alternate model accepts RGBD images and exploits depth information to improve geometry perception.

2. Related Work

Related work spans multi-view stereo, classical and neural novel-view synthesis, and neural scene representations. These approaches trade off reconstruction quality, input sparsity, rendering quality, and per-scene optimization cost.

  • Multi-View Stereo estimates dense scene representations from multiple overlapping images using feature extraction, warping, cost-volume aggregation, and post-processing.
  • Classical novel-view synthesis blends reference pixels or synthesizes radiance fields on meshes and point clouds, but depends on reconstruction quality.
  • Low-textured and reflective regions often cause stereo reconstruction failures in reconstruction-dependent view-synthesis methods.
  • NeRF represents scene geometry and appearance continuously and produces high-quality novel views, but requires hours or days of optimization for each new scene.

3. Method

GeoNeRF combines cascaded cost volumes with Transformer-based multi-view reasoning to estimate geometry and appearance for novel-view rendering. Its renderer samples geometry-informed points, excludes occluded views, predicts densities and colors, and applies classical volume rendering.

  • 3.3. Renderer: For each sampled point, four shared Multi-Head Attention layers aggregate one global token with view-dependent tokens from nearby views.Interpolated 2D and 3D features form the tokens, while occlusion masks prevent invalid views from contributing.
  • 3.3. Renderer: An auto-encoder regularizes tokens along each ray to predict coherent volume densities, using convolutional processing that exploits local geometric continuity.The authors report that this inductive bias reduces computation relative to attention-based alternatives.
  • 3.3. Renderer: Point colors are estimated as weighted sums of non-occluded source-view colors, then ray colors and depths are produced through volume rendering.View-pose similarity contributes to color weighting, and rendered depth also supports training.
  • 3.1. Geometry Reasoner: GeoNeRF constructs three-level cascaded cost volumes for each source view by homography-warping nearby-view features.The geometry reasoner uses FPN feature pyramids, plane sweeps, group-wise correlations, and 3D regularization to produce multi-scale geometric priors.
  • 3.2. Ray Sampling: The renderer samples Nc uniform points and Nf additional points from full-resolution cost-volume probabilities before processing each ray.This concentrates samples near likely surfaces without duplicating coarse and fine networks.
  • 3.5. Compatibility with RGBD data: GeoNeRF supports RGBD inputs through a small modification to its geometry reasoner, assuming incomplete, low-resolution, noisy depth maps.The RGBD-compatible variant is designed to exploit depth information from embedded sensors.

4. Experiments

GeoNeRF is evaluated across synthetic and real datasets, standard metrics, qualitative comparisons, source-view sensitivity, and RGBD inputs. It outperforms prior generalizable models, remains competitive with per-scene NeRF after brief fine-tuning, and supports incomplete low-resolution depth inputs.

  • Quantitative results: GeoNeRF is superior to existing generalizable NeRF models without per-scene optimization across PSNR, SSIM, and LPIPS experiments.The reported comparison includes pixelNeRF, IBRNet, and MVSNeRF.
  • Evaluation setup: GeoNeRF is evaluated on DTU MVS, LLFF, and NeRF realistic synthetic test scenes using established dataset-specific protocols.The evaluation includes 16 DTU scenes, 8 LLFF scenes, and 8 NeRF synthetic scenes.
  • Qualitative results: Qualitative comparisons show that GeoNeRF preserves scene details and produces fewer artifacts than IBRNet and MVSNeRF on synthetic and real datasets.Compared with NeRF, it produces smoother Drums surfaces and higher-quality textures in Ship and Horns; on DTU it renders sharper images with fewer artifacts.
  • Quantitative results: After 10k fine-tuning iterations, GeoNeRF10k produces results competitive with vanilla NeRF, which requires 200k–500k per-scene optimization iterations.GeoNeRF1k is also reported as comparable with NeRF after approximately one hour on a single V100 GPU.
  • Sensitivity to source views: GeoNeRF remains robust to sparse source views and to using distant source views, although distant views degrade quality without causing a significant performance decrease.The source-view analyses are reported on LLFF test scenes.
  • Results with RGBD images: GeoNeRF+D adapts to incomplete, low-resolution depth inputs and renders higher-quality outputs than the original GeoNeRF on the DTU RGBD evaluation.RGB images are 600 × 800, while incomplete depth maps are resized to 150 × 200.

5. Conclusion

GeoNeRF is presented as a generalizable novel view synthesis method that renders state-of-the-art images for complex scenes without per-scene optimization, using geometry and occlusion reasoning from source views.

  • GeoNeRF renders state-of-the-art quality images for complex scenes without per-scene optimization.
  • The geometry reasoner constructs cascaded cost volumes for source views to understand scene geometry and occlusions.
  • An attention-based network aggregates source-view information to synthesize images for novel poses.
  • Selecting nearby views more effectively or adaptively approximating the required number of cost volumes are proposed extensions.

Supplementary Materials for GeoNeRF: Generalizing NeRF with Geometry Priors

The supplementary materials identify the paper’s authors, affiliations, and arXiv version information.

  • Mohammad Mahdi Johari is affiliated with Idiap Research Institute and EPFL.
  • François Fleuret is affiliated with the University of Geneva and EPFL.
  • The paper is listed as arXiv version 2 in the computer-vision category, dated 21 March 2022.

1. Additional Technical Details

The additional technical details describe the geometry reasoner’s cascaded cost volumes and provide network architecture references.

  • The geometry reasoner uses cascaded cost volumes with 48, 32, and 8 depth planes from coarsest to finest resolution.The group-wise correlation similarity calculations use channel size C = 8.
  • Training constructs each cost volume from 3–5 nearby source views.
  • The architectures of the Feature Pyramid Network, 3D regularizer, and auto-encoder are provided in Tables 1, 2, and 3.

2. Additional Qualitative Analysis

The supplementary qualitative materials show rendered novel views and depth maps, while referring readers to the listed network architecture details and fine-tuned rendering setup.

  • Additional Qualitative Analysis: Full-size GeoNeRF examples include novel-view images and rendered depth maps from LLFF and the NeRF realistic synthetic dataset.
  • Additional Qualitative Analysis: Table 1 documents the Feature Pyramid Network architecture, including its output feature pyramids and convolutional operations.
  • Additional Qualitative Analysis: The 3D feature map feat and probability output prob are used to regress the depth map.
  • Additional Qualitative Analysis: GeoNeRF10k images are produced after fine-tuning the model on each scene for 10k iterations, unlike generalizable GeoNeRF images.

3. Per-Scene Breakdown

Per-scene results remain consistent with the aggregate evaluation: generalizable GeoNeRF outperforms existing generalizable methods on average, while fine-tuned GeoNeRF is on par with per-scene optimized vanilla NeRF.

  • GeoNeRF outperforms all existing generalizable methods on average across the reported per-scene evaluations.The breakdown covers real forward-facing LLFF and NeRF realistic synthetic scenes.
  • Fine-tuned GeoNeRF is on par with per-scene optimized vanilla NeRF in the reported per-scene results.GeoNeRF10k is compared with vanilla NeRF optimized for each scene.
  • Figures 1 and 2 show novel images and depth maps from generalizable GeoNeRF and fine-tuned GeoNeRF10k models.Examples come from LLFF and NeRF realistic synthetic test scenes.
  • Tables 4 and 5 report LLFF per-scene comparisons using PSNR, SSIM, and LPIPS.PSNR and SSIM are higher-is-better metrics, while LPIPS is lower-is-better.
  • Tables 6 and 7 report the corresponding per-scene comparisons on the NeRF realistic synthetic dataset.The fine-tuning comparisons include GeoNeRF10k and GeoNeRF1k against vanilla NeRF.

4. Ablation Study

The ablation study evaluates how GeoNeRF’s self-supervision, positional encoding, sampling, occlusion handling, attention, and cascaded cost-volume design affect performance and qualitative outputs.

  • Ablation Study: Table 8 contrasts GeoNeRF variants that remove or alter the model’s key components.Evaluated changes include self-supervision, positional encoding, ray sampling, occlusion exclusion, renderer attention, and cost-volume construction.
  • Ablation Study: The ablation compares models with and without self-supervision loss and positional encoding.These variants test the contributions of training supervision and positional representation.
  • Ablation Study: The study tests uniform ray sampling, retaining occluded views, and removing the renderer’s attention mechanism.These variants probe sampling, visibility reasoning, and feature aggregation choices.
  • Qualitative Analysis: Qualitative examples from challenging NeRF synthetic views correspond to the experiments listed in Table 8.Figure 3 is intended to contrast the behavior of the ablated models visually.
  • Ablation Study: One ablation constructs a volume per view instead of using cascaded multi-level cost volumes.This directly contrasts the proposed cascaded geometry representation with a per-view alternative.

5. Limitations

GeoNeRF can fail when stereo reconstruction in its geometry reasoner produces incorrect priors, particularly in textureless areas.

  • Limitations: GeoNeRF failure cases arise when stereo reconstruction fails in the geometry reasoner and misleads the renderer with incorrect geometry priors.The limitation is linked to the geometry reasoner’s multi-view stereo design.
  • Limitations: Textureless areas are a practical failure boundary because stereo reconstruction is prone to errors there.Figure 4 presents examples of this failure mode.
Loading 2111.13539v2…