PaperlayerView exact revision on arXiv

Source-linked AI summary

Light Field Neural Rendering

Mohammed Suhail, Carlos Esteves, Leonid Sigal, Ameesh Makadia

arXiv:2112.09687v2cs.CV

TL;DR

Sparse-view synthesis must reconcile light-field rendering’s ability to represent view-dependent effects with geometric methods’ dependence on accurate depth. This paper uses epipolar-constrained transformer fusion over a four-dimensional light field and reports higher-fidelity results across forward-facing and 360° captures, with up to 5 dB improvement in challenging scenes.

  • Problem

    Sparse-view novel-view synthesis remains challenging, and existing approaches trade dense light-field sampling against geometric methods’ difficulty modeling non-Lambertian effects.

    Source: §1 · pp. 1–2

  • Method

    A four-dimensional light-field renderer uses epipolar geometric constraints and a two-stage transformer to aggregate features along epipolar lines and then across reference views for each target ray.

    Source: §1 · pp. 1–2, §3.2 · p. 3, §3.3.2 · p. 5

  • Results

    The model produces higher-fidelity renderings than state of the art on forward-facing and 360° captures, with up to 5 dB improvement in the most challenging scenes.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The design also allows dense correspondences, disparity maps, and epipolar-plane images to be extracted without additional training.

    Source: §6 · p. 8

    The model produces fuzzy details on texture-less thin repeating structures, including grill-like structures and wiremesh.

    Source: §5 · p. 8

Abstract

from arXiv · show

Classical light field rendering for novel view synthesis can accurately reproduce view-dependent effects such as reflection, refraction, and translucency, but requires a dense view sampling of the scene. Methods based on geometric reconstruction need only sparse views, but cannot accurately model non-Lambertian effects. We introduce a model that combines the strengths and mitigates the limitations of these two directions. By operating on a four-dimensional representation of the light field, our model learns to represent view-dependent effects accurately. By enforcing geometric constraints during training and inference, the scene geometry is implicitly learned from a sparse set of views. Concretely, we introduce a two-stage transformer-based model that first aggregates features along epipolar lines, then aggregates features along reference views to produce the color of a target ray. Our model outperforms the state-of-the-art on multiple forward-facing and 360{\deg} datasets, with larger margins on scenes with severe view-dependent variations.

1. Introduction

The paper targets sparse-view novel-view synthesis that preserves non-Lambertian effects, combining a light-field representation with epipolar geometric constraints and transformer-based ray fusion. It reports higher-fidelity results across forward-facing and 360° captures, with gains up to 5 dB in the most challenging scenes.

1. Introduction

  • NeRF’s 5D representation maps scene points and viewing directions to opacity and color, but its smooth radiance prior limits fidelity for non-Lambertian effects.The cited passage explains that restricting viewing-direction input helps avoid degenerate shape-radiance solutions, at the expense of fidelity.Source: §1 · p. 1
  • Sparse light-field rendering lacks a straightforward solution, while geometry-based methods can depend on accurate depth estimates that are difficult to obtain for non-Lambertian surfaces.Dense light-field sampling supports interpolation without a 3D geometry model, but sparse input views do not.Source: §1 · p. 2
  • The proposed model renders a sparse light field by fusing nearby-view rays with a transformer and an epipolar-geometry constraint, without explicit 3D information.The contribution passage describes the constraint as an inductive bias for sparse-view ray fusion.Source: §1 · pp. 1–2
  • The model achieves higher-fidelity renderings than state of the art on forward-facing and 360° captures, with up to 5 dB improvement in the most challenging scenes.The introduction also highlights reconstruction of refraction in the Shiny Lab scene, where NeX and NeRF fail to synthesize it.Source: Figure 1 · p. 1, §1 · p. 2

2. Related work

Prior light-field methods face tradeoffs in input density, motion range, or scene complexity, while geometry-based and neural rendering approaches use different scene representations and rendering strategies. This work focuses on complex sparse-view scenes with non-Lambertian effects and differs from related transformer-based methods by using light-field rendering and staged feature aggregation.

Light field rendering

  • Earlier learning-based light-field methods require dense input sampling, have limited motion range, or are restricted to simple scenes; this work targets complex scenes with non-Lambertian effects from sparse viewpoints.Proxy geometry in Lumigraph rendering addresses aliasing from irregularity or view undersampling.Source: §2 · p. 2

Neural scene representation

  • NeRF uses volumetric rendering, while NeX uses a multiplane image representation but remains challenged by reflection or refraction interference patterns through liquid.The paper positions its model for cases with complex view-based effects where other methods fail.Source: §2 · p. 2

Image-based rendering

  • Image-based rendering methods borrow pixel values from input images, with approaches using global meshes, inferred depth maps, or GAN-based synthesis for view-dependent effects.Some methods estimate depth uncertainty to refine images when depth estimation is error-prone.Source: §2 · p. 2
  • Unlike IBRNet and NerFormer, this method uses a light-field framework without volumetric rendering and aggregates features first along epipolar lines, then across views.The two stages reason about correspondences, then occlusion and lighting effects, to produce the target-ray color.Source: §2 · pp. 2–3

3. Approach

The approach represents rays in a four-dimensional light field and uses epipolar geometry to render novel views from sparse observations. Two transformer stages aggregate epipolar-point features within reference views and then combine those views to predict target-ray color.

3.1. Light field parametrization

  • The model uses four-dimensional ray coordinates, with light-slab parametrization for forward-facing captures and two-sphere parametrization for 360° scenes.It maps ray coordinates to radiance and evaluates the model for each target pixel.Source: §3.1 · p. 3

3.2. Model overview

  • For a target camera, the model selects neighboring reference views and samples target-ray points that project onto epipolar lines in those views.Training randomly selects K views from a subset of N closest views; inference deterministically selects the closest K.Source: §3.2 · p. 3

3.3. Network architectures

  • Transformers model both stages because epipolar points form a sequence and reference-view features form a set with relational information.The first stage creates a feature per reference view; the second aggregates those features to predict target-ray color.Source: §3.2 · p. 3, §3.3 · p. 4, Figure 2 · p. 4

3.3.1 Epipolar feature transformer (f1)

  • The epipolar transformer self-attends over features for points along each reference-view line, then attention-weights the outputs to form that view’s feature.Point features include ray and 3D-point coordinates, and the 3D point supplies positional information associated with depth along the target ray.Source: §3.3.1 · p. 4

3.3.2 View feature transformer (f2)

  • The view transformer combines the target ray with reference-view features, then aggregates them with learned weights to predict the target pixel color.Its output feature is linearly projected and passed through a sigmoid.Source: §3.3.2 · p. 5

3.4. Loss

  • An auxiliary L2 loss on an attention-weighted color prediction encourages lower-entropy epipolar weights and higher weights for unoccluded views.The passage states that this auxiliary loss also leads to more accurate renderings.Source: §3.4 · p. 5

4. Experiments

The experiments compare the model with prior methods on forward-facing and 360° datasets, report qualitative and average-score results, and probe design choices and geometry-related interpretations.

4.2.1 Real-forward-facing (RFF) dataset

  • On RFF, renderings retain sharper textures and more consistent leaf and petal boundaries than baselines; Table 1 reports average metrics across all eight scenes.Source: Table 1 · p. 6, §4.2.1 · p. 6

4.1. Implementation details

  • The implementation uses eight-block, 256-feature transformers, Fourier coordinate encodings for the light slab and 3D points, and spherical-harmonic encodings for two-sphere coordinates.Training runs for 250,000 iterations; a Blender scene takes around 23 hours to train on a 32-core TPUv3 slice.Source: §4.1 · p. 6

4.2.2 Shiny dataset

  • On Shiny, the model consistently improves over NeX, IBRNet, and NeRF across all metrics, reconstructing disk interference patterns and bottle reflections with greater detail than baselines.Table 2 reports average scores across the dataset’s scenes.Source: Table 2 · p. 6, §4.2.2 · pp. 6–7

4.2.3 Blender dataset

  • The model improves over NeRF, IBRNet, and Mip-NeRF on all Blender metrics, including an around 4 dB PSNR improvement over Mip-NeRF on the reflective materials scene.Table 3 reports average performance across all Blender scenes.Source: Table 3 · p. 6, §4.2.3 · p. 7

4.3. Ablation studies

  • Ablations compare a geometry-free MLP, transformer replacements, and variants removing visual features, 3D coordinates, camera embeddings, attention aggregation, or the auxiliary loss.The reported comparison averages metrics across RFF scenes downsampled to 25% resolution (504 × 378).Source: §4.3 · p. 7

4.4. Interpreting the model

  • Attention weights provide potential target-ray correspondences: examples include unimodal peaks, an uncertain distribution, and a multimodal distribution with its highest peak near the correct correspondence.Figure 4 visualizes per-point weights on epipolar lines for four target rays and one reference view.Source: §4.4 · p. 7, Figure 4 · p. 7
  • A disparity map is estimated by combining per-point and per-view attention weights into a weighted average of putative epipolar-point depths.Figure 5 shows an example for a test view from the Shiny dataset’s Crest scene.Source: §4.4 · pp. 7–8, Figure 5 · p. 8
  • Epipolar-plane images show diffuse points as lines with slopes inversely proportional to depth, while specular points can appear as curves as viewpoint changes.The CD scene’s interference-pattern regions exhibit curves associated with changing virtual apparent depth.Source: §4.4 · p. 8

5. Limitations

The method struggles with texture-less thin repeating structures and incurs substantial computational cost. Epipolar-plane images provide a view of how depth and view-dependent effects appear in the 4D light field.

5. Limitations

  • The model produces fuzzy or distorted details on texture-less thin repeating structures, including grill-like and microphone wiremesh patterns.Examples include the Tools scene from Shiny [56] and the Mic scene from Blender [32].Source: §5 · p. 8, Figure 7 · p. 8
  • The method is around 8 times slower than Mip-NeRF [4] on the same hardware.The authors attribute slow training and inference to transformer computational expense.Source: §5 · p. 8
  • In an epipolar-plane image, depth appears as lines with different slopes, while view-dependent effects appear as curves.Each EPI vertically stacks images along a camera-motion line.Source: Figure 6 · p. 8
  • Random device memory access and host-device data transfer add overhead, which the authors believe engineering effort could reduce.On the same hardware, rendering an 800 × 800 image takes 60–70 seconds for this method versus around 180 seconds for NerFormer [38].Source: §5 · p. 8

6. Conclusion

The paper presents neural rendering over a four-dimensional light field for novel-view synthesis. Its two-stage framework uses epipolar constraints to address sparse sampling, and the conclusion reports improvements over prior state of the art, especially for challenging view-dependent effects.

6. Conclusion

  • The method learns a four-dimensional light field to naturally handle real-world illumination effects in novel-view synthesis.Unlike prior volumetric rendering methods, the model represents the light field over a four-dimensional space.Source: §6 · p. 8
  • A two-stage framework incorporates epipolar constraints to address the dense-sampling dependency of light-field rendering.The constraints provide geometric inductive bias.Source: §6 · p. 8
  • The model significantly improves over previous state-of-the-art view-synthesis methods, especially on scenes with challenging view-dependent effects.Source: §6 · p. 8
  • The model can extract dense correspondences, disparity maps, and epipolar-plane images without additional training.Source: §6 · p. 8

A.1. MLP architecture in ablation

The appendix describes the 1-MLP architecture, spherical light-field encoding and its ablation, and metric-computation choices for different datasets.

A.1. MLP architecture in ablation

  • The 1-MLP architecture was selected by sweeping model depths; increasing capacity further led to poor generalization.Its architecture uses DenseGeneral layers, skip connections every fourth layer in DG12, and a final layer for predicted color.Source: §A.1 · p. 11, Table A.1 · p. 11

A.2. Spherical light field encoding

  • For 360° scenes, rays use two sphere-intersection points encoded with spherical harmonics, concatenating zonal and sectoral harmonics through degree 4.The appendix evaluates this encoding against regular NeRF positional encoding in a Blender-dataset ablation.Source: §A.2 · p. 11

A.3. Metric computation

  • SSIM uses scikit-image, while LPIPS uses VGG implementations from [63] for forward-facing scenes and TensorFlow Hub for Blender.The two LPIPS implementations are used to ensure fair comparison.Source: §A.3 · p. 11

B.1. Real-forward-facing dataset (RFF)

The appendix describes the RFF dataset and points to its scene-wise results and metric sources.

B.1. Real-forward-facing dataset (RFF)

  • The RFF dataset introduced by Mildenhall et al. [31] contains eight forward-facing captures, each with around 20 to 62 images.Source: §B.1 · p. 11
  • Table B.3 presents a scene-wise breakdown of the RFF results.Source: §B.1 · p. 11
  • The NeRF and NeX metrics are those reported in NeX [56].Source: §B.1 · p. 11

B.2. Shiny dataset

The Shiny dataset contains eight forward-facing scenes with challenging view-dependent effects, and the section reports scene-wise quantitative results.

B.2. Shiny dataset

  • Shiny contains eight forward-facing scenes with challenging view-dependent effects.Scene-wise quantitative results are presented in Table B.4.Source: §B.2 · p. 11, Table B.4 · p. 12
  • NeRF and NeX metrics are reported from NeX [56].Source: §B.2 · p. 11

B.3. Blender dataset

The Blender dataset comprises eight scenes of 800 × 800 images rendered from randomly sampled viewpoints on a hemisphere around each object.

B.3. Blender dataset

  • Blender contains eight scenes of 800 × 800 images rendered from viewpoints randomly sampled on a hemisphere around each object.Source: §B.3 · p. 12
  • Scene-wise quantitative results are presented in Table B.5, with NeRF and Mip-NeRF metrics taken from Mip-NeRF [4].Source: §B.3 · p. 12

C. Additional experiments and visualizations

The additional experiments find small PSNR advantages for scene-specific ray parametrizations and show how the model handles varied view-dependent effects and uses reference-view attention.

C.1. Pl¨ucker coordinates

  • Plücker coordinates lower PSNR by 0.18 dB on RFF and 0.25 dB on Blender relative to the scene-specific light slab and two-sphere parametrizations.The authors suggest the specific, lower-dimensional parametrizations have a slight advantage for these configurations over generic Plücker coordinates.Source: §C.1 · p. 12

C.2. Handling view-dependent effects

  • Attention can downweight geometric constraints when they are not useful, and the authors speculate that this and convolutional context explain performance on view-dependent effects.Source: §C.2 · p. 12
  • Table B.5 presents scene-wise quantitative results for Blender.Source: Table B.5 · p. 13
  • On the Shiny Lab scene, our model handles both a refraction crop and a diffuse crop with sharp details, unlike Vanilla-NLF and NeX, which each miss one type of effect.Vanilla-NLF retrieves most refraction details but misses high frequencies; NeX reproduces sharp details but not refractions.Source: §C.2 · p. 12, Figure C.8 · p. 13

C.3. Visualizing view attention

  • The attention weights βj represent the importance of each reference view when rendering a target pixel.Figure C.9 maps three reference-view weights to RGB channels; the chair cushion is mostly green, while its back mixes red and blue.Source: §C.3 · p. 13, Figure C.9 · p. 14

Related papers

Loading 2112.09687v2…