Source-linked AI summary
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, Fredo Durand
TL;DR
Inferring 3D scene shape and appearance from limited 2D observations remains a central problem. The paper introduces Light Field Networks, which directly represent 360-degree light fields and use meta-learning for single-view reconstruction; the resulting system supports fast rendering and sparse depth extraction, with scope limited to simple scenes.
Problem
Inferring 3D scene shape and appearance from impoverished 2D observations is a fundamental problem in computer graphics, computer vision, and artificial intelligence.
Method
Light Field Networks directly parameterize 360-degree light fields with neural implicit representations, using Plücker coordinates and meta-learning for multi-view-consistent reconstruction.
Results
LFNs outperform globally conditioned baselines in single-shot novel-view synthesis while being three orders of magnitude faster and less memory-intensive than volumetric rendering approaches.
Takeaways & Limitations
LFNs provide real-time neural rendering with one evaluation per ray and support sparse depth-map extraction without ray-casting.
Takeaways & Limitations
The method is currently constrained to reconstructing simple scenes such as single objects and simple room-scale scenes.
Abstract
from arXiv · showhide
Inferring representations of 3D scenes from 2D observations is a fundamental problem of computer graphics, computer vision, and artificial intelligence. Emerging 3D-structured neural scene representations are a promising approach to 3D scene understanding. In this work, we propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field parameterized via a neural implicit representation. Rendering a ray from an LFN requires only a single network evaluation, as opposed to hundreds of evaluations per ray for ray-marching or volumetric based renderers in 3D-structured neural scene representations. In the setting of simple scenes, we leverage meta-learning to learn a prior over LFNs that enables multi-view consistent light field reconstruction from as little as a single image observation. This results in dramatic reductions in time and memory complexity, and enables real-time rendering. The cost of storing a 360-degree light field via an LFN is two orders of magnitude lower than conventional methods such as the Lumigraph. Utilizing the analytical differentiability of neural implicit representations and a novel parameterization of light space, we further demonstrate the extraction of sparse depth maps from LFNs.
1 Introduction
The paper addresses inferring 3D scene shape and appearance from limited 2D observations by proposing Light Field Networks, with scope restricted to simple scenes.
- 1 Introduction: Light Field Networks directly parameterize a scene’s 360-degree light field as a neural representation rather than encoding it in 3D space.The representation maps oriented camera rays to observed radiance.
- 1 Introduction: Rendering an LFN requires one network evaluation per ray, avoiding the tens or hundreds required by occupancy, signed-distance, or volumetric approaches.The proposed representation therefore removes the need to query 3D properties along a ray or ray-march toward a surface.
- 1 Introduction: LFNs use 6-dimensional Plücker coordinates to represent continuous 360-degree light fields and analytically extract sparse depth maps from derivatives.The parameterization supports a-priori unbounded scenes and the geometry extraction relies on analytical differentiability.
- 1 Introduction: Meta-learning embeds LFNs in a learned multi-view-consistent space, supporting light-field reconstruction and novel-view synthesis from sparse 2D supervision.The contributions include reconstruction from as little as a single image observation.
- 1 Introduction: The method is currently constrained to simple scenes, including single objects and simple room-scale scenes.This scope matches the regime of recent work on learning generative models cited by the authors.
2 Related Work
The related work spans neural scene representations, fast light-field rendering, and learned reconstruction; LFNs combine these directions through a neural 360-degree light-field parameterization.
- Neural Scene Representations and Neural Rendering: Neural scene representations aim to infer useful 3D feature representations for downstream graphics, vision, and machine-learning tasks.Prior approaches include neural fields and other coordinate-based representations.
- Light fields and their reconstruction: The overview maps oriented rays to radiance through an LFN and contrasts one evaluation per ray with hundreds for SRNs, NeRF, and DVR.It also connects the representation to multi-view-consistent reconstruction and real-time novel-view synthesis of simple scenes.
- Light fields and their reconstruction: LFNs represent 360-degree light fields as neural fields and infer them with meta-learning from as few as a single 2D image observation.The paper demonstrates that these representations encode both scene geometry and appearance.
- Light fields and their reconstruction: Light fields were developed as sampled scene representations for fast image-based rendering and later used for novel-view synthesis and computational photography.The related-work lineage includes the plenoptic function and 4D light fields.
3 Background: 3D-structured Neural Scene Representations
3D-structured neural scene representations model continuous scene properties in 3D and use differentiable rendering, but conventional renderers require many evaluations along each ray.
- Neural fields: Neural fields model a scene as a continuous MLP that maps a 3D coordinate to a representation of the scene at that location.This representation is denoted Φ3D and produces a feature vector or other local property representation v.
- Differentiable rendering: A differentiable renderer combines the 3D representation with a ray to compute the scene color observed along that ray.The renderer is denoted m and the resulting color is c.
- Rendering methods: Existing renderers use either sphere tracing or volumetric rendering to evaluate the 3D representation repeatedly along a ray.These approaches search for surfaces or solve alpha compositing along sampled ray locations.
- Rendering methods: Tens or hundreds of evaluations of Φ3D along one ray increase rendering time and memory complexity and affect both training and test time.Training is additionally affected because error backpropagation passes through the renderer.
4 The Light Field Network Scene Representation
LFNs represent a scene as a continuous 360-degree neural light field that maps oriented rays directly to radiance. Their parameterization supports single-evaluation rendering, analytical geometry extraction, and meta-learned reconstruction with multi-view consistency.
- Implicit representations for 360 degree light fields: An LFN maps each oriented ray in the four-dimensional light-ray space directly to its observed RGB radiance.This avoids querying opacity and color at sampled 3D locations along the ray.
- Implicit representations for 360 degree light fields: Six-dimensional Plücker coordinates provide a continuous representation of all oriented rays without singular directions or special cases.Although represented as six-tuples, valid rays lie on a curved four-dimensional subspace; unlike several conventional parameterizations, this supports 360-degree light fields.
- Rendering LFNs: Single-evaluation rendering computes each pixel by converting its camera ray to Plücker coordinates and evaluating the LFN once.The camera intrinsics and extrinsics determine each ray’s Plücker coordinates before network evaluation.
- The geometry of Light Field Networks: Epipolar Plane Images extracted from LFNs encode scene geometry because rays through a surface point form constant-color lines whose slopes determine that point’s 3D coordinates.Local two-plane bases are formed from two parallel lines intersecting the selected ray, and varying their coordinates samples an EPI.
- The geometry of Light Field Networks: Analytical light-field derivatives enable sparse depth extraction, but estimates are invalid where gradients vary strongly, including changing surface colors and tangent rays.Meaningful depth estimates require nonzero derivatives along the ray; dense extraction is identified as possible future extension.
- Meta-learning with conditional Light Field Networks: Meta-learning imposes multi-view consistency by learning a prior over light fields, with a hypernetwork mapping each scene’s latent code to its LFN parameters.This framework is designed to reconstruct a new scene’s light field from a single 2D observation by optimizing its latent variable.
5 Experiments
Experiments evaluate LFNs for novel-view reconstruction, rendering efficiency, storage, and sparse geometry recovery on simple scenes. LFNs offer strong reconstruction results with substantially lower computational and memory costs, while geometry evaluation remains scoped to sparse valid depths.
- Multi-class single-view reconstruction: LFNs outperform DVR and SRNs on all but two ShapeNet classes by an average of 1dB, using more than an order of magnitude fewer network evaluations per ray.The multi-class single-view experiment also reports often crisper reconstructions than both baselines.
- Class-specific single-view reconstruction: Class-specific reconstruction is better than SRNs in PSNR for cars and on par in SSIM, while chairs show the opposite PSNR–SSIM pattern.The class-specific setting uses cars and chairs and requires an order of magnitude fewer network evaluations with real-time rendering.
- Global vs. local conditioning: Locally conditioned pixelNeRF outperforms LFNs in single-shot reconstruction, whereas LFNs require three orders of magnitude less rendering time and memory.LFNs are qualitatively comparable for many objects but confuse object class on others.
- Real-time rendering and storage cost: A single LFN uses approximately 1.6 MB for about 400k parameters, compared with 146 MB for a conventional 360-degree light field.The conventional light field uses a 256×256×17×17 six-plane representation.
- Real-time rendering and storage cost: LFNs require three orders of magnitude less compute than volumetric rendering approaches and support real-time rendering.Rendering complexity is measured for 256 × 256 images on an NVIDIA RTX 6000 GPU.
- Evaluation of reconstructed geometry: Backprojected sparse depth maps enable 3D point-cloud reconstruction, and LFNs achieve lower mean L1 depth error than SRNs on valid depth estimates.The comparison favors LFN-valid sparse depths, including occlusion boundaries where sphere tracing can incur high error; the authors do not claim geometry-specific competitiveness.
6 Discussion and Conclusion
The paper concludes that LFNs provide real-time neural rendering and sparse depth extraction through direct 360-degree light-field parameterization. It also identifies scene complexity, view placement, local conditioning, non-Lambertian scenes, and misuse as important boundaries and future directions.
- Conclusion: LFNs directly parameterize a full 360-degree, 4D light field and enable real-time rendering with one neural-scene-representation evaluation per ray.They also support sparse depth-map extraction without ray-casting.
- Conclusion: LFNs outperform globally conditioned baselines in single-shot novel-view synthesis while being three orders of magnitude faster and less memory-intensive than volumetric rendering approaches.
- Societal impacts: Extending few-observation novel-view synthesis could reduce the cost of non-consensual impersonations, creating a potential abuse pathway.The paper refers readers to a review of neural rendering for further discussion.
–Supplementary Material–
The supplementary material accompanies a paper authored across multiple institutions and presented at NeurIPS 2021 in Sydney, Australia.
- Author affiliations: The listed affiliations include MIT CSAIL, Columbia University, IAFI, MIT BCS, and CBMM.
- Authorship: The paper marks its authors as contributing equally.
- Publication: The work appeared at the 35th Conference on Neural Information Processing Systems in Sydney, Australia, in 2021.
1 Additional results on the local two-plane parameterization
The appendix derives a local two-plane parameterization for light fields represented with Plücker coordinates, then uses analytical derivatives to estimate ray-generating points and depth.
- Local parameterization: A ray and two points along it define parallel lines that provide local coordinates (s, t) for a two-plane light-field slice.The construction uses a direction not parallel to the selected ray and parameterizes lines through the two points.
- Geometric derivatives: For Lambertian surfaces, the gradient of the light-field color is orthogonal to the family of rays through a scene point, so J∇s,tc follows that family.J denotes a 90-degree rotation in the local (s, t) plane.
- Depth recovery: The similarity relationships between nearby rays yield a relation between color derivatives and the distance D between the two parameterizing lines.The appendix derives this relation from coplanar lines and nearby rays through the same scene point.
- Depth recovery: Given a normalized Plücker ray (d, w), x = d × w provides the closest point on the ray to the origin before constructing the local two-plane basis.The procedure then selects an auxiliary direction and a second point x + Dd along the ray.
2 Reproducibility
The reproducibility materials specify public code and datasets, hardware, network architectures, optimization settings, and separate training procedures for the evaluated scene classes.
- Resources: The authors state that all code and datasets will be made publicly available.This is presented as part of the experiment-reproduction details.
- Hardware: Each model was trained separately on one NVIDIA RTX 6000 GPU with 24 GB of memory, using up to four GPUs in parallel overall.The per-model and aggregate GPU usage are reported separately.
- Architecture: LFNs use six-layer fully connected ReLU networks with 256 hidden units per layer and layer normalization without affine transforms.Hypernetworks use three fully connected ReLU layers, layer normalization with affine transforms, and 256-unit hidden layers.
- Training: Cars and rooms models are trained in two resolution stages, from 64 × 64 to 128 × 128, with ADAM at a step size of 10^-4.The total training duration is approximately three days, with λlat set to 1e2.
3 Additional Results
Additional experiments examine qualitative rendering, computational complexity, conditioning, training-set scale, and the distinction between multi-view consistency and object priors. They also show that direct scene overfitting lacks consistency for unobserved rays.
- Qualitative results: Novel views, Epipolar Plane Images, and extracted depth maps are shown for training-set objects.The paper points readers to the supplemental video for additional qualitative results.
- Rendering complexity: The authors provide additional comparisons of rendering complexity for 800 × 800 pixel images.The comparison continues in Table 1, whose other architecture data come from prior work.
- Conditioning: Conditioning by concatenation performs significantly worse than hypernetwork conditioning in the 13-class ShapeNet ablation.The authors do not claim their conditioning method is superior to all alternatives and identify optimal conditioning as future work.
- Training-set scale: PSNR improves consistently as the training set grows from 1 to up to 2500 shapes in single-class and multi-class reconstruction experiments.The trend suggests that additional training-set objects would likely improve performance further.
- Meta-learning: Meta-learning across 3D scenes serves both to learn multi-view-consistent light fields and to learn a prior enabling few-shot reconstruction.A separate experiment is used to disentangle these two roles.
- Overfitting versus generalization: An LFN can reproduce densely sampled context views with high-frequency detail, but intermediate views become basically random without a learned multi-view consistency prior.This contrasts direct overfitting with prior-based reconstruction from incomplete observations.