PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Lumigraph Rendering

Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, Gordon Wetzstein

arXiv:2103.11571v1cs.CVcs.GR

TL;DR

Novel-view synthesis requires photorealistic rendering, but state-of-the-art neural volume methods are not real-time. This paper jointly learns an implicit surface and radiance field from posed 2D images, then exports the result for traditional real-time rendering while retaining high image quality.

  • Problem

    The central problem is achieving photorealistic novel-view synthesis without the slow rendering rates of volumetric neural methods.

  • Method

    The method jointly optimizes an SDF-based SIREN scene representation and radiance field from posed multi-view images using sphere-traced neural rendering.

  • Results

    The framework provides comparable state-of-the-art image quality while exporting shape and view-dependent appearance for real-time traditional graphics rendering.

  • Takeaways & Limitations

    The representation connects neural rendering with conventional graphics pipelines by producing meshes with view-dependent textures.

  • Takeaways & Limitations

    The method is limited to fixed-lighting emissive radiance, requires annotated object masks, and produces less accurate proxy shapes than some alternatives.

Abstract

from arXiv · show

Novel view synthesis is a challenging and ill-posed inverse rendering problem. Neural rendering techniques have recently achieved photorealistic image quality for this task. State-of-the-art (SOTA) neural volume rendering approaches, however, are slow to train and require minutes of inference (i.e., rendering) time for high image resolutions. We adopt high-capacity neural scene representations with periodic activations for jointly optimizing an implicit surface and a radiance field of a scene supervised exclusively with posed 2D images. Our neural rendering pipeline accelerates SOTA neural volume rendering by about two orders of magnitude and our implicit surface representation is unique in allowing us to export a mesh with view-dependent texture information. Thus, like other implicit surface representations, ours is compatible with traditional graphics pipelines, enabling real-time rendering rates, while achieving unprecedented image quality compared to other surface methods. We assess the quality of our approach using existing datasets as well as high-quality 3D face data captured with a custom multi-camera rig.

1. Introduction

The paper targets photorealistic novel-view synthesis while addressing the lack of real-time rendering in neural approaches. It combines SIREN-based implicit shape and appearance modeling with exportable view-dependent textures and evaluates the framework on standard and custom face data.

  • Motivation: Neural rendering achieves photorealistic novel-view synthesis, but volumetric methods do not provide real-time framerates.The limitation is attributed to volumetric scene representations and custom neural volume rendering.
  • Approach: The framework uses an SDF-based SIREN representation and a custom loss to fit 3D shapes from sparse multi-view images.The loss preserves high-capacity image encoding while constraining angular behavior to reduce view overfitting.
  • Results: 10× higher rendering rates than NeRF are achieved with comparable state-of-the-art image quality and an implicitly defined surface.This combines faster rendering with a surface representation that can be extracted for graphics pipelines.
  • Exportability: Shape and view-dependent appearance can be exported and rendered in real time using traditional graphics pipelines.The framework therefore connects neural scene optimization with mesh-based rendering workflows.
  • Evaluation: A custom camera array provides face and head datasets for evaluating the method and competing baselines.The captured data are made available through the project website.

2. Related Work

Prior work separates high-quality view synthesis from explicit or implicit surface reconstruction. The paper positions its approach as combining these capabilities through a SIREN-based implicit representation.

  • Traditional and image-based rendering: Traditional pipelines estimate geometry and textures from 2D views, but their re-rendered images do not achieve photorealistic quality.Image-based rendering methods such as Lumigraph instead leverage proxy scene representations.
  • Explicit representations: Neural scene representations use explicit proxies including meshes, depth maps, templates, multi-plane images, multi-sphere images, and volumes.These representations span several geometric and image-based forms.
  • Implicit representations: Neural implicit representations define shapes or scenes continuously with networks and can use either 3D data or 2D multi-view images for supervision.They are described as 3D-structure-aware, continuous, and memory-efficient.
  • Closest methods: NeRF provides leading view-synthesis quality without directly modeling object shape, whereas IDR provides strong shape estimation without matching NeRF’s view-synthesis quality.The paper uses SIREN to target both capabilities.

3. Neural Rendering Pipeline

The pipeline represents shape with an SDF and appearance with a direction-conditioned radiance field, then renders by sphere tracing the surface and sampling radiance. Training uses image, geometric, mask, and angular regularization to fit sparse multi-view supervision while limiting overfitting.

  • Representation: Shape is represented as the zero-level set of an SDF, while appearance is modeled by a spatially varying emission function.Both functions use SIREN networks, whose periodic activations represent higher-complexity signals than common non-periodic MLPs.
  • Representation: The radiance field is conditioned on viewing direction and local surface normal, enabling fine spatial details and microreflections.The formulation models emissive radiance under fixed lighting and does not support relighting.
  • Neural rendering: Rendering first finds the nearest SDF zero-level surface along each camera ray, then samples local radiance for its appearance.Sphere tracing iteratively minimizes the SDF magnitude and retains a converged foreground surface after 16 steps.
  • Loss function: Training uses four constraints: L1 image reconstruction, an eikonal SDF constraint, a soft mask loss, and radiance regularization.The eikonal term supports the metric properties needed for efficient sphere tracing, while radiance regularization limits view overfitting.
  • Loss function: Angular smoothness regularization controls SIREN’s view-dependent radiance behavior between sparse training views.The formulation uses a smoothness term in the angular domain, with weights selected for the combined loss.
  • Optimization: Large uniformly sampled ray batches, Fourier features, and initialization to a small sphere support fitting sparse views and high-frequency appearance.The implementation uses 50,000 rays per batch and Fourier features for angular resolution.

4. Real-time Rendering Pipeline

The pipeline converts the learned surface and appearance into a textured mesh, then uses unstructured lumigraph rendering to preserve view-dependent effects at real-time rates. This rasterized renderer exceeds 60 fps at HD resolution while retaining high image quality.

  • 4.1. Mesh extraction: The method extracts a high-resolution mesh from a voxelized SDF and resamples emissivity into projective textures for efficient viewing-space coverage.Marching cubes operates on an SDF voxelized at 512^3, with an offset iso-surface selected to improve image quality.
  • 4.2. Rendering: Unstructured lumigraph rendering blends the five textures whose viewpoints have the smallest angular distances to the current camera.The renderer rasterizes the mesh, computes viewpoint-ray angles, and blends contributions after sorting them in ascending order.
  • 4.2. Rendering: Occluded texture samples are discarded using depth comparisons, while the weighting formulation converges to an exclusive texture mapping as the angular distance approaches zero.This preserves epipolar consistency and uses a shadow-mapping-style occlusion test.
  • 4.3. Evaluation: Only Colmap and NLR-RAS achieve truly real-time performance above 60 fps at HD resolution on commodity hardware.The comparison includes the rasterized neural lumigraph renderer, the sphere-traced variant, and other baselines.
  • 4.3. Evaluation: NLR-RAS achieves PSNRs far superior to other explicit and implicit surface representations, although NLR-ST renders higher-quality images.The reported comparison covers quantitative tables and qualitative examples.

5. Camera Array and Data

The authors evaluate their approach on multiview facial-expression captures recorded with a custom 22-camera array. The setup combines high-resolution narrow-FoV cameras with wider-FoV action cameras and synchronizes recordings to within one frame.

  • 5. Camera Array and Data: The human head video dataset contains seven multiview captures of a person performing facial expressions.
  • 5. Camera Array and Data: The custom camera array uses 16 GoPro HERO7 cameras and 6 Back-Bone H7PRO cameras.The Back-Bone cameras use standard C-Mount lenses and provide a narrower field of view for more detailed subject capture.
  • 5. Camera Array and Data: The cameras record at 4k / 30 fps for Back-Bone cameras and 1080p / 60 fps for GoPro cameras.Back-Bone footage is captured in portrait orientation, while GoPro footage is captured in landscape orientation.
  • 5. Camera Array and Data: ArUco-marker flashes synchronize the loosely synchronized videos with an accuracy of 1 frame or better.The first frame observing the marker is detected separately in each video.

6. Experiments

Experiments show that the method preserves state-of-the-art image reconstruction quality while supporting efficient surface reconstruction and view interpolation across datasets. Ablations attribute performance to the SIREN representation, angular regularization, batch size, and directional Fourier features.

  • Overview: The method achieves state-of-the-art image reconstruction quality comparable with NeRF while enabling efficient surface reconstruction for real-time rendering.The comparison includes Colmap, Neural Volumes, NeRF, and IDR.
  • DTU reconstruction: On DTU, training-image error is comparable with NeRF and significantly better than IDR, while shape error is on par with other techniques but worse than IDR.The shape trade-off favors view-dependent, high-frequency image detail over view-consistent geometry.
  • View interpolation: Held-out interpolated views show a measurable quality drop from training views, but retain many favorable characteristics of the regression results.The angular smoothness loss is designed to prevent collapse of the emissivity function for interpolated views.
  • Human representation: On high-resolution human captures, the method gains a bigger advantage for 3000 × 4000 px detailed images without sacrificing interpolation properties.The authors speculate that competing ReLU networks reach their capacity at this resolution, while explicit representations lack easy scaling.
  • Ablation study: The ablation shows that ReLU lacks capacity for detailed representations, SIREN overfits supervised pixels, and angular smoothing, larger batches, and Fourier features address these issues.The final changes improve angular behavior, spatial uniformity, and low-frequency noise in E.

7. Discussion

The framework combines periodic-activation neural representations with an SDF-based surface and neural rendering to achieve high image quality while supporting real-time traditional graphics pipelines. Its current scope is constrained by fixed-lighting assumptions, annotated masks, imperfect proxy shapes, calibration artifacts, and single-scene overfitting.

  • 7. Discussion: The framework jointly optimizes an SDF-based implicit scene representation and view-dependent appearance from multi-view images.Its representation uses periodic activations and a sphere-tracing-based neural renderer.
  • 7. Discussion: The method achieves image quality comparable with state-of-the-art novel-view synthesis methods while retaining an implicitly defined surface.
  • 7. Discussion: The learned representation can be converted into a mesh with view-dependent textures for real-time rendering in traditional graphics pipelines.
  • 7. Discussion: The method models only emissive radiance functions, making it adequate for fixed lighting but not dynamic lighting and shading.
  • 7. Discussion: The approach requires annotated object masks, produces less accurate proxy shapes than some alternatives, can show calibration-induced seam artifacts, and overfits a single scene.

A. Real-time rendering analysis

The real-time rendering analysis evaluates exported geometries and textures for held-out view interpolation and extrapolation. NLR geometry performs favorably across texture choices, and denser neural-texture sampling further benefits its view synthesis.

  • Real-time rendering analysis: The evaluation measures PSNR on a held-out sixth view using five of six high-resolution cameras as texture inputs, averaged over all six test-view choices.
  • Real-time rendering analysis: NLR geometry performs favorably against IDR and Colmap across texture choices, while Colmap’s incomplete geometry creates rendering holes.IDR and NLR show similar view consistency with captured textures, with NLR retaining a small margin.
  • Real-time rendering analysis: IDR’s neural textures lack detail relative to NLR’s, producing comparatively lower PSNR scores.
  • Real-time rendering analysis: Texture-space upsampling subdivides the original 3 × 2 camera layout from 6 textures at 1× to 15 at 2× and 45 at 3×.
  • Real-time rendering analysis: Denser texturing improves IDR and SineSDF, but NLR benefits most by exploiting the high spatial detail of its neural textures.

B. Additional Results

Additional results examine calibration-controlled reconstruction and supplemental renderings across neural and traditional methods. On a synthetic head scene, both methods reconstruct clean shapes, while NLR provides higher image fidelity despite IDR’s better Chamfer distance.

  • Additional Results: Supplemental videos compare neural renderings, extracted shapes, and real-time renderings from NLR and baseline methods.
  • Additional Results: The calibration-controlled experiment renders a 5 × 5 grid of views from a Unity head model with known poses and masks.The scene uses a 2160 × 2160-pixel resolution and surface specular properties.
  • Additional Results: Both NLR and IDR produce clean shape reconstructions without artifacts on the synthetic head scene.The model’s semi-transparent billboard hair remains challenging to reconstruct accurately.
  • Additional Results: NLR achieves higher image quality than IDR, although Chamfer distance favors IDR.

E. Datasets

The evaluation spans captured and benchmark datasets, with different camera layouts, resolutions, masks, and reconstruction resources. Supplementary tables report held-out DTU metrics and cross-dataset average image metrics, while baseline implementations use method-specific computational settings.

  • Datasets: The captured dataset contains 22 images per capture: six high-resolution narrow-FOV views and sixteen lower-resolution wide-FOV views.All views train the evaluated methods, but reconstruction quality is evaluated on the six high-quality views.
  • Datasets: Supplementary Tables 5 and 6 report held-out DTU image metrics and cross-dataset average PSNR, SSIM, and LPIPS scores.Table 6 notes the number of scenes and that only five scenes were tested for one result.
  • Datasets: DTU scans contain 49 or 64 calibrated 1600 × 1200-pixel images covering approximately a 90 × 90-degree viewing zone.Structured-light stereo point captures provide ground-truth shapes for Chamfer distance.
  • Datasets: Volucap evaluates a full-body reconstruction from a video frame, while Digital Ira provides seven very high-resolution facial closeups with calibrated cameras.
  • Datasets: The authors reduce their dataset training resolution by a factor of two because of memory requirements, from 3000 × 4000 to 1500 × 2000 pixels.A four-fold reduction produced 28.62 dB versus 28.58 dB with the two-fold reduction.
  • Datasets: Baseline evaluation uses original implementations with resource-dependent settings, including reduced resolution for Neural Volumes and fixed calibrated poses for IDR.

G. Metrics

The evaluation uses foreground-masked PSNR for image reconstruction and one-directional Chamfer distance for shape reconstruction, with procedures designed to avoid background and occlusion penalties.

  • PSNR: PSNR is computed between reconstructed and ground-truth images only within object-mask pixels.The same procedure is used for all methods, including methods that do not use masks to recover shape.
  • Chamfer distance: Chamfer distance measures the shortest distance from each ground-truth 3D point to the reconstructed shape in one direction.Open3D computes the distances using a procedure similar to the DTU dataset scripts.
  • Chamfer distance: The one-directional Chamfer procedure avoids penalizing reconstructed background and occluded object parts.Neural Volumes is excluded because surface extraction was not demonstrated in the cited evaluation.

H. Sphere tracing implementation details

The implementation uses bidirectional sphere tracing to bracket surface locations, refine the first zero crossing, and obtain a differentiable final surface point, although convergence is not guaranteed for general shapes.

  • Convergence limitation: Convergence of Eq. 5 toward the zero-level set S0 is not guaranteed for general shape configurations.The SDF can underestimate the optimal step when the nearest surface is not orthogonal to the ray.
  • Bidirectional tracing: Bidirectional sphere tracing narrows a possible surface-position range without relying on sphere-tracer convergence.Additional solvers then refine the surface location.
  • Bidirectional tracing: The method first solves Eq. 5 in the forward direction, masks sufficiently converged rays, and searches in the opposite direction for a far zero-level set.Forward tracing uses n = 16, while rays satisfying |S(x_n)| < 5e−5 are not further optimized.
  • Surface refinement: When the far candidate is closer than the near candidate, the method concludes that no surface exists; otherwise it samples the range for the first zero crossing.The crossing is refined by eight bisection steps assuming a locally monotonic S.
  • Differentiable tracing: A final forward sphere-tracing step with gradient tracking produces differentiable sphere tracing, with denominator clamping to avoid division by zero.The procedure is otherwise not differentiated for performance, stability, and memory efficiency.
Loading 2103.11571v1…