PaperlayerView exact revision on arXiv

Source-linked AI summary

HDR-NeRF: High Dynamic Range Neural Radiance Fields

Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, Xuan Wang, Qing Wang

arXiv:2111.14451v4cs.CV

TL;DR

Novel-view synthesis is limited by low-dynamic-range inputs and existing NeRF radiance, while conventional HDR methods do not render novel views and often require HDR supervision. HDR-NeRF jointly learns an HDR radiance field and tone mapper from differently exposed LDR views, then renders HDR and exposure-controlled LDR views. Experiments on synthetic and real-world scenes show high-dynamic-range novel views and accurate exposure control, with strong reported performance on the authors’ dataset.

  • Problem

    Existing novel-view methods are limited to low dynamic range, while conventional HDR methods cannot render novel views and learning-based HDR methods require HDR supervision.

  • Method

    HDR-NeRF jointly learns a radiance field for density and scene radiance and a tone mapper for camera-response colors from differently exposed LDR views, using volume rendering with only LDR supervision.

  • Results

    HDR-NeRF renders novel HDR views and LDR views with controlled exposures, achieving similar major-metric scores to NeRF-GT and best performance against SOTAs on a new HDR dataset.

  • Takeaways & Limitations

    The method provides neural rendering of novel views with high dynamic range while retaining control over the exposure of synthesized LDR views.

  • Takeaways & Limitations

    The recovered HDR radiance is relative rather than absolute, with three unknown RGB scaling factors affecting white balance; the tone mapper also omits ISO gain and aperture effects.

Abstract

from arXiv · show

We present High Dynamic Range Neural Radiance Fields (HDR-NeRF) to recover an HDR radiance field from a set of low dynamic range (LDR) views with different exposures. Using the HDR-NeRF, we are able to generate both novel HDR views and novel LDR views under different exposures. The key to our method is to model the physical imaging process, which dictates that the radiance of a scene point transforms to a pixel value in the LDR image with two implicit functions: a radiance field and a tone mapper. The radiance field encodes the scene radiance (values vary from 0 to +infty), which outputs the density and radiance of a ray by giving corresponding ray origin and ray direction. The tone mapper models the mapping process that a ray hitting on the camera sensor becomes a pixel value. The color of the ray is predicted by feeding the radiance and the corresponding exposure time into the tone mapper. We use the classic volume rendering technique to project the output radiance, colors, and densities into HDR and LDR images, while only the input LDR images are used as the supervision. We collect a new forward-facing HDR dataset to evaluate the proposed method. Experimental results on synthetic and real-world scenes validate that our method can not only accurately control the exposures of synthesized views but also render views with a high dynamic range.

1. Introduction

HDR-NeRF addresses the limited dynamic range of existing neural novel-view synthesis by recovering an HDR radiance field from differently exposed LDR views. It models radiance, camera response, and rendering jointly, enabling HDR and exposure-controlled LDR novel views without HDR supervision.

  • Motivation: Existing NeRF methods produce high-quality novel views, but their recovered radiance is limited to a low dynamic range while physical scenes span 0 to +∞.The cited methods encode volumetric densities and colors with neural networks, with radiance commonly restricted to 0–255.
  • Motivation: Conventional HDR imaging reconstructs images from exposure stacks but cannot render novel views, while learning-based methods require HDR supervision and can introduce motion artifacts.Large motion can cause artifacts in alignment-and-merging approaches, and image-based novel-view methods struggle with view consistency.
  • Method: HDR-NeRF recovers an HDR radiance field from multiple LDR views with different exposures using an end-to-end neural rendering method.The method is designed to learn from differently exposed LDR views rather than requiring ground-truth HDR images.
  • Method: A modeled camera response function enables HDR views and LDR views rendered under varying exposures from the recovered radiance field.The system represents scene density and radiance with a radiance field and maps radiance to colors with a tone mapper.
  • Results: On a new synthetic and real-world HDR dataset, HDR-NeRF achieves the best performance compared with SOTAs and similar major-metric scores to NeRF-GT.The method also renders arbitrary-exposure LDR views and novel HDR views in the reported comparisons.

2. Related Work

Prior work spans image-based, light-field, voxel-based, and NeRF-based novel-view synthesis, alongside exposure-stack HDR reconstruction. These approaches achieve high-quality views or HDR images, but the cited NeRF methods had not addressed novel-view synthesis with high dynamic range.

  • Novel View Synthesis: Image-based rendering methods estimate image-wise radiance for novel HDR views but may struggle to preserve view consistency in challenging scenes.The related methods combine image-based rendering with HDR imaging rather than learning a scene-wide neural radiance field.
  • Novel View Synthesis: Learning-based novel-view methods include voxel-grid, application-specific, and NeRF-family representations.The NeRF family represents scenes with neural radiance fields, while other categories use CNN-compensated voxel grids or application-specific representations.
  • Novel View Synthesis: Although recent NeRF methods achieve high-quality rendered novel views, none of them had tackled synthesizing a novel view with high dynamic range.This establishes the gap addressed by HDR-NeRF.
  • High Dynamic Range Imaging: Traditional HDR imaging reconstructs HDR images by calibrating the camera response function or directly merging differently exposed LDR images.These methods use exposure stacks captured from the same pose and can suffer ghosting with handheld cameras or dynamic scenes.
  • HDR-NeRF: HDR-NeRF’s pipeline combines an HDR radiance field for scene radiance and densities with a tone mapper for camera-response colors.This two-module design models the physical imaging process within neural rendering.

3. Background

NeRF represents scenes with an implicit function that maps ray positions and directions to colors and densities, then uses volume rendering to predict pixels. Camera response functions additionally model how irradiance and exposure time become recorded pixel values.

  • Neural Radiance Fields: NeRF maps each ray’s origin and direction to scene color and density using an implicit neural function.The ray is parameterized by origin o, direction d, and position along the ray.
  • Neural Radiance Fields: Volume rendering integrates density-weighted colors along each ray to estimate its expected pixel color.Transmittance accounts for accumulated occlusion between the ray’s near and far boundaries.
  • Neural Radiance Fields: The color reconstruction loss compares rendered pixel colors with ground-truth colors across camera rays at a target pose.The ray set is denoted R(P), where P specifies the target camera position.
  • Neural Radiance Fields: Positional encoding and coarse-to-fine optimization help NeRF represent high-frequency color and geometry variation.The coarse model biases ray-sample selection for the fine model.
  • Camera Response Functions: A camera response function maps irradiance and exposure time into stored pixel values through linear and nonlinear image processing.Here, H denotes irradiance, Z the pixel value, and Δt the exposure time.

4. HDR Neural Radiance Fields

HDR-NeRF separates scene radiance from camera tone mapping, recovering an HDR radiance field from LDR supervision while supporting both exposure-controlled LDR and HDR rendering. It combines neural radiance and density prediction with a differentiable tone mapper and volume rendering, while an auxiliary unit-exposure constraint fixes radiance scale.

  • HDR Neural Radiance Fields: HDR-NeRF aims to recover radiance over 0 to +∞ from LDR images captured with different exposures.The central challenge is aggregating differently exposed LDR observations into an HDR radiance field.
  • HDR Neural Radiance Fields: The radiance-field MLP F maps a ray to scene radiance e and density σ rather than color and density.This separates HDR scene representation from the camera-dependent conversion to pixel colors.
  • Tone Mapping: A differentiable tone mapper converts predicted radiance multiplied by exposure time into image color.The method uses separate MLPs for the RGB channels because their camera response functions may differ.
  • Tone Mapping: In the logarithm radiance domain, the tone-mapping relation becomes c(r, Δt) = g(ln e(r) + ln Δt(r)).The transformation assumes the tone mapper is monotonic and invertible, with g defined from the inverse camera response.
  • Rendering: Volume rendering produces LDR colors by integrating tone-mapped radiance with density and transmittance, while HDR rendering removes tone mapping and integrates radiance directly.Both renderings use the same near and far ray bounds and accumulated transmittance.
  • Optimization: Color reconstruction minimizes MSE between rendered and ground-truth LDR views using coarse and fine models.The loss sums errors from both predicted models over camera rays.
  • Optimization: Because color reconstruction determines radiance only up to scale α, HDR-NeRF fixes g(0) to C0 with a unit-exposure loss.The constraint assumes pixels valued C0 correspond to unit exposure; real scenes use the midway pixel value as C0.
  • Optimization: The final objective combines color reconstruction and unit-exposure losses as L = L_c + λ_uL_u.λ_u controls the weight of the unit-exposure loss.

5. Experiments

Experiments evaluate HDR-NeRF on synthetic and real scenes using quantitative comparisons, qualitative renderings, CRF estimation, and ablations. The method renders exposure-controlled LDR views and HDR views, with results comparable to NeRF-GT and improved over varying-exposure baselines in supported settings.

  • Evaluation Dataset: The evaluation uses 8 synthetic scenes and 4 real scenes, with 35 poses and 5 exposure times per real-scene pose.Synthetic scenes provide 35 HDR views per scene, which are tone-mapped into LDR inputs.
  • Quantitative Comparison: HDR-NeRF is compared with NeRF, NeRF-W, and NeRF-GT across LDR-OE, LDR-NE, and HDR metrics averaged over synthetic and real scenes.LDR-OE uses exposures t1, t3, and t5; LDR-NE uses t2 and t4.
  • Quantitative Comparison: HDR-NeRF achieves similar LDR performance to NeRF-GT on synthetic scenes and is comparable to NeRF-GT for HDR rendering, while real-scene LDR PSNR is lower.The lower real-scene PSNR is attributed to bias in the estimated blue-channel CRF caused by training-view noise.
  • Qualitative Results: HDR-NeRF renders novel LDR views close to ground truth under novel exposures, whereas NeRF shows serious artifacts and NeRF-W exhibits inconsistent color.The comparison uses zoom-in insets and error maps with MSE values.
  • Ablation Studies: Using 3 or 5 exposures substantially outperforms using 2 exposures, while independent RGB-channel CRF modeling performs better on real scenes.The exposure-count comparison reports PSNR, SSIM, and LPIPS averaged over synthetic scenes; unit exposure loss is also evaluated in an ablation.
  • Qualitative Results: HDR-NeRF controls rendered-view exposure using a specified exposure time and produces tone-mapped HDR views whose histograms and details resemble ground truth.Tone-mapped HDR views reveal details in over-exposed and under-exposed areas, and estimated CRFs model camera response functions.

6. Conclusion

HDR-NeRF recovers a high dynamic range neural radiance field from LDR views with different exposures, rendering novel HDR views and high-fidelity LDR views at specified exposures.

  • HDR-NeRF recovers a high dynamic range neural radiance field from LDR views with different exposures.
  • The method renders novel HDR views without ground-truth HDR supervision.
  • HDR-NeRF produces high-fidelity LDR views with specified exposures.
  • The method models how scene radiance is captured and mapped into image pixel values.
  • Compared with prior works, HDR-NeRF performs better in rendering LDR views and is described as the first neural rendering method to synthesize novel high-dynamic-range views.

Supplemental Materials

The supplementary materials provide implementation details, baseline descriptions, dataset information, additional results, and video comparisons for the test scenes.

  • The supplementary material includes implementation details of the method, baselines, and the collected HDR dataset.
  • Additional results are presented to further demonstrate the method’s performance.
  • A supplementary video presents results on test scenes and comparisons with baselines.

B. Additional Implementation Details

The implementation maps rays to normalized device coordinates and evaluates a Reinhard-based tone mapper that incorporates exposure value by scaling HDR pixel values.

  • During training and testing, rays are mapped from camera space to normalized device coordinate space.
  • Synthetic HDR views are converted to LDR views using a simple global tone-mapping function based on classical Reinhard tone mapping.
  • The tone-mapping function uses HDR pixel value E as its input.
  • Exposure value EV scales the HDR pixel value E to generate LDR views with different exposures.
  • The implementation treats 2^EV as the exposure time, with Δt = 2^EV.

C. Baseline Methods Implementation Details

Baseline implementations use matched key settings for fair comparison, while the HDR-view baseline is treated as an upper bound whose high dynamic range remains difficult for NeRF to encode reliably.

  • Baseline methods use the same numbers of ray samples, position encoding settings, and batch sizes as HDR-NeRF for fair comparison.
  • All models are trained with Adam for about 200,000 iterations.
  • The HDR-view NeRF baseline is considered an upper bound for HDR-NeRF.
  • Even with tone-mapped predicted colors in the loss, NeRF may fail to encode all scene areas because of the scenes’ high dynamic range.

D. HDR Dataset Details

The authors collect a new HDR dataset covering synthetic and real-world scenes to evaluate HDR-NeRF.

  • The dataset contains synthetic scenes and real-world scenes for evaluating HDR-NeRF.Synthetic HDR views are rendered with Blender’s Cycles path-tracer, while real-world LDR views at different exposures are captured using a Nikon D90 camera.

E. Additional Results

Additional experiments show that HDR-NeRF controls novel-view exposure accurately, produces HDR views close to the NeRF-GT upper bound, and models camera tone mapping effectively. Comparisons also indicate advantages over image-wise HDR reconstruction followed by vanilla NeRF.

  • Qualitative results: HDR-NeRF accurately controls exposure in rendered LDR views and produces results reasonably close to NeRF-GT.The method also better reconstructs small textures in rendered HDR views.
  • Tone-mapping analysis: HDR-NeRF correctly models the camera’s tone-mapping operation, with alternative exposure inputs producing similar results.The reported differences are PSNR: ±0.1, SSIM: ±0.02, and LPIPS: ±0.01.
  • Comparison with HDR imaging: Image-wise HDR reconstruction followed by vanilla NeRF can produce view-inconsistent HDR images and artifacts because radiance scales vary across views.HDR-NeRF is presented as an end-to-end framework with fewer inputs and better performance than this pipeline.
  • Quantitative comparisons: Tables 4–6 report quantitative comparisons on synthetic and real scenes for original-exposure LDR, novel-exposure LDR, and HDR outputs.LDR-OE averages exposures t1, t3, and t5; LDR-NE averages exposures t2 and t4.
  • Visualizations: Figures 9–11 compare rendered views with ground truth and baseline outputs, while Figure 12 displays estimated camera response functions.Figure 9 includes error maps and MSE values; Figures 10 and 11 show real-scene and synthetic-scene qualitative results.
Loading 2111.14451v4…