Source-linked AI summary
NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images
Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T. Barron
TL;DR
Conventional NeRF relies on tonemapped LDR images that lose dynamic range and distort raw sensor noise, limiting reconstruction in dark or high-contrast scenes. RawNeRF instead trains on linear raw images and uses many posed views to produce HDR scene representations, enabling flexible rendering and near-darkness reconstruction. The resulting method outperforms NeRF on noisy datasets and is competitive with multi-image denoisers for wide-baseline static scenes.
Problem
Tonemapped LDR inputs clip highlights, compress dynamic range, and bias raw noise, limiting NeRF for dark and high-contrast scenes.
Method
RawNeRF modifies NeRF to reconstruct scenes in linear HDR color space by supervising directly on noisy raw images and posed multiview inputs.
Results
RawNeRF outperforms NeRF on noisy real and synthetic datasets and is competitive with multi-image denoisers for wide-baseline static scenes.
Takeaways & Limitations
RawNeRF enables HDR novel-view applications that vary exposure, tonemapping, and focus, while reconstructing scenes captured in near-darkness.
Takeaways & Limitations
RawNeRF requires large raw-image storage, robust COLMAP camera poses, static scenes, and substantially more computation than feed-forward networks.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) is a technique for high quality novel view synthesis from a collection of posed input images. Like most view synthesis methods, NeRF uses tonemapped low dynamic range (LDR) as input; these images have been processed by a lossy camera pipeline that smooths detail, clips highlights, and distorts the simple noise distribution of raw sensor data. We modify NeRF to instead train directly on linear raw images, preserving the scene's full dynamic range. By rendering raw output images from the resulting NeRF, we can perform novel high dynamic range (HDR) view synthesis tasks. In addition to changing the camera viewpoint, we can manipulate focus, exposure, and tonemapping after the fact. Although a single raw image appears significantly more noisy than a postprocessed one, we show that NeRF is highly robust to the zero-mean distribution of raw noise. When optimized over many noisy raw inputs (25-200), NeRF produces a scene representation so accurate that its rendered novel views outperform dedicated single and multi-image deep raw denoisers run on the same wide baseline input images. As a result, our method, which we call RawNeRF, can reconstruct scenes from extremely noisy images captured in near-darkness.
1. Introduction
RawNeRF adapts NeRF to noisy linear HDR raw images, preserving information lost by LDR postprocessing while enabling HDR rendering and near-dark reconstruction.
- RawNeRF reconstructs scenes directly in linear HDR color space from noisy raw input images.
- Raw inputs preserve dynamic range and noise characteristics that camera postprocessing compresses, clips, or biases in LDR images.
- RawNeRF enables rendered-view changes to exposure, tonemapping, and focus, including synthetically rendered bokeh.
- RawNeRF uses NeRF’s joint optimization across many images to reduce noise and reconstruct scenes captured in near-darkness.
- With 25-200 input images, RawNeRF can remove more noise than denoisers using only 1-5 images per output.
- The method outperforms NeRF on noisy real and synthetic datasets and competes with multi-image denoisers for wide-baseline static scenes.
2. Related Work
Related work spans neural view synthesis, raw-image denoising, HDR imaging, and synthetic defocus, providing the components and applications surrounding RawNeRF.
- Novel view synthesis: Novel view synthesis reconstructs a scene from images and camera poses so that new views can be rendered.
- Novel view synthesis: NeRF optimizes a neural volumetric scene representation to match input images through gradient descent on a rendering loss.
- Raw-image denoising: Raw-image denoising research found that denoisers trained on synthetic sRGB noise failed to generalize to real raw images.
- Raw-image denoising: Noise2Noise showed denoising can use pairs of independent noisy observations without corresponding clean ground truth.
- Computational photography: Raw data preserves dynamic range and permits post-capture changes to exposure, white balance, and tonemapping.
- Computational photography: HDR reconstruction methods recover or hallucinate clipped-highlight detail from single or multiple LDR inputs.
- Synthetic defocus: Synthetic defocus methods approximate lens rendering with depth maps and depth-varying blur kernels.
- Synthetic defocus: HDR-space blur is critical for correctly rendering defocused bright highlights, or bokeh.
3. Noisy Raw Input Data
Raw image formation begins with noisy high-bit-depth sensor measurements and camera color processing, followed by nonlinear LDR tonemapping that alters the noise distribution.
- Raw camera measurements: Raw sensors record high-bit-depth digital signals from photon arrivals, typically using 10 to 14 bits.
- Raw camera measurements: After black-level subtraction, raw measurements combine shot and read noise whose error is zero mean.
- Color filter demosaicking: Bayer-filtered sensors measure one red, green, or blue channel per pixel, requiring demosaicking to produce full-resolution color.
- Color correction and white balance: Color correction and white balance use a linear 3 × 3 transform that correlates noise between color channels.
- Gamma compression and tonemapping: Gamma compression and tonemapping clip values outside [0, 1] and apply a nonlinear curve that allocates more encoding precision to dark regions.
- Gamma compression and tonemapping: Tonemapping maps linear HDR values to nonlinear LDR space and most drastically changes the noise distribution through clipping and nonlinear transformation.
4. RawNeRF
RawNeRF adapts NeRF to reconstruct scenes directly from noisy linear raw HDR images, addressing lost detail and biased noise introduced by LDR processing. Its loss, exposure handling, and multiview aggregation support HDR editing, extreme dynamic-range capture, and denoising.
- NeRF represents scenes with a neural network that maps 3D position and viewing direction to density and color for volumetric rendering.
- Standard LDR conversion clips bright detail, compresses image detail, and makes per-pixel noise biased through nonlinear tonemapping and clipping.
- RawNeRF trains NeRF directly on linear raw HDR data, preserving dynamic range for post hoc exposure, tonemapping, focus, and bokeh editing.
- 4.1. Loss function: A gradient-supervision loss uses a log tonemap weighting to emphasize dark-region errors while avoiding bias from applying a nonlinear tonemap directly to noisy observations.
- 4.2. Variable exposure training: Variable-exposure training scales raw outputs by shutter speed, learns per-channel corrections for miscalibration, and clips saturated predictions before applying the loss.
- 4.3. Implementation details: RawNeRF combines information across wide-baseline noisy views while jointly synthesizing and denoising, achieving competitive denoising results and outperforming LDR NeRF and Un+RawNeRF.
5. Results
RawNeRF is robust to noisy wide-baseline inputs and supports HDR view synthesis beyond conventional LDR NeRF. Its reconstruction can denoise across many views while enabling post hoc exposure, tonemapping, and focus manipulation.
- Synthetic noise ablation: With any nonzero noise, optimizing NeRF in raw space is more beneficial than LDR training, although LDR is superior for perfectly clean inputs.Raw-space optimization preserves an unbiased noise distribution.
- Synthetic noise ablation: 120 synthetic HDR training images support an ablation over shot and read noise at 8 shutter speeds, evaluated with PSNR in LDR sRGB space.The synthetic experiment uses the Lego scene and reports PSNR only on the object region.
- Denoising: RawNeRF is competitive with dedicated deep denoisers while simultaneously performing denoising and novel view synthesis.It combines information across wide-baseline inputs without explicitly learned image priors or clean training data.
- Denoising: RawNeRF outperforms LDR NeRF on noisy real and synthetic data, while preprocessing inputs with a single-image denoiser produces oversmoothed renderings.Despite LDR NeRF directly minimizing sRGB mean-squared error, RawNeRF achieves better sRGB metrics.
- HDR view synthesis: RawNeRF renders linear HDR outputs whose exposure and tonemapping can be changed after reconstruction, and whose depth layers support synthetic defocus.In nighttime scenes, it recovers bright-source intensity for oversaturated bokeh effects and 3D refocusing unavailable from a single image and depth map.
6. Discussion
RawNeRF broadens robust scene capture to noisy, high-dynamic-range environments, but its benefits come with substantial storage, pose-estimation, motion, and computation constraints.
- Limitations: RawNeRF requires minimally compressed full-resolution raw images, creating huge storage requirements for tens or hundreds of images per scene.This is a practical tradeoff of capturing many raw inputs.
- Limitations: COLMAP pose estimation can prevent RawNeRF from capturing scenes below a certain light level.Jointly optimizing RawNeRF and camera poses is suggested as a possible remedy.
- Limitations: RawNeRF is not a general-purpose denoiser because it cannot handle scene motion and requires orders of magnitude more computation than a feed-forward network.Its denoising capability is tied to static scenes and multiview reconstruction.
- Implications: RawNeRF can make wider-dynamic-range and darker-scene reconstruction possible through variable-exposure raw training and noise robustness.The authors frame this as increasing the fraction of environments available for photorealistic view synthesis.
- Limitations: RawNeRF occasionally produces high-frequency grid-like patterns due to positional-encoding bias, despite its minimal tendency to hallucinate.The method largely averages input data rather than relying on generative image priors.
B. Additional qualitative results
Additional qualitative results show RawNeRF outputs for dark and high-contrast scenes, with the authors directing readers to the supplemental video for animated comparisons.
- Qualitative results: Additional qualitative results cover both dark and high-contrast scenes in Figures 9 and 10.The authors note that the supplemental video presents the results more compellingly when animated.
C.1. Full derivation of gradient-weighted loss
The gradient-weighted loss uses a local linearization around the network prediction so training can approximate tonemapped-loss gradients while converging toward an unbiased result. A weight-variance regularizer separately suppresses floater artifacts, with stronger weighting in noisier scenes but a sharpness tradeoff.
- The derivation linearizes the error term around the prediction ˆyi because it tends toward the true signal xi = E[yi].
- Under weighted L2 training, ˆyi → E[yi] = xi in expectation, making the reweighted gradient approach ψ′(xi)(ˆyi−yi).
- The reweighted loss gradient is a linear approximation of the gradient from the tonemapped loss.
- C.2. Weight variance regularizer: The weight-variance regularizer is defined from compositing weights over ray segments and their piecewise-constant probability distribution.
- 1 × 10−2 to 1 × 10−1 weights are applied relative to rendering loss, with higher values for noisier scenes; high weights can slightly reduce sharpness.Annealing the regularizer from 0 to 1 can ameliorate the sharpness loss.
- A sweep found ϵ = 1 × 10−3 and p = 1 best qualitatively, while standard unweighted losses tended to diverge early.Reweighted L1 and a camera-noise negative log-likelihood performed worse than reweighted L2; alternative tone-curve treatments also caused artifacts or divergence.
C.4. Quality limitations
The quality pipeline converts raw sensor data into normalized, color-corrected linear RGB for RawNeRF training and applies exposure and tonemapping for visualization. Real-world quality remains limited by extreme noise, pose estimation, storage, and artifacts from unstable optimization.
- Quality limitations: RawNeRF cannot scale to arbitrary noise in real scenes because COLMAP may fail or produce poorly aligned poses at very low light.Nighttime captures may require repeated COLMAP runs or parameter tuning, with corner misalignment persisting in some reconstructions.
- Optimization stability: Gradient clipping is necessary because high supervision noise can make standard L2 losses produce NaN gradients, whereas the reweighted loss is more stable.
- Raw-to-RGB pipeline: Raw images are loaded as 12-bit data, converted to 32-bit floats, and rescaled so black level is 0 and white level is 1 for RawNeRF training.
- Raw-to-RGB pipeline: The pipeline applies demosaicking, white balance, and a combined color correction transform from camera RGB through canonical XYZ to RGB.
- Raw-to-RGB pipeline: Exposure is adjusted to place the white level at the 97th percentile, after which the standard sRGB gamma curve can tonemap linear RGB data.
D.3. Camera shutter speed miscalibration
Shutter-speed miscalibration causes exposure-normalized brightness to vary with speed and color channel, producing visible color shifts that must be corrected before comparing variable-exposure captures.
- Correction and evaluation: RawNeRF learns per-color-channel scaling for each shutter speed to account for miscalibration in variable-exposure inputs.
- Measured miscalibration: Perfect calibration would keep exposure-normalized brightness at 1, but measured brightness decays at faster shutter speeds and at different rates across Bayer channels.The effect remains when using the correct black level 528 and worsens when the black level is shifted.
- Visible color shift: Shutter speeds of 1/1104 and 1/181 seconds produced raw-domain channel shifts of (0.89, 0.93, 0.75) for red, green, and blue.Undoing this shift before postprocessing restores the global color balance qualitatively.
- Correction and evaluation: Because color alignment is unreliable across exposures, the real test set primarily reports affine-aligned metrics.
- Possible causes: The issue is not fully understood and may involve sensor temperature, fast-exposure timing, or other low-level hardware factors.A DSLR or mirrorless camera with a better sensor may avoid the issue.
E.2. Synthetic Lego dataset details
The synthetic Lego experiment uses rendered linear-space images with controlled training and test poses, while its preprocessing creates a clean upper-bound condition. Reported unmasked PSNR is strongly affected by LDR NeRF color bias in black backgrounds.
- Synthetic preprocessing: Unprocessing and reprocessing omit remosaicking, demosaicking, and quantization so the infinite-shutter-speed case remains perfectly clean.This provides an upper bound on possible performance.
- Synthetic preprocessing: The diffusely lit Lego object does not strongly test RawNeRF’s ability to encode high dynamic range.
- Dataset construction: The synthetic Lego dataset contains 120 training images and 40 test images rendered from randomly sampled Blender poses in linear color space.Metric values are not comparable to the original scene because the poses and postprocessing pipeline differ.
- Evaluation: Unmasked LDR sRGB PSNR is heavily skewed by LDR NeRF’s color bias in black background regions.Table 3 reports unmasked values, while the main text reports masked PSNR in Table 2.
F.1. Training with iPhone JPEG inputs
The supplement contrasts RawNeRF’s raw-domain processing with standard JPEG-based NeRF inputs and documents rendering and scene-metadata details. It also examines Bayer masking choices and synthetic defocus rendering.
- F.1. Training with iPhone JPEG inputs: LDR NeRF comparisons use a custom pipeline that converts raw data into LDR sRGB inputs, whereas standard NeRF typically uses camera JPEG images.The JPEG pipeline may include stronger noise reduction and nonlinear tonemapping.
- F.1. Training with iPhone JPEG inputs: Applying the loss only to each ray’s Bayer-measured color channel avoids the bright color noise observed when supervising all three bilinearly demosaicked channels.The implementation renders all three colors but selects the measured channel with a one-hot mask.
- F.1. Training with iPhone JPEG inputs: Hot pixels may amplify artifacts after bilinear demosaicking, but preliminary experiments found no benefit from additionally masking them; a four-pixel image border was removed.The hot-pixel explanation is presented as a potential cause rather than a confirmed mechanism.
- Synthetic defocus rendering: Defocused images are rendered from a precomputed MPI containing fronto-parallel RGBA planes in linear HDR space, sampled linearly in disparity.The MPI is computed from the trained RawNeRF at a central camera pose to reduce rendering cost.
- Synthetic defocus rendering: In synthetic defocus rendering, ifocus selects the focal plane, ∆r controls aperture size, and ∆d controls in-plane camera translation.Circular blur kernels, Fourier convolution, bilinear translation, and alpha compositing produce the final image.
- Scene metadata: Table 4 summarizes scene image metadata, including shutter speed, ISO, and time of day, while supplement figures use an “A” prefix.The table covers scenes shown in the paper and video.