Source-linked AI summary
SeaThru-NeRF: Neural Radiance Fields in Scattering Media
Deborah Levy, Amit Peleg, Naama Pearl, Dan Rosenbaum, Derya Akkaynak, Simon Korman, Tali Treibitz
TL;DR
NeRF formulations assume clear-air images, leaving scattering scenes such as underwater and foggy environments unmodeled. SeaThru-NeRF incorporates a scattering-media image-formation model into NeRF and learns scene and medium properties. The authors report photorealistic novel views and recovery of clean appearance and far-scene structure in simulated and real-world scenes.
Problem
NeRF formulations assume clear-air images, leaving scenes where scattering or absorption affects image formation outside their modeled setting.
Source: §1 · p. 1Method
SeaThru-NeRF incorporates the SeaThru scattering-media image-formation model into NeRF, learning separate scene and medium properties.
Source: §1 · p. 1Results
Takeaways & Limitations
Jointly reconstructing scene and medium supports better scene reconstruction in medium and clean, and estimation of depth and medium parameters.
Source: §6 · p. 8The method does not model multiple scattering or artificial illumination, requires beforehand camera poses, and struggles when scenes violate its assumptions, such as significant underwater flickering.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Research on neural radiance fields (NeRFs) for novel view generation is exploding with new models and extensions. However, a question that remains unanswered is what happens in underwater or foggy scenes where the medium strongly influences the appearance of objects. Thus far, NeRF and its variants have ignored these cases. However, since the NeRF framework is based on volumetric rendering, it has inherent capability to account for the medium's effects, once modeled appropriately. We develop a new rendering model for NeRFs in scattering media, which is based on the SeaThru image formation model, and suggest a suitable architecture for learning both scene information and medium parameters. We demonstrate the strength of our method using simulated and real-world scenes, correctly rendering novel photorealistic views underwater. Even more excitingly, we can render clear views of these scenes, removing the medium between the camera and the scene and reconstructing the appearance and depth of far objects, which are severely occluded by the medium. Our code and unique datasets are available on the project's website.
1. Introduction
NeRF assumes clear-air images in which a ray encounters an opaque object with no medium between camera and object. SeaThru-NeRF models scattering media within volumetric rendering, separating scene and medium properties to synthesize views and recover scene information.
1. Introduction
- NeRF formulations assume clear-air images composed solely of object radiance, with a single opaque object and zero density between camera and object.That assumption does not account for significant light scattering or absorption.Source: §1 · p. 1
- SeaThru-NeRF extends NeRF to scattering media, where the whole volume—not only the object—contributes to image intensity.The framework assigns separate color and density parameters to the scene and medium, adopting the SeaThru image formation model.Source: §1 · p. 1
- SeaThru-NeRF learns separate per-color-channel medium parameters as functions of viewing angle, constant only along each 3D ray.It uses the SeaThru model, a generalization of wavelength-independent attenuation for wavelength-dependent attenuation.Source: §1 · pp. 1–2
- The method produces photorealistic novel views in simulated and challenging real-world scattering scenes, while supporting color restoration, far-area 3D structure estimation, and medium-parameter estimation.The authors report that its scene structure estimates surpass structure-from-motion and current NeRFs especially in far areas of poor visibility.Source: §1 · p. 2
2. Related Work
Related work spans NeRF extensions for challenging scenes and computational imaging, physical models of scattering, and single- or multiview scene reconstruction. SeaThru-NeRF distinguishes itself by explicitly modeling a continuous medium within rendering while jointly reconstructing scene appearance and structure.
2. Related Work
- NeRFs have also been applied to computational-imaging tasks including HDR [50], de-blurring [20], superresolution [49], low-light enhancement [23], and denoising.SeaThru-NeRF models image degradation to recover its parameters and reconstruct clean scenes and novel views.Source: §2 · p. 2
- SeaThru-NeRF differs from component-decomposition methods such as NeRF-W [21] and NeRFReN [14] by explicitly modeling a continuous medium that is not an object.Those methods decompose images into transient/static or transmitted/reflected components, whereas this method models direct and backscatter components.Source: §2 · p. 2
- Scattering-media rendering work includes radiative-transfer modeling [10], simplified rendering assumptions such as single scattering, and methods for estimating medium properties.An MLP has also been proposed for rendering synthetic atmospheric clouds [17].Source: §2 · p. 2
- Underwater image-formation models share structure with bad-weather models under ambient illumination, while artificial illumination adds nonuniform-source terms.SeaThru addresses wavelength-dependent underwater medium parameters in wideband camera channels.Source: §2 · p. 2
- Scattering-media reconstruction is ill-posed and has been approached with multiple frames, single-image priors, and deep-learning methods.A concurrent and independent work [11] presented a NeRF for haze.Source: §2 · pp. 2–3
- Unlike approaches that reconstruct 3D structure before image restoration or, SeaThru-NeRF reconstructs scene and structure simultaneously; WaterNeRF [40] estimates medium parameters separately from rendering.The authors report empirical benefits from modeling scattering in the rendering equations.Source: §2 · p. 2
3. Scientific Background
NeRF represents a scene as a continuous function and renders color by integrating contributions along camera rays, with a discretized approximation used in practice. In scattering media, image formation combines attenuated object radiance and accumulated backscatter, with medium behavior depending on the setting.
3.1. Neural Radiance Fields (NeRFs)
- NeRF represents a scene with an MLP mapping 3D position and viewing direction to density and emitted color.Along a camera ray, expected image color is accumulated from pointwise contributions weighted by transmittance and density.Source: §3.1 · p. 3
- In practice, ray rendering discretizes the near-to-far range into intervals and assumes constant density and color within each interval.The model is trained with a differentiable reconstruction loss comparing rendered training pixels with ground-truth colors, then used for novel views.Source: §3.1 · p. 3
3.2. Image Formation in Scattering Media
- In scattering media, object radiance is attenuated with distance and wavelength, while backscatter adds in-scattered light that accumulates along the line of sight.Backscatter is independent of scene contents; the combined effects reduce visibility and distort distant-object colors.Source: §3.2 · p. 3
- The adopted image-formation model represents captured intensity as direct scene radiance plus backscatter, with separate attenuation and backscatter coefficients.The model defines J as the clear scene, B∞ as backscatter color at infinity, and the coefficients as medium parameters; Figure 2 assumes at most one opaque object per ray and distinct densities for medium backscatter and object.Source: §3.2 · p. 3, Figure 2 · p. 4
- The full model uses distinct attenuation and backscatter parameters; fog or haze can use one wavelength-independent coefficient, while underwater models sometimes equate the two coefficients as a simplifying assumption.The latter underwater simplification reduces the model to three unknowns.Source: §3.2 · pp. 3–4
4. SeaThru-NeRF
SeaThru-NeRF extends NeRF volume rendering by separating object and medium contributions. It constrains and parameterizes the medium to model scattering in the rendered image.
4.1. Basic Model Derivation
- The rendering model sums distinct object and medium color contributions, and reduces to standard NeRF rendering when medium density is zero.The paper generalizes the clear-air equations to account for light contributed by the medium.Source: §4.1 · p. 4
- Medium parameters are constrained to be constant along each viewing ray, while opaque objects have near-zero density except at their locations.The medium density is separate per color channel, a less restrictive constraint than constancy per image or scene.Source: §4.1 · p. 4
4.2. Relation to Underwater Image Formation
- Under the stated opaque-object and fixed-interval setup, the rendering equations reduce precisely to the commonly used scattering-media image-formation model.In that reduced model, the medium density acts as the same attenuation coefficient for direct and backscatter components, and medium color corresponds to veiling light.Source: §4.2 · pp. 4–5
4.3. Final Model
- The final model uses distinct medium attenuation parameters for object and medium contributions, with object properties and medium parameters handled by separate networks.The object density depends on position, while the medium network takes viewing direction as input.Source: §4.3 · p. 5
4.4. Loss Function
- Training combines reconstruction, proposal, and object-transmittance normalization losses, with the latter encouraging transmittance values near 0 or 1.The reconstruction loss is adapted from RawNeRF for linear images, and λ is set to 0.0001 through cross-validation.Source: §4.4 · pp. 5–6
4.5. Implementation and Optimization
- The implementation trains for 250,000 iterations with batches of 16,384 rays, taking around 10 hours on an Nvidia A100 GPU.It builds on Mip-NeRF-360 code and disables the infinite farthest interval so the model can explain medium contributions along rays.Source: §4.5 · p. 6
5. Experiments and Results
SeaThru-NeRF improves in-medium novel-view rendering over MIP-NeRF-360 on the Red Sea validation set and performs better in most reported comparisons across real and simulated scenes. The experiments also assess clean-scene reconstruction, depth detail, and ablations of the medium parameters and rendering equations.
5.1. Experiments
- The evaluation uses underwater scenes from three seas and simulated fog and underwater Fern scenes, with comparisons against NeRF and single-image restoration methods.The real-world collections contain 20 Red Sea, 20 Caribbean Sea, and 18 Pacific Sea images; three images from each set are held out for validation.Source: §5.1 · p. 6
5.2. Results
- In the Fern simulations, the method estimates zero backscatter for a clean image and separates clean and backscatter components under fog and underwater effects.The reported underwater reconstruction has only a slight PSNR reduction.Source: Figure 4 · p. 6, §5.2 · p. 7
- ∼0.8 dB higher average validation PSNR than MIP-NeRF-360 on Red Sea; across real-world and simulation scenes, SeaThru-NeRF is better in most PSNR, SSIM, and LPIPS comparisons.The comparison uses the validation sets; Table 2 reports PSNR, SSIM, and LPIPS against the baseline.Source: Table 2 · p. 7, §5.2 · p. 7
- On Red Sea, SeaThru-NeRF separates the medium without object details in backscatter and reconstructs farther scene regions more completely than NeRF-W and NeRFReN.SeaThru is the strongest single-image method in the cited comparison, but requires depth maps and shows far-area color artifacts when depth is inaccurate.Source: §5.2 · p. 7, Figure 6 · p. 8
- The six-parameter medium model achieves higher PSNR than the one- and three-parameter variants, while the final rendering equations score lower than the basic equations.Table 1 reports average PSNR for in-medium rendering on the Red Sea set and includes other methods as well as the ablation variants.Source: Table 1 · p. 7, §5.2 · p. 8
6. Discussion
SeaThru-NeRF models scattering and absorbing media alongside opaque objects, including forward-facing scenes with object-free regions. Its all-image approach supports reconstruction and depth and medium-parameter estimation, but depends on modeling assumptions and data conditions.
6. Discussion
- The formulation represents opaque objects in a wavelength-dependent, scattering and absorbing medium, and explains forward-facing scenes that include areas with no objects.It is demonstrated on challenging real-world scenes with complex 3D structure.Source: §6 · p. 8
- Water effects in distant regions hamper single-image methods and multiview feature matching; using all images holistically supports reconstruction in medium and clean views, depth, and medium-parameter estimation.Source: §6 · p. 8
- The method does not model multiple scattering or artificial illumination, requires camera poses that can be challenging to extract in poor visibility, and learns medium parameters better with sufficient scene-range variation across viewpoints.It also struggles with scenes that violate its assumptions, such as underwater scenes with significant flickering.Source: §6 · p. 8