Source-linked AI summary
NeRF for Outdoor Scene Relighting
Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, Christian Theobalt
TL;DR
Outdoor relighting requires separating geometry, reflectance, and illumination so lighting can be edited without altering scene structure, yet existing methods do not support this broadly in uncontrolled outdoor settings. NeRF-OSR uses a self-supervised neural scene representation with explicit intrinsic decomposition and shadow modelling to edit illumination and viewpoint. It enables controllable, higher-quality outdoor relighting with realistic self-shadowing and is evaluated using a new calibrated benchmark dataset.
Problem
Existing relighting methods do not broadly provide semantically meaningful illumination and viewpoint control for outdoor scenes captured under uncontrolled conditions.
Method
NeRF-OSR learns a self-supervised neural scene representation that decomposes occupancy, illumination, shadowing, and diffuse albedo while rendering outdoor scenes from arbitrary viewpoints and illuminations.
Results
NeRF-OSR enables simultaneous illumination and viewpoint editing with higher-quality relighting and realistic shadow reproduction, supported by a calibrated benchmark containing eight buildings, 3240 viewpoints, and 110 times.
Takeaways & Limitations
The approach provides direct, semantically meaningful control over outdoor scene illumination, including local shading, shadows, and albedo, from uncontrolled multi-view photographs.
Takeaways & Limitations
The spherical-harmonics illumination model remains limited for high-frequency, specular, and spatially varying illumination, including challenging nighttime conditions.
Abstract
from arXiv · showhide
Photorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination. A delicate manipulation of the lighting can then be performed while keeping the scene albedo and geometry unaltered. We present NeRF-OSR, i.e., the first approach for outdoor scene relighting based on neural radiance fields. In contrast to the prior art, our technique allows simultaneous editing of both scene illumination and camera viewpoint using only a collection of outdoor photos shot in uncontrolled settings. Moreover, it enables direct control over the scene illumination, as defined through a spherical harmonics model. For evaluation, we collect a new benchmark dataset of several outdoor sites photographed from multiple viewpoints and at different times. For each time, a 360 degree environment map is captured together with a colour-calibration chequerboard to allow accurate numerical evaluations on real data against ground truth. Comparisons against SoTA show that NeRF-OSR enables controllable lighting and viewpoint editing at higher quality and with realistic self-shadowing reproduction. Our method and the dataset are publicly available at https://4dqv.mpi-inf.mpg.de/NeRF-OSR/.
1 Introduction
NeRF-OSR addresses outdoor scene relighting from uncontrolled photographs by jointly enabling illumination and viewpoint editing through an explicit, decomposed neural scene representation. The work also introduces a benchmark dataset designed for quantitative evaluation under varied viewpoints and illumination.
- Its explicit intrinsic controls include local shading, shadows, and albedo, supporting semantically meaningful lighting edits while preserving other scene components.
- NeRF-OSR is presented as the first neural-radiance-field approach for simultaneously editing outdoor scene illumination and camera viewpoint in uncontrolled photographs.
- The method models image formation and decomposes scenes into spatial occupancy, illumination, shadowing, and diffuse albedo reflectance.
- NeRF-OSR is trained self-supervised on outdoor images captured from various viewpoints and under different illuminations.
- The benchmark contains eight buildings photographed from 3240 viewpoints and at 110 different times, with colour-calibrated 360° environment maps for numerical evaluation.
2 Related Work
Prior outdoor relighting methods differ in input setting, scene scope, and physical control, while NeRF-based approaches often lack outdoor shadow modelling or semantic intrinsic controls. NeRF-OSR is distinguished by explicit scene decomposition and a dedicated shadow component for relightable outdoor scenes.
- Outdoor relighting methods include object insertion, full-scene processing, single-image inverse rendering, and self-supervised learning from uncontrolled outdoor images.
- Several NeRF-based methods require single or known illumination during training or target specific object classes, limiting their match to uncontrolled outdoor settings.
- NeRD handles scenes observed under different illuminations but does not explicitly model shadows, which are described as crucial for high-quality outdoor relighting.
- Style-based appearance editing lacks physical understanding of illumination and explicit parametric control over local shading and shadows.
3 Method
NeRF-OSR extends neural radiance fields into a relightable outdoor-scene representation by explicitly modelling albedo, illumination, surface normals, and shadows. It is trained self-supervised on uncontrolled outdoor images and supports rendering from user-specified viewpoints and illumination conditions.
- Overview: NeRF-OSR takes multi-view outdoor RGB images captured at different times and viewpoints, then renders scenes under arbitrary viewpoints and illuminations.The method is self-supervised and explicitly estimates scene intrinsics and illumination, including a dedicated shadow component.
- Neural Radiance Fields: Standard NeRF renders novel views but lacks semantic scene-intrinsic control and direct lighting manipulation.Its colour representation models fixed-lighting appearance rather than an explicitly relightable scene.
- Spherical Harmonics NeRF: NeRF-OSR introduces a second-order spherical-harmonics lighting model that combines accumulated albedo with illumination evaluated at accumulated surface normals.The per-image SH coefficients provide explicit control over relighting, while the normal is extracted from accumulated ray density.
- Shadow Generation Network: A dedicated shadow network predicts a scalar shadow value from spatial samples and grayscale SH coefficients, enabling efficient shadow-aware relighting.The shadow estimator uses a single forward pass alongside albedo and geometry and supports new illumination conditions.
- Objective Function: The training objective combines image-colour reconstruction with shadow regularisation so the shadow network does not absorb illumination that should be represented by SH.Removing the regulariser generally causes the shadow network to learn most grayscale illumination components, making SH lighting ineffective.
4 A New Benchmark for Outdoor Scene Relighting
The paper introduces a larger outdoor relighting benchmark designed for accurate real-data evaluation against ground truth. It expands site, viewpoint, weather, and session coverage while colour-correcting environment maps against DSLR recordings.
- Existing outdoor datasets mainly target 3D reconstruction and lack environment maps needed for numerical relighting evaluation against ground truth.
- Yu et al.’s benchmark provides environment maps but is limited to one site and lacks colour calibration between the maps and DSLR recordings.
- The new benchmark contains eight sites, 3240 viewpoints, and 110 recording sessions across multiple viewpoints, times, and weather conditions.
- Each recording session includes a 360° environment map and a simultaneously captured GretagMacbeth ColorChecker for DSLR-to-environment-map colour calibration.
- The dataset targets scenes with minimal specular effects and uses exposure brackets for both DSLR photographs and environment maps.
5 Results
NeRF-OSR is evaluated on diverse real-world outdoor sites for relighting, novel-view synthesis, interactive rendering, and intrinsic editing. Across these applications, the model supports illumination and viewpoint changes, realistic shadows, and independent albedo or shadow manipulation.
- Evaluation setup: The evaluation covers sites containing sculptures, trees, umbrellas, self-shadowing surfaces, water, and surrounding buildings.
- Relighting comparison: NeRF-OSR simultaneously edits illumination and viewpoint, whereas the primary comparison methods edit illumination only.
- Relighting comparison: NeRF-OSR models scene intrinsics and dedicated shadows, producing photorealistic results with both soft- and hard-shadow synthesis.
- Quantitative evaluation: NeRF-OSR outperforms related methods qualitatively and quantitatively when reproducing novel illumination from ground-truth environment maps.
- Quantitative evaluation: The full NeRF-OSR model achieves the best performance in the reported ablation study, with metrics computed over masked regions.
- Quantitative evaluation: The advantage over Yu et al. remains after downscaling NeRF-OSR outputs to Yu et al.’s default resolution.
- Interactive rendering: The extracted geometry and albedo support real-time VR rendering with user-controlled sun direction on the reported desktop system.
- Intrinsic editing: The model permits independent post-render editing of scene albedo and shadow strength without changing illumination or other intrinsic components.
6 Discussion and Conclusion
The second-order spherical harmonics model produces plausible relightings, while learned shadows help reproduce hard outdoor shadows. The method remains limited by illumination-model expressiveness and geometry-estimation inaccuracies.
- Discussion and Conclusion: Second-order SH lighting produces plausible relightings, with learned shadows compensating for shadows that SH may represent poorly.The learned shadow component can reproduce novel hard shadows, including in sunlit environments.
- Discussion and Conclusion: The SH illumination model is limited for high-frequency illumination, specularities, and spatially varying illumination.Capturing these effects could support view-dependent effects and more challenging scenes, including nighttime conditions.
- Discussion and Conclusion: Geometry inaccuracies can cause blur because small geometric bumps substantially alter normals and computed illumination.The model may smooth geometry to obtain more accurate lighting, while more sophisticated geometry models remain a future direction.
–Supplementary Material–
The supplement provides additional technical and experimental material for NeRF-OSR, including dataset details, spherical harmonics estimation, ablations, experiments, and visualizations.
- Supplementary Material: The paper identifies Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt as authors.The listed affiliations include MPI for Informatics, Saarland University, and the University of York.
- Supplementary Material: The authors are affiliated with MPI for Informatics, Saarland University, and the University of York.The affiliations are listed as MPI for Informatics and SIC, Saarland University and SIC, and the University of York.
- Supplementary Material: The supplemental document provides more details on the new dataset for outdoor scene relighting.It also covers spherical harmonics environment estimation, ablative studies, experiments, and additional visualizations.
A SH Environment Estimation
NeRF-OSR represents target illumination with spherical harmonics coefficients estimated from 360-degree environment photos. These coefficients support numerical evaluation against ground truth and direct user control for interactive relighting.
- A SH Environment Estimation: Target illumination is represented through spherical harmonics coefficients.The coefficients define the illumination used in the relighting experiments.
- A SH Environment Estimation: The SH coefficients are estimated from a 360-degree environment photo using least squares.This follows the estimation approach described as being in line with Yu et al.
- A SH Environment Estimation: 360-degree environment maps enable numerical evaluation against ground truth, while users can directly control SH coefficients in an interactive relighting application.The passage connects environment capture to evaluation and coefficient control to interaction.
B Statistics of the Dataset
The benchmark dataset contains eight outdoor sites recorded across 110 sessions, with 3240 views and environment-map and colour-calibration data enabling real-data evaluation against ground truth.
- B Statistics of the Dataset: 3240 views were captured across eight outdoor sites and 110 different sessions.The dataset statistics report the sites, sessions, and total number of views together.
- B Statistics of the Dataset: Each session includes a 360-degree environment map, and the dataset includes captured colour chequerboards.Together, these enable numerical evaluation of relighting methods on real data against ground truth.
- B Statistics of the Dataset: The dataset is presented as a benchmark for numerical evaluation of outdoor scene relighting against ground truth.Its evaluation capability is attributed to the environment maps and captured colour chequerboards.
C Ablative Study
The ablation study shows that frequency annealing and shadow regularization materially affect relit novel-view quality.
- Removing frequency annealing causes circular artefacts on the ground and visible artefacts on the building.
- Removing the shadow regularizer often makes the shadow layer absorb illumination components.
D Video Results
Video results demonstrate relighting and viewpoint editing separately and jointly, including unseen combinations and synthetic illumination. The method also recovers interpretable scene intrinsics and produces strong hard shadows.
- NeRF-OSR edits camera viewpoint and illumination independently or simultaneously, including combinations unseen during training.The demonstrations include fixed-view timelapse relighting, viewpoint changes under fixed illumination, and joint changes.
- The method produces photorealistic outputs under known, novel, and even unrealistic synthetic lighting.Some displayed lighting conditions cannot occur in real life, yet the synthesized results remain highly photorealistic.
- NeRF-OSR recovers interpretable normals, albedo, shadows, and shading, enabling independent manipulation of scene intrinsics.The results include strong hard-shadow synthesis and separate editing of albedo and shadows.
E Additional Renderings of Various Sites
Additional figures show reconstruction, novel-view synthesis, relighting, and novel lighting-and-viewpoint combinations across multiple dataset sites. The visualizations also include design-choice comparisons and synthetic or natural illuminations.
- Additional renderings cover reconstruction, novel-view synthesis, and lighting synthesis across more sites in the proposed dataset.
- The visualizations include novel lighting and viewpoint synthesis, while sky inaccuracies are outside the work’s scope.
- Design-choice comparisons evaluate four view-lighting combinations against ground truth, with the full model giving the best result.
- Additional examples show relighting under both natural and unrealistic light sources and qualitative comparisons with NeRF-W.