Source-linked AI summary
Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras
Roger Marí, Gabriele Facciolo, Thibaud Ehret
TL;DR
Satellite imagery presents camera-model and appearance challenges for direct use of NeRF. Sat-NeRF combines RPC cameras with shadow-aware modeling and transient uncertainty weighting, achieving state-of-the-art satellite novel-view synthesis and 3D modeling with results similar to stereo pipelines.
Problem
Directly applying NeRF to satellite images is limited by satellite-specific camera models, large camera-to-scene distances, and multi-date appearance inconsistencies.
Source: §1 · p. 1Method
Sat-NeRF casts rays with RPC camera models and combines shadow-aware modeling with uncertainty weighting for transient phenomena.
Source: §1 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
Satellite image-processing techniques, including RPC cameras and bundle adjustment, improve reconstruction quality and can provide additional depth-supervision cues.
Source: §5 · p. 8Sampling uses altitude bounds [hmin, hmax], which may be selected from a low-resolution elevation model.
Source: §3.3 · p. 5
Abstract
from arXiv · showhide
We introduce the Satellite Neural Radiance Field (Sat-NeRF), a new end-to-end model for learning multi-view satellite photogrammetry in the wild. Sat-NeRF combines some of the latest trends in neural rendering with native satellite camera models, represented by rational polynomial coefficient (RPC) functions. The proposed method renders new views and infers surface models of similar quality to those obtained with traditional state-of-the-art stereo pipelines. Multi-date images exhibit significant changes in appearance, mainly due to varying shadows and transient objects (cars, vegetation). Robustness to these challenges is achieved by a shadow-aware irradiance model and uncertainty weighting to deal with transient phenomena that cannot be explained by the position of the sun. We evaluate Sat-NeRF using WorldView-3 images from different locations and stress the advantages of applying a bundle adjustment to the satellite camera models prior to training. This boosts the network performance and can optionally be used to extract additional cues for depth supervision.
1. Introduction
Sat-NeRF adapts neural rendering to satellite imagery, where multi-date appearance changes and satellite camera models challenge direct use of NeRF. It learns realistic views and surface geometry, with altitude error similar to handcrafted-feature stereo.
1. Introduction
- High-resolution satellite imagery supports depth estimation from multiple views for applications relying on Earth's surface geometry and its changes.Common outputs include point clouds and digital surface models.Source: §1 · p. 1
- Sat-NeRF's depth-derived DSM has lidar mean absolute altitude error similar to handcrafted-feature multi-view stereo.Figure 1 compares the network-derived DSM with a lidar equivalent.Source: Figure 1 · p. 1
- Sat-NeRF combines existing NeRF advances to handle multi-date radiometric inconsistencies, including sun-driven shadows and small transient objects.The cited examples include trees and cars in open-air parkings.Source: §1 · pp. 1–2
- Using each image's rational polynomial camera model to cast object-space rays improves results over approximate pinhole cameras and avoids dependence on a particular satellite system.The RPC camera model is used directly for ray casting.Source: §1 · p. 2
- Correcting RPC inconsistencies before training with bundle adjustment improves model performance and can provide sparse points for geometry learning.The study also reuses the sparse point cloud employed in bundle adjustment.Source: §1 · p. 2
- Evaluation uses 10–20 multi-date WorldView-3 RGB crops per area and a 0.5 m/pixel lidar DSM as geometry ground truth.The areas of interest cover 256 × 256 m; comparisons include NeRF variants and a traditional satellite stereo pipeline.Source: §1 · p. 2
2. Related work
Satellite stereo pipelines face challenges in matching, pair selection, geometry aggregation, and parameter tuning; neural rendering offers a self-supervised alternative. The section reviews NeRF’s ray-based rendering and variants addressing scene variation, satellite shadows, and depth cues.
2. Related work
- Satellite stereo relies on hand-crafted matching, uncertain stereo-pair selection, inconsistent geometry aggregation, and tuning across image sources or types.Neural rendering learns scene-adapted features and operations from image color without explicit geometry supervision.Source: §2 · p. 2
2.1. Neural Radiance Fields
- NeRF models a static scene with a continuous volumetric function, predicting color from position and viewing direction and density from position.Allowing color to depend on viewing direction supports reproducing specular reflections.Source: §2.1 · p. 2
- NeRF renders rays through a scene by integrating predicted colors at sampled 3D points, with density-derived opacity and transmittance accounting for occlusion.The network is optimized against input-image colors using mean squared error.Source: §2.1 · pp. 2–3
2.2. NeRF variants
- NeRF assumes constant scene density, radiance, and illumination, conditions the paper describes as uncommon outside laboratory settings.Source: §2.2 · p. 3
- NeRF-W separates transient phenomena from the static scene and uses predicted uncertainty to reduce unreliable pixels’ influence during training.Source: §2.2 · p. 3
- S-NeRF uses solar-ray direction to learn sunlight reaching scene points; Sat-NeRF extends it with transient-object modeling and satellite-adapted camera models.Source: §2.2 · p. 3
- DS-NeRF adds sparse 3D-point depth supervision to accelerate learning and reduce required input images; related work also converts sparse clouds into dense depth priors.Source: §2.2 · p. 3
3. Method
Sat-NeRF jointly learns static scene geometry and albedo while modeling shadows and view-specific transient phenomena, using satellite RPC cameras and bundle-adjusted inputs. Its method also uses optional bundle-adjustment points for depth supervision and a multi-task loss.
3. Method
- Sat-NeRF learns a static surface’s geometry and albedo alongside outputs for transient phenomena, without letting those phenomena induce changes in scene geometry.Its inputs include spatial coordinates, solar-ray direction, and an image-specific transient embedding; outputs include density, albedo, shading, ambient color, and uncertainty.Source: §3 · pp. 3–4
3.1. Shadow-aware irradiance model
- The irradiance model combines albedo, solar-direction-dependent shading, and ambient color to represent shadows and shadow hues.Because solar direction is closely related to acquisition date, ambient color can also capture date-specific effects such as weather or seasonal vegetation changes; a secondary solar-correction term supervises shading for unseen solar directions.Source: §3.1 · p. 4
3.2. Uncertainty weighting for transient objects
- Uncertainty weighting gives Sat-NeRF a way to discount color inconsistencies from transient objects that the static surface and solar metadata cannot explain.The per-ray uncertainty is integrated from point predictions and depends on spatial coordinates and the training-image embedding; it is introduced after the first two epochs to avoid overlooking shadows.Source: §3.2 · p. 4
3.3. Point sampling from satellite RPC models
- Sat-NeRF casts rays using each satellite image’s RPC localization function rather than replacing its camera model with a simplified pinhole camera.Ray endpoints come from localizing pixels at minimum and maximum scene altitudes; the RPC-based approach is described as more general and leads to better results than the pinhole representation used in S-NeRF.Source: §3.3 · p. 5
3.4. RPC refinement for improved performance
- Bundle adjustment corrects RPC inconsistencies across images before training, improving ray sampling and providing sparse 3D points for optional depth supervision.The adjustment uses SIFT-keypoint correspondences; depth supervision uses bundle-adjustment points during the initial 25% of training, weighted by their reprojection error.Source: §3.4 · pp. 5–6
3.5. Multi-task loss and network architecture
- The loss combines rendered-RGB supervision with secondary solar-correction and depth-supervision terms, while the network uses separate heads for static properties and shading, ambient color, and uncertainty.The paper sets λ_SC = 0.1/3 and λ_DS = 1000/3; the architecture predicts density and albedo in its main block, with smaller shading and single-layer ambient-color and uncertainty heads.Source: §3.5 · p. 6, Figure 3 · p. 6
4. Evaluation
Evaluation compares Sat-NeRF with NeRF variants and traditional stereo on WorldView-3 imagery, while ablations test its irradiance, uncertainty, solar-correction, RPC-refinement, and depth-supervision components. Sat-NeRF produces detailed DSMs with slightly better global altitude MAE than S2P, though its surfaces are less regular.
4. Evaluation
- Evaluation used 26 Jacksonville WorldView-3 images collected from 2014–2016, with RGB crops covering 256×256 m per area of interest.Training and test image counts varied by selected area of interest.Source: §4 · p. 6
4.1. Ablation study
- The irradiance model outperforms basic NeRF and benefits from solar correction; the authors attribute row 0–3 metric differences to RPC-based sampling and bundle adjustment.The comparison uses results reported in the original S-NeRF work [14].Source: §4.1 · p. 7
- Adding solar correction to Sat-NeRF further decreases altitude MAE, and this configuration outperforms all previous configurations across all AOIs.Without solar correction, Sat-NeRF also improves PSNR/SSIM and has similar or smaller altitude MAE than the best S-NeRF configuration.Source: §4.1 · p. 7
- Unrefined RPCs worsen PSNR/SSIM and altitude MAE, while sparse-point depth supervision improves altitude MAE in some areas but slightly worsens it where outliers occur.The reported degradation is in AOIs 004 and 068; gains are reported in AOIs 214 and 260.Source: §4.1 · p. 7
4.2. Comparison to traditional stereo pipelines
- Sat-NeRF’s global altitude MAE is slightly better than S2P’s, with more detailed structures but less regular surfaces.The DSM comparison also notes single-point outliers in S2P and local irregularities in Sat-NeRF.Source: §4.2 · p. 7, Table 2 · p. 8, Figure 5 · p. 8
5. Conclusion
Sat-NeRF adapts NeRF to multi-date satellite imagery, learning permanent-scene geometry and appearance alongside shadows and transient objects. It achieves state-of-the-art satellite novel-view synthesis and 3D modeling results, with DSMs that show finer details than S2P but retain local irregularities.
5. Conclusion
- Sat-NeRF DSMs show finer detail and sharper edges than S2P, but exhibit local irregularities.Figure 6 visualizes the lidar, Sat-NeRF, and S2P DSMs for area 068.Source: Figure 6 · p. 8
- A main backbone learns permanent structures’ geometry and appearance, while secondary heads learn shadows and transient objects.This architecture supports the method’s adaptation to multi-date satellite collections.Source: §5 · p. 8
- Sat-NeRF achieves state-of-the-art results in novel-view synthesis and 3D modeling from satellite imagery.The conclusion highlights RPC camera models and prior bundle adjustment as satellite-processing techniques incorporated into the NeRF framework.Source: §5 · p. 8