Source-linked AI summary
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, Daniel Duckworth
TL;DR
NeRF is not designed for uncontrolled internet photographs, where illumination, processing, and transient objects violate its static-scene assumptions. NeRF-W extends NeRF with learned appearance variation and shared/image-dependent scene components, achieving high-fidelity, temporally consistent landmark renderings that outperform prior methods.
Problem
NeRF assumes static geometry, materials, radiance, and consistent imaging, whereas internet photo collections contain variable illumination, processing, and transient occluders.
Method
NeRF-W learns per-image appearance embeddings and decomposes scenes into shared static and image-dependent transient components.
Results
NeRF-W produces detailed, high-fidelity, temporally consistent novel views and surpasses NRW by an average 4.4dB in PSNR and up to 40% in MS-SSIM.
Takeaways & Limitations
The method supports photorealistic novel-view rendering from unstructured internet photo collections, including appearance interpolation across lighting conditions.
Takeaways & Limitations
Rendering quality degrades in rarely observed or highly oblique regions, and camera-calibration errors can cause blurry reconstructions.
Abstract
from arXiv · showhide
We present a learning-based method for synthesizing novel views of complex scenes using only unstructured collections of in-the-wild photographs. We build on Neural Radiance Fields (NeRF), which uses the weights of a multilayer perceptron to model the density and color of a scene as a function of 3D coordinates. While NeRF works well on images of static subjects captured under controlled settings, it is incapable of modeling many ubiquitous, real-world phenomena in uncontrolled images, such as variable illumination or transient occluders. We introduce a series of extensions to NeRF to address these issues, thereby enabling accurate reconstructions from unstructured image collections taken from the internet. We apply our system, dubbed NeRF-W, to internet photo collections of famous landmarks, and demonstrate temporally consistent novel view renderings that are significantly closer to photorealism than the prior state of the art.
1. Introduction
Novel view synthesis is important for computer vision and AR/VR, but NeRF assumes static, consistently illuminated scenes and therefore struggles with internet photo collections. NeRF-W relaxes these assumptions using learned appearance variation and shared versus image-dependent scene components, producing high-fidelity, temporally consistent renderings.
- Novel view synthesis from sparse captured images is a long-standing computer-vision problem and a prerequisite for many AR and VR applications.
- NeRF performs well in controlled settings but degrades with moving objects or variable illumination in large-scale in-the-wild collections.Such collections may contain images captured hours or years apart, with pedestrians and vehicles moving through scenes.
- NeRF assumes the world is geometrically, materially, and photometrically static, requiring photographs from the same position and orientation to be identical.Internet photos violate this through changes in people, vehicles, construction, seasons, weather, sunlight, exposure, color correction, and tone-mapping.
- NeRF-W models per-image appearance variations in a learned low-dimensional latent space and decomposes scenes into shared and image-dependent elements.The appearance embedding explains photometric and environmental differences, while the decomposition supports transient-content handling.
- NeRF-W produces detailed, high-fidelity novel views that surpass prior state of the art by large margins on PSNR and MS-SSIM while maintaining temporal consistency.It improves over NeRF with appearance variation and transient occluders while achieving similar quality in controlled settings.
2. Related Work
Related work spans classical reconstruction, image-based rendering, neural rendering, and methods that model lighting, reflectance, or transient objects. NeRF-W combines neural radiance-field reconstruction with learned appearance and transient representations for unconstrained photo collections.
- Novel View Synthesis: Structure-from-Motion and bundle adjustment reconstruct sparse point clouds and recover camera parameters for novel-view synthesis.
- Novel View Synthesis: Phototourism scaled reconstruction techniques to unconstrained photo collections for view synthesis, while light-field and image-based rendering generally require dense scene capture.
- Novel View Synthesis: Recent approaches infer light and reflectance properties from unconstrained collections or use semantic knowledge to reconstruct transient objects.
- Neural Rendering: Neural rendering applies image-translation networks to re-render traditional reconstruction results using representations including latent textures, point clouds, voxels, and plane-sweep volumes.
- Neural Rendering: NeRF-W addresses in-the-wild variation using an architecture that produces static and transient scene components from positions, viewing directions, and learned embeddings.
3. Background
NeRF represents a scene as a continuous volumetric radiance field modeled by MLPs, then uses volume rendering along camera rays to synthesize pixel colors. Training minimizes reconstruction error over calibrated image rays, with coarse and fine networks improving sampling efficiency.
- The scene representation maps 3D positions and viewing directions through an MLP to density and RGB color.Density uses a nonnegative activation, while color uses a sigmoid constrained to [0, 1].
- NeRF estimates each pixel’s expected color by numerically approximating the volume-rendering integral along its camera ray.The rendering combines sampled colors and densities using quadrature points between near and far planes.
- Training assumes calibrated camera parameters and consistency across views, so corresponding 3D points observed from the same position and direction should have the same intensity.
- A camera ray is defined from the camera center of projection by its origin and direction, with points parameterized as r(t) = o + td.
- NeRF jointly optimizes coarse and fine MLPs, using coarse-model density to determine fine-model quadrature sampling and minimizing reconstruction error.
4. NeRF in the Wild
NeRF-W extends NeRF for unstructured photo collections by modeling image-dependent appearance and disentangling transient objects from a shared static scene representation. It jointly learns these components, uses uncertainty to discount anomalous regions, and renders only the static scene at test time.
- NeRF-W extends NeRF to reconstruct 3D scenes from in-the-wild photo collections with violated consistency assumptions.
- Challenges: Photometric variation from illumination and imaging pipelines, together with transient objects, causes inconsistencies across internet photographs.
- Latent Appearance Modeling: Per-image appearance embeddings model exposure, lighting, weather, and post-processing while preserving shared 3D geometry.
- Transient Objects: A transient head separately predicts image-varying color and density, allowing transient phenomena to be disentangled from the static representation without explicit supervision.
- Transient Objects: Uncertainty weighting reduces the influence of pixels likely belonging to transient objects while regularization discourages transient density from explaining static phenomena.
- Rendering: At test time, NeRF-W omits transient and uncertainty fields and renders only the shared static density and color fields.
5. Experiments
Experiments evaluate NeRF-W on unconstrained landmark photo collections against NeRF, NRW, and two ablations. NeRF-W improves quantitative and qualitative reconstruction quality while modeling appearance variation, transient occluders, and view-consistent renderings.
- Evaluation setup: NeRF-W is evaluated on six Phototourism cultural-landmark scenes against NRW, NeRF, NeRF-A, and NeRF-U.The scenes include Trevi Fountain, Sacre Coeur, Brandenburg Gate, Taj Mahal, Prague Old Town Square, and Hagia Sophia.
- Ablations: NeRF-W combines the complementary strengths of NeRF-A and NeRF-U, producing sharper and more accurate renderings than either ablation.NeRF-A reduces fogginess but misses high-frequency detail, whereas NeRF-U captures detail but cannot model varying photometric effects.
- Quantitative results: NeRF-W outperforms NRW across all datasets on PSNR and MS-SSIM, improving PSNR by an average 4.4dB and MS-SSIM by up to 40%.It improves LPIPS over the prior state of the art in 3 of 6 scenes and remains competitive in the remainder.
- Controllable appearance: Appearance embeddings interpolate lighting and color while keeping geometry fixed and excluding people and lights present only in training images.The interpolations transition smoothly between endpoint appearances without introducing geometric artifacts.
- Qualitative results: NeRF-W models appearance variation, removes transient occluders, and reconstructs fine scene details simultaneously.Qualitative results show these capabilities respectively in the top, middle, and bottom examples.
- View-consistency: NeRF-W produces more view-consistent renderings with less flickering than NRW or NeRF during camera motion.NRW lacks temporal consistency for transient objects, while NeRF embeds view-dependent effects as colored fog.
- Limitations: Rendering quality degrades in rarely observed or highly oblique regions, and camera calibration errors can produce blurry reconstructions.The limitation is illustrated for areas such as the ground and applies similarly to NeRF regarding calibration sensitivity.
6. Conclusion
The paper presents NeRF-W for reconstructing complex environments from unstructured internet photographs. It learns image-specific appearance embeddings and separates transient from shared scene components, yielding qualitative and quantitative improvements over prior state-of-the-art approaches.
- Contribution: NeRF-W reconstructs complex environments from unstructured internet photo collections by extending NeRF.The method targets 3D scene reconstruction in uncontrolled photographic settings.
- Method: NeRF-W learns per-image latent embeddings for photometric variation and decomposes image-dependent and shared components to disentangle transient elements from the static scene.These components support reconstruction from in-the-wild data containing changing appearance and transient content.
- Results: Experiments on real-world and synthetic data demonstrate significant qualitative and quantitative improvement over previous state-of-the-art approaches.The conclusion reports improvement without restricting it to a single metric or dataset.
Supplemental Materials
Supplemental materials provide implementation details and additional results context. They document selected hyperparameters and encourage inspection of video results.
- Video results: The supplemental materials include video results that the authors encourage readers to inspect.The videos are presented as additional evidence for the reported rendering behavior.
- Implementation details: Hyperparameters were selected by grid search on the Brandenburg Gate scene and reused for the remaining scenes.The documented configuration includes an MLP with Base, Static, and Transient / Uncertainty components.
B. Evaluation
Evaluation on in-the-wild collections is difficult because perceptual similarity depends on appearance, while the model represents a family of appearance-conditioned solutions. The evaluation therefore estimates appearance from part of each ground-truth image rather than using a fixed embedding.
- Perceptual image similarity is challenging because aligned images of the same 3D structure can differ substantially under different lighting or times of day.
- Pre-processing discards images dominated by transient objects or falling below a NIMA quality threshold.
- After training, appearance embeddings are optimized only for training images, so test evaluation requires identifying an appearance embedding for each test image.
- NeRF-A and NeRF-W are evaluated by optimizing an appearance embedding on the left half of each ground-truth image before computing image comparisons.
- NRW encodes an entire held-out image as appearance and therefore cannot isolate geometry from appearance in the same way as NeRF-W.
C. Phototourism Dataset
The Phototourism evaluation uses internet photo collections of landmark scenes, filters low-quality or heavily occluded images, and handselects representative unobstructed test photos. Scene-level image and pixel counts are tabulated separately.
- Low-quality images and images with transient objects covering more than 80% of the area are removed during pre-filtering.
- The test set is handselected to contain well-focused photos without occluders or extreme photometric effects.
- Table 2 reports the number of images and pixels, measured in millions, for each Phototourism scene.
- Random test-image selection is avoided because PSNR, MS-SSIM, and LPIPS cannot ignore transient objects that NeRF-W is designed to remove.
D. Lego Dataset
The Lego ablation introduces controlled in-the-wild effects through random color changes and transient occluders. These perturbations isolate variable illumination, exposure, and occlusion while preserving a reference view.
- The controlled ablation constructs Lego variants inspired by effects expected in in-the-wild imagery.
- Color Perturbations: Color perturbations simulate variable lighting and exposure by randomly scaling and shifting each RGB channel.
- The first training image remains unperturbed so its appearance embedding can serve as a reference for novel-view rendering.
- Occlusions: Randomly positioned, randomly colored striped squares simulate transient occluders without a consistent 3D location across images.
D.1. Experiments
Experiments compare NeRF and its extensions on controlled and perturbed Lego data, then assess qualitative reconstruction behavior on Phototourism scenes. NeRF-W matches controlled performance while handling combined appearance and occlusion changes better than the baselines.
- D.1. Experiments: Models are trained on 100 Lego images and evaluated on 200 additional images for 125,000 steps using four GPUs.
- Original: On the original Lego dataset, all model variations perform similarly, with NeRF having slightly higher PSNR while other metrics indicate indistinguishable quality.
- Color Perturbations: Approximately 10dB of PSNR is lost by NeRF under color perturbations, whereas NeRF-A and NeRF-W preserve nearly their original metrics using appearance embeddings.
- Occluders: 14dB of average PSNR is lost by NeRF with occluders, which NeRF and NeRF-A model as colored fog that obscures the Lego figure.
- Color Perturbations and Occluders: With color and occluder perturbations together, NeRF-W outperforms all baselines, while NeRF and NeRF-U show substantial seed variation and occasional scene-modeling failures.
- Phototourism Qualitative Results: Phototourism renderings capture reflections, maintain distant scene geometry, and eliminate transient occluders.