Source-linked AI summary
Hallucinated Neural Radiance Fields in the Wild
Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang, Jue Wang
TL;DR
Ha-NeRF addresses the challenge of reconstructing realistic, view-consistent radiance fields from tourism images with variable appearances and occluders. It uses appearance encoding and visibility modeling to hallucinate unseen appearances and separate static content, achieving competitive PSNR and SSIM while outperforming other methods on LPIPS across datasets. Its quality remains sensitive to noisy camera extrinsics and blurred or defocused inputs.
Problem
Tourism-image NeRF methods struggle to render view-consistent novel views at unlearned appearances while handling variable illumination and transient occlusions.
Method
Ha-NeRF combines a CNN-based appearance encoder and view-consistent loss with an image-dependent visibility mask and anti-occlusion loss.
Results
Ha-NeRF achieves competitive PSNR and SSIM and outperforms the other methods on LPIPS across all datasets.
Takeaways & Limitations
Ha-NeRF can directly encode appearance from an input image without using test images during training and produce competitive results against NeRF-W.
Takeaways & Limitations
Ha-NeRF is affected by noisy camera extrinsic parameters, and synthesized-image quality degrades for motion-blurred or defocused inputs.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) has recently gained popularity for its impressive novel view synthesis ability. This paper studies the problem of hallucinated NeRF: i.e., recovering a realistic NeRF at a different time of day from a group of tourism images. Existing solutions adopt NeRF with a controllable appearance embedding to render novel views under various conditions, but they cannot render view-consistent images with an unseen appearance. To solve this problem, we present an end-to-end framework for constructing a hallucinated NeRF, dubbed as Ha-NeRF. Specifically, we propose an appearance hallucination module to handle time-varying appearances and transfer them to novel views. Considering the complex occlusions of tourism images, we introduce an anti-occlusion module to decompose the static subjects for visibility accurately. Experimental results on synthetic data and real tourism photo collections demonstrate that our method can hallucinate the desired appearances and render occlusion-free images from different views. The project and supplementary materials are available at https://rover-xingyu.github.io/Ha-NeRF/.
1. Introduction
Ha-NeRF addresses realistic novel-view synthesis from tourism images with changing appearances and transient occlusions. It combines appearance hallucination with anti-occlusion modeling to transfer unseen appearances consistently and separate static scene content.
- Problem: Tourism images often vary in illumination and contain occluding objects, violating NeRF’s assumption of constant, visible scene radiance.These variations can be integrated into the reconstructed volume and disturb the recovered scene.
- Related work: NeRF-W handles variable appearances and occlusions but requires optimizing appearance embeddings for new images and struggles with unlearned, view-consistent appearances.Its controllable appearance is based on embeddings optimized from training samples.
- Ha-NeRF: Ha-NeRF reconstructs hallucinated radiance fields from unconstrained tourist images containing variable appearances and occluders.The framework is designed for realistic radiance-field recovery under both sources of variation.
- Appearance hallucination: The appearance hallucination module uses a CNN-based encoder and view-consistent appearance loss to transfer consistent photometric appearance to novel views.This design allows appearance transfer from unlearned images.
- Anti-occlusion: The anti-occlusion module learns an image-dependent 2D visibility mask with an anti-occlusion loss to separate static components during training.The module is intended to perceive ray visibility and separate static subjects accurately.
2. Related Work
Related work develops view synthesis, NeRF extensions, and appearance-recovery methods for unconstrained imagery. Existing appearance methods commonly rely on simplifying assumptions that limit nighttime or smoothly varying day-to-night reconstruction.
- Novel View Synthesis: Traditional novel-view synthesis combines image-based warping with geometric structure, while newer methods use explicit representations and alpha compositing.Deep-learning approaches also combine convolutional networks with scene geometry to predict depth or planar homographies.
- Neural Radiance Fields: NeRF uses an MLP to restore a radiance field, and subsequent work extends it to dynamic scenes, faster training and rendering, and scene editing.NeRF-W separately models appearance and occlusion through optimized appearance embeddings and static and dynamic volumes.
- Appearance Modeling: Appearance-recovery methods estimate coherent albedos, separate illumination, retrieve sun location, or assume fixed views for photo collections.These approaches target appearance variation using geometry, illumination, timestamps, geolocation, or view constraints.
- Appearance Modeling: Several methods assume simple lighting models that do not apply to nighttime appearances, while day-and-night reconstructions may lack smooth appearance gradation.This leaves limitations for modeling broad temporal and weather-related appearance changes.
3. Preliminary
NeRF represents scenes as continuous volumetric functions that map locations and viewing directions to color and density, then renders pixels by volume compositing along camera rays. Ha-NeRF extends this setup with image-conditioned appearance and visibility modeling.
- NeRF Representation: NeRF models a scene with an MLP-based continuous volumetric function that maps 3D location and viewing direction to emitted color and volume density.Positional encodings are applied to the location and direction inputs.
- Volume Rendering: NeRF renders a pixel by numerically approximating the volume-rendering integral along a camera ray.The ray is emitted from the camera center through a pixel on the image plane.
- Volume Rendering: The rendering approximation combines sampled colors and densities, where δ_k is the distance between adjacent quadrature points.Stratified sampling selects quadrature points between the camera’s near and far planes.
- Volume Rendering: Alpha compositing uses 1 − exp(−σ_kδ_k) as the ray-termination probability, while T_k represents accumulated transmittance to the sampled location.This gives the rendering weights an interpretation as termination probability modulated by prior transmission.
- Ha-NeRF Extensions: Ha-NeRF conditions synthesis on an appearance latent vector encoded from an image and maps pixel locations to image-dependent visibility values for occlusion disentanglement.The visibility model supports separating static and transient phenomena with an occlusion loss.
- Optimization: NeRF optimizes its parameters by minimizing squared differences between observed image-ray colors and corresponding rendered outputs.Camera rays are precomputed from registered intrinsic and extrinsic camera parameters, and C(r_ij) denotes the observed ray color.
4. Method
Ha-NeRF reconstructs an appearance-independent radiance field from tourism images, then conditions it on encoded appearances to hallucinate consistent novel views while separating transient occlusions from static geometry.
- Appearance Hallucination: Ha-NeRF encodes each input image into an appearance latent vector and modulates the radiance field with that representation.The learned encoder is trained jointly with the radiance field and supports appearances beyond the training set.
- View-consistent Hallucination: View-consistent loss requires an image and a randomly rendered novel view conditioned on the same input appearance to produce matching appearance vectors.This encourages appearance to remain global across views and prevents image geometry from entering the appearance representation.
- View-consistent Hallucination: The method samples a random grid of rays rather than rendering a whole image during training, assuming the global appearance vector remains unchanged after sampling.A second camera view is rendered to form the hallucinated image used for consistency training.
- Occlusion Handling: An image-dependent 2D visibility map predicts each pixel’s static-scene visibility, replacing a 3D transient field for more accurate static–transient separation.An implicit function maps pixel location and a transient embedding to visible possibility, trained with an occlusion loss.
- Occlusion Handling: The occlusion loss weights reconstruction errors by visibility and regularizes invisible probability so the model does not disregard static phenomena.The full Ha-NeRF objective jointly trains the radiance field, appearance encoder, visibility model, and per-image transient embeddings.
- Optimization: 24.04 PSNR, .8773 SSIM, and .1391 LPIPS are reported for Ha-NeRF on Brandenburg Gate, while the method outperforms alternatives on LPIPS across all datasets.The qualitative results show transferred appearances and removal of transient occlusions in novel views.
5. Experiments
Ha-NeRF is evaluated on tourism-image datasets for appearance hallucination, interpolation, cross-dataset transfer, and anti-occlusion rendering. It combines an appearance encoder with visibility modeling, producing realistic, appearance-consistent, and occlusion-reduced novel views.
- Experimental setup: Ha-NeRF is evaluated on three Phototourism landmark datasets: Brandenburg Gate, Sacre Coeur, and Trevi Fountain.Images are downsampled by two times during training, and comparisons include NeRF, NeRF-W, and two Ha-NeRF ablations.
- Ablation comparison: Ha-NeRF combines the strengths of its appearance and transient ablations, producing better appearance and anti-occlusion renderings.The appearance ablation misses high-frequency details, while the transient ablation cannot model varying photometric effects.
- Quantitative comparison: Ha-NeRF achieves competitive PSNR and SSIM against NeRF-W and outperforms the other methods on LPIPS across all datasets.Unlike NeRF-W, Ha-NeRF does not use test images during training to encode test appearance.
- Appearance hallucination: Ha-NeRF conditions rendered color on an encoded appearance vector, enabling lighting changes and example-guided appearance transfer without altering 3D geometry.The appearance encoder supports transferring appearance from example images.
- Appearance hallucination: Ha-NeRF captures high-frequency appearance details, including sunshine and colored light reflections, while NeRF-W suffers from global color shifts.These comparisons are shown for Brandenburg Gate and Trevi Fountain hallucinations.
- Appearance interpolation: Ha-NeRF smoothly interpolates appearances between two encoded endpoints, whereas NeRF-W ignores sunset glow and entangles variable appearance with transient people.Adding NeRF-W’s transient field restores sunset glow but exposes its poor disentanglement.
- Cross-appearance hallucination: Ha-NeRF transfers appearance across datasets and radically different scenes, while NeRF-W requires embeddings optimized on images depicting the same place.The cross-dataset demonstration conditions Brandenburg Gate on Trevi Fountain appearance examples.
- Occlusion handling: Ha-NeRF’s visibility map separates static scenes from transient objects, enabling occlusion-free renderings where NeRF-W leaves boards, people, and fences.NeRF-W also entangles variable appearance and transient occlusion in its 3D transient field.
6. Conclusion
Ha-NeRF recovers realistic radiance fields from tourism images with variable appearances and complex occlusions. It combines appearance hallucination with anti-occlusion processing to render free-occlusion views under hallucinated appearances.
- Ha-NeRF recovers realistic radiance fields from images with variable appearances and complex occlusions.
- Its appearance hallucination module transfers time-varying appearances to novel views.
- Its anti-occlusion module learns an image-dependent 2D visibility mask to separate static subjects accurately.
- Experiments on synthetic data and tourism photo collections demonstrate free-occlusion rendering and appearance hallucination.
- Codes and models are made publicly available to facilitate reproducible research.