Source-linked AI summary
NeRF--: Neural Radiance Fields Without Known Camera Parameters
Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, Victor Adrian Prisacariu
TL;DR
NeRF−− addresses novel view synthesis from sparse forward-facing images when camera intrinsics and 6DoF poses are unknown or unavailable. It jointly optimizes camera parameters and a NeRF through photometric reconstruction, supported by BLEFF and camera-motion analyses, and achieves results comparable to COLMAP-based NeRF in the reported settings.
Problem
NeRF-based novel view synthesis generally requires camera parameters that are rarely accessible in real image-capture scenarios and may be erroneous or fail during preprocessing.
Method
NeRF−− jointly optimizes camera poses, intrinsics, and a volumetric scene representation end-to-end using photometric reconstruction from RGB images.
Results
NeRF−− achieves similar novel view synthesis quality to COLMAP-based NeRF, with perceptual quality metric ΔSSIM and ΔLPIPS = 0.05.
Takeaways & Limitations
The joint-optimization framework can recover camera parameters and produce high-fidelity novel views without pre-computing camera parameters in many forward-facing scenarios.
Takeaways & Limitations
The method is designed for forward-facing scenes, may fail beyond rotation perturbations of ±20°, and cannot handle 360° scenes or track-to-object motion reliably.
Abstract
from arXiv · showhide
Considering the problem of novel view synthesis (NVS) from only a set of 2D images, we simplify the training process of Neural Radiance Field (NeRF) on forward-facing scenes by removing the requirement of known or pre-computed camera parameters, including both intrinsics and 6DoF poses. To this end, we propose NeRF$--$, with three contributions: First, we show that the camera parameters can be jointly optimised as learnable parameters with NeRF training, through a photometric reconstruction; Second, to benchmark the camera parameter estimation and the quality of novel view renderings, we introduce a new dataset of path-traced synthetic scenes, termed as Blender Forward-Facing Dataset (BLEFF); Third, we conduct extensive analyses to understand the training behaviours under various camera motions, and show that in most scenarios, the joint optimisation pipeline can recover accurate camera parameters and achieve comparable novel view synthesis quality as those trained with COLMAP pre-computed camera parameters. Our code and data are available at https://nerfmm.active.vision.
1 Introduction
Novel view synthesis can produce realistic unseen views from sparse images, but conventional NeRF pipelines require camera parameters that are often unavailable or unreliable. NeRF−− jointly learns camera parameters and a scene representation for forward-facing scenes, while BLEFF and motion analyses evaluate this approach.
- Novel view synthesis generates unseen views from a small set of images captured from diverse viewpoints.
- NeRF and related methods achieve high-quality rendering by optimizing volumetric scene representations that capture view-dependent effects.
- Camera parameters are often unavailable in real scenarios, so conventional NeRF pipelines typically pre-compute intrinsics and extrinsics with COLMAP.
- COLMAP preprocessing adds complexity and can fail or become erroneous under homogeneous regions, changing appearance, or ambiguous camera trajectories.
- NeRF−− treats camera poses and intrinsics as learnable parameters and jointly optimizes them with a volumetric scene representation using photometric reconstruction.
- BLEFF provides path-traced synthetic scenes for benchmarking camera estimation and novel view synthesis, while motion analyses compare NeRF−− and COLMAP across perturbations.
2 Related Work
Related work spans NVS methods that assume known cameras and reconstruction systems that estimate cameras from RGB images. NeRF−− combines camera estimation with NeRF training end-to-end to retain photo-realistic rendering without requiring camera parameters as inputs.
- With Known Camera Parameters: NVS systems commonly require images together with known camera parameters to construct scene representations for novel viewpoints.
- With Unknown Camera Parameters: SLAM and SfM systems jointly estimate camera parameters and 3D geometry from RGB images using feature correspondences or photometric losses.
- With Unknown Camera Parameters: RGB-only reconstruction methods often assume diffuse appearance and cannot recover view-dependent effects, producing unrealistic novel-view renderings.
- With Unknown Camera Parameters: NeRF−− jointly optimizes camera poses, intrinsics, and a NeRF end-to-end on forward-facing scenes, unlike concurrent methods requiring intrinsics or a pretrained NeRF.
3 Preliminary
NeRF represents scenes as continuous radiance fields and renders pixels by integrating predicted radiance and density along camera rays. Training minimizes photometric discrepancies between observed and rendered images, traditionally requiring camera parameters.
- NVS seeks a scene representation that generates realistic images from novel viewpoints given sparse images and associated intrinsics and 6DoF poses.
- NeRF models view-dependent appearance with a continuous function mapping 3D locations and viewing directions to radiance color and density.
- A differentiable rendering function obtains each pixel color by aggregating radiance along a ray from the camera through the image plane.
- In practice, NeRF approximates the volume-rendering integral by accumulating radiance and densities from sampled points along each ray.
- Standard NeRF training minimizes photometric error between observed and synthesized views under known camera parameters, usually estimated with COLMAP.
4 Method
NeRF-- jointly optimises camera parameters and a volumetric scene representation from sparse forward-facing images using differentiable photometric reconstruction.
- NeRF-- represents camera intrinsics and 6DoF poses as learnable parameters jointly optimised with a scene representation.
- Camera Parameters: Under the shared-camera assumption, intrinsic estimation reduces to learning the focal length because principal points are fixed near the image centre.
- Camera Parameters: Each camera pose uses a trainable translation vector and an axis-angle rotation representation converted into a rotation matrix.
- Joint Optimisation: For each sampled image pixel, the method constructs a ray using the current focal length, camera pose, and pixel coordinates.
- Joint Optimisation: NeRF samples 3D points along each ray, predicts radiance and density, and aggregates them with the rendering function to obtain pixel colours.
- Joint Optimisation: The fully differentiable pipeline minimises reconstruction error on randomly sampled pixels, enabling joint optimisation of NeRF and camera parameters.
5 Blender Forward Facing Dataset
BLEFF is a path-traced synthetic dataset designed to evaluate camera estimation and novel view synthesis together across controlled forward-facing camera perturbations.
- BLEFF contains 14 path-traced scenes rendered with multiple levels of rotation and translation perturbations.
- Dataset Motivation: BLEFF addresses the difficulty of obtaining real images with reliable ground-truth camera parameters for joint evaluation.
- Dataset Generation: Each scene starts from a forward-facing trajectory with cameras moving in the xy-plane, then receives 16 rotation, translation, or full-6DoF trajectory variants.
- Dataset Generation: Trajectory variants encode translation and rotation perturbation magnitudes using t and r notation, with translation scaled to each trajectory’s maximum dimension.
- Dataset Motivation: Motion-capture timing lags can produce blurry renderings, while large simulated indoor datasets may lack scene diversity.
6 Experiments
Experiments evaluate NeRF-- on LLFF-NeRF, RealEstate10K, and BLEFF, comparing joint camera-scene optimisation with COLMAP-enabled NeRF across rendering quality, estimation accuracy, and camera motions. The method generally matches baseline quality, but performance depends on motion type and can fail in track-to-object or large-rotation settings.
- Novel View Synthesis Quality: NeRF-- achieves NVS quality comparable to baseline NeRF on LLFF-NeRF, with ΔSSIM and ΔLPIPS = 0.05 and ΔPSNR = 1.0.The baseline uses COLMAP-estimated camera parameters, while NeRF-- jointly optimises cameras and the scene representation.
- Novel View Synthesis Quality: On RealEstate10K, NeRF-- performs better in some examples and slightly worse in others, indicating comparable overall NVS performance.
- Camera Parameter Estimation: In BLEFF t010r010, NeRF-- has about 5° rotation error and 25-pixel focal-length error, while NVS remains comparable to baseline NeRF.COLMAP has slightly larger rotation error but lower focal-length error in this setting.
- Breaking Point in Forward-Facing Scenes: Both NeRF-- and COLMAP begin breaking beyond ±20% translation noise or ±20° rotation noise; NeRF-- is better for large translations but worse for large rotations.The authors attribute the rotation advantage to feature descriptors handling appearance changes better than photometric methods, while feature matching remains vulnerable to degenerate translation cases.
- Controlled Motion Patterns: COLMAP produces inaccurate estimates in translational experiments and can fail on degenerate RealEstate10K cases, whereas NeRF-- handles these cases better.The comparison is attributed to the difference between feature-matching-based estimation and the proposed photometry-based method.
- Controlled Motion Patterns: NeRF-- optimisation becomes inaccurate for track-to-object motion because nearly unchanged object composition and blurry early renderings provide little photometric supervision.
- Limitations and Future Work: The forward-facing assumption limits NeRF-- under rotations larger than ±20°, and the method cannot handle 360° scenes.
7 Conclusions
The paper concludes that jointly optimising camera parameters and a NeRF can remove pre-computed cameras for forward-facing NVS while achieving results on par with COLMAP-enabled NeRF. BLEFF and motion analyses support the framework’s viability, but the reported scope excludes 360° scenes.
- NeRF-- jointly optimises camera parameters and scene representation for forward-facing scenes, eliminating pre-computed camera parameters while matching the COLMAP-enabled NeRF baseline.
- BLEFF provides ground-truth camera parameters and path-traced images for evaluating jointly optimised NVS systems.
- Experiments demonstrate viability across different camera trajectories, including cases where COLMAP fails to estimate camera parameters.
Appendix A Additional Results
Additional results confirm that NeRF-- achieves comparable novel-view rendering quality to baseline NeRF while using RGB images as the only inputs.
- The appendix organises additional results around NVS quality, camera-estimation accuracy, breaking-point analysis, and optional refinement.
- On LLFF-NeRF, NeRF-- produces comparable novel-view synthesis quality to baseline NeRF using RGB images as the only inputs.
A.2 On Camera Parameter Estimation
The method accurately estimates camera poses while matching COLMAP-enabled NeRF in novel-view rendering quality. A special mirrored-pose case shows that high-quality rendering can remain possible despite a large rotation error.
- The method produces accurate pose estimations and comparable NVS rendering quality to COLMAP-enabled NeRF, with ΔSSIM = 0.
- Pose Ambiguity: Roundtable exhibits a 179.10° rotation error for COLMAP poses, yet COLMAP-NeRF still produces high-quality novel-view renderings.The mirrored poses and reconstructed geometries remain self-consistent, yielding plausible synthesis with high PSNR and SSIM.
A.3 Breaking Point Analysis
The analysis compares joint camera–NeRF optimisation with COLMAP across perturbations and describes implementation choices supporting direct optimisation. Joint optimisation is stronger under translation perturbations but weaker under rotation perturbations, while refinement can sharpen renderings.
- Breaking Point Comparison: Joint optimisation performs better than COLMAP under translation perturbation but worse under rotation perturbation.
- Camera Refinement: The optional refinement step reinitialises NeRF while retaining pretrained camera parameters, then repeats joint optimisation to generally produce sharper images.
- Per-scene Analysis: Figure 14 presents per-scene breaking-point results, where larger green areas indicate better performance and grey regions denote invalid trajectories.Invalid trajectories leave the Blender scene after perturbation, making image rendering and parameter recovery meaningless.
- Camera Refinement: Refining COLMAP-estimated camera parameters jointly with NeRF slightly improves novel-view results.
- Joint Optimisation Pipeline: The implementation jointly updates trainable camera parameters, focal length, and NeRF weights through online ray construction and photometric loss backpropagation.The pseudo-code samples points along rays, predicts radiance and density, renders images, computes loss, and updates all parameter groups.
- NeRF Implementation: The smaller NeRF uses half-width hidden layers without hierarchical sampling while retaining the original positional encoding and skip connections.
B.3 Training Details
Training uses a smaller NeRF, configurable camera initialisation and alignment, and focal-length parameterisations designed for stable optimisation. BLEFF provides path-traced scenes and controlled trajectories for evaluating camera estimation and rendering, while the method is scoped to static forward-facing scenes.
- Training Cost: A BLEFF scene with 27 training images trains in 5.5 hours on one 1080Ti GPU, compared with 5 hours for simplified original NeRF.The additional time mainly comes from constructing ray directions online from learned poses and focal length.
- Evaluation: The method aligns camera trajectories with Sim(3) and then performs photometric pose alignment before computing rendering metrics.
- Focal Length Parameterisation: The system uses one learnable focal length f rather than separate horizontal and vertical focal lengths because no performance difference was observed.
- Focal Length Parameterisation: Scaling focal length through s avoids direct optimisation in pixel units, whose larger magnitude creates numerical difficulties.
- Focal Length Parameterisation: The 2nd-order parameterisation ˜s^2 produces slightly better novel-view PSNR than the 1st-order s parameterisation.
- Scope: The method is trained per scene and assumes static forward-facing images with shared intrinsic parameters, so it cannot handle dynamic objects such as animals or humans.