Source-linked AI summary
NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior
Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, Victor Adrian Prisacariu
TL;DR
Unposed-NeRF methods struggle with dramatic camera motion, motivating a method that jointly estimates poses and a NeRF without pre-computed poses. NoPe-NeRF corrects monocular-depth scale and shift distortions, uses the resulting maps to constrain consecutive-frame relative poses, and reports improved rendering and pose accuracy on challenging scenes.
Problem
Existing unposed-NeRF methods can jointly optimise poses and a NeRF in forward-facing scenes but struggle with dramatic camera motion.
Method
NoPe-NeRF optimises monocular-depth scale and shift parameters during training, then uses undistorted maps with inter-frame and depth-based surface-rendering losses to constrain relative poses.
Results
NoPe-NeRF handles challenging camera trajectories and outperforms existing methods in novel view synthesis quality and camera trajectory accuracy.
Takeaways & Limitations
Constraining relative poses with mono-depth leads to better pose estimation, novel view synthesis quality, and geometry reconstruction on challenging scenes.
Takeaways & Limitations
The optimisation is affected by non-linear distortions and mono-depth estimation accuracy, which the method does not consider.
Abstract
from arXiv · showhide
Training a Neural Radiance Field (NeRF) without pre-computed camera poses is challenging. Recent advances in this direction demonstrate the possibility of jointly optimising a NeRF and camera poses in forward-facing scenes. However, these methods still face difficulties during dramatic camera movement. We tackle this challenging problem by incorporating undistorted monocular depth priors. These priors are generated by correcting scale and shift parameters during training, with which we are then able to constrain the relative poses between consecutive frames. This constraint is achieved using our proposed novel loss functions. Experiments on real-world indoor and outdoor scenes show that our method can handle challenging camera trajectories and outperforms existing methods in terms of novel view rendering quality and pose estimation accuracy. Our project page is https://nope-nerf.active.vision.
1. Introduction
NeRF training normally depends on estimated camera parameters, but existing pose-optimisation methods struggle with dramatic camera motion. NoPe-NeRF integrates undistorted monocular depth to constrain relative poses and jointly optimise camera poses and a NeRF.
- NeRF training requires camera-parameter estimation, commonly obtained with COLMAP, but this preprocessing is time-consuming and non-differentiable.Without initial parameters, recent joint-optimisation methods handle only forward-facing scenes and fail under dramatic camera motion.
- Estimating each camera pose independently ignores relative poses between adjacent frames, while joint NeRF and camera optimisation adds ambiguity and can become unstable.The relevant ambiguities include shape-radiance ambiguity and the additional uncertainty introduced by jointly estimating camera parameters.
- Monocular depth supplies geometry cues, enables relative-pose constraints between adjacent frames through Chamfer Distance, and does not require camera parameters as input.It is also lightweight compared with multi-view stereo depth estimation.
- NoPe-NeRF corrects scale and shift distortions during training, transforming monocular depth into multiview-consistent maps for pose and surface-rendering losses.The two resulting constraints are an inter-frame Chamfer Distance loss and a depth-based surface rendering loss.
- The method jointly optimises camera poses and a NeRF from image sequences with large camera motion using undistorted mono-depth and inter-frame losses.A depth-based surface rendering loss further regularises relative-pose estimation.
- NoPe-NeRF handles large camera motion and outperforms state-of-the-art methods in novel-view synthesis quality and camera trajectory accuracy.
2. Related Work
NeRF is a popular photorealistic scene representation, but many enhancements still require camera parameters from Structure-from-Motion. Unposed-NeRF methods remove this preprocessing by jointly optimising camera poses and the NeRF, among other strategies.
- Novel View Synthesis: NeRF became popular for photorealistic rendering among representations used for novel view synthesis.
- Novel View Synthesis: NeRF performance has been improved with regularisation, depth priors, surface enhancements, latent codes, and accelerated training or rendering.Most of these approaches nevertheless require camera parameters pre-computed with Structure-from-Motion algorithms.
- NeRF With Pose Optimisation: SLAM-style pose-removal methods either require RGB-D inputs or depend on accurate poses generated by SLAM tracking.
- NeRF With Pose Optimisation: Unposed-NeRF directly optimises camera poses with NeRF, using strategies including joint intrinsics and extrinsics optimisation, coarse-to-fine positional encoding, camera-distortion modelling, geometric losses, and alternative network architectures.
3. Method
NoPe-NeRF jointly optimises a NeRF, camera poses, and monocular-depth distortion parameters from image sequences, using undistorted depth to constrain relative poses during challenging motion.
- Joint optimisation: The method jointly recovers NeRF parameters and camera poses from image sequences without pre-computed poses.Camera pose is represented as a rotation and translation for each frame.
- Depth undistortion: Monocular depth maps are transformed with per-frame scale and shift parameters to produce multi-view-consistent depth maps.The transformation is D*_i = α_i D_i + β_i, with the parameters jointly optimised against NeRF-rendered depth.
- Depth undistortion: The depth loss couples monocular-depth geometry with NeRF-rendered depth, reducing shape-radiance ambiguity while allowing NeRF to provide multi-view consistency.Both the monocular depth and NeRF benefit from this coupling.
- Relative pose constraints: Relative poses between consecutive frames are constrained by matching point clouds back-projected from undistorted depth maps with Chamfer Distance.The point-cloud loss optimises the relative transformation between consecutive point clouds.
- Relative pose constraints: A surface-based photometric loss complements 3D point matching by penalising appearance differences between associated pixels across frames.Associations are obtained by projecting an undistorted-depth point cloud into the two images.
4. Experiments
Experiments evaluate novel-view synthesis, pose accuracy, and depth quality across ScanNet and Tanks and Temples, including qualitative comparisons and ablations. The method consistently outperforms pose-unknown baselines and remains effective for challenging trajectories, while its performance depends on modelling depth distortions and inter-frame constraints.
- Experimental Setup: Experiments use Tanks and Temples and ScanNet to evaluate novel-view synthesis, camera poses, and depth maps with standard metrics.The evaluation includes PSNR, SSIM, LPIPS, ATE, RPE, and standard depth metrics.
- Novel View Synthesis: NoPe-NeRF outperforms pose-unknown baselines by a large margin in novel-view synthesis quality.Quantitative results are reported in Table 1, with qualitative novel-view and depth comparisons in Figure 3.
- Novel View Synthesis: Interpolated novel-view videos remain photo-realistic for NoPe-NeRF, whereas previous methods produce visible artifacts.The videos are generated by fitting learned trajectories with Bezier curves and uniformly sampling new viewpoints.
- Camera Pose: NoPe-NeRF significantly outperforms other baselines across pose metrics, with estimated trajectories better aligned to reference poses and rotation estimates two orders of magnitude more accurate.Pose trajectories and relative rotation errors are visualised in Figure 4; Tanks and Temples uses COLMAP estimates as reference poses.
- Depth: Rendered depth maps achieve superior accuracy over prior alternatives and outperform DPT after undistortion on ScanNet.The depth evaluation uses ground-truth depths and is summarised in Table 3.
- Ablation Study: Ablations show that ignoring depth distortions, removing inter-frame losses, or removing the depth loss degrades pose accuracy or causes drift.Scale and shift modelling, pairwise point-cloud and surface-based losses, and depth supervision each contribute to pose estimation.
5. Conclusion
NoPe-NeRF jointly estimates camera poses and a NeRF from image sequences, using mono-depth to address complex trajectories and improve reconstruction outcomes.
- NoPe-NeRF is an end-to-end differentiable model for joint camera pose estimation and novel view synthesis from image sequences.
- Mono-depth maps constrain relative frame poses and regularise NeRF geometry during optimisation.
- Improved pose estimation leads to better novel view synthesis quality and geometry reconstruction compared with other approaches.
Appendix A. Implementation Details
The appendix introduces the datasets, training procedure, and evaluation metrics used in the study.
- The following sections describe the datasets used by the study.
- The appendix includes details about the training procedure.
- The evaluation metrics are also described in the following sections.
A.1. Dataset
The study evaluates sequences with dramatic camera motion from ScanNet and Tanks and Temples, using specified temporal sampling and train/test splits.
- Dataset: Sequences with dramatic camera motions are selected from ScanNet and Tanks and Temples for training and evaluation.
- Dataset: Starting from the 5th image, every 8th image is sampled as a test image.
- Dataset: Maximum relative rotation angle between any two frames is reported as Max rotation.
- Dataset: The selected sequences test whether the method handles dramatic camera motion that previous methods cannot handle beyond forward-facing scenes.
A.2. Training Details
Training uses fixed ray and point sampling, scheduled learning-rate decay, and depth-scale handling to avoid scale collapse.
- Training Details: Each training image uses 1024 pixels or rays, with 128 sampled points along each ray for all approaches and baselines.
- Training Details: All approaches use the same predefined near-and-far sampling range and sample uniformly within it.
- Training Details: The NeRF learning rate decays every 10 epochs, while the camera-pose learning rate decays every 100 epochs.
- Training Details: The last depth map is manually assigned scale 1.0 to avoid scale collapse during training.
- Training Details: Normalised point clouds are used when computing the inter-frame point-cloud loss.
A.3. Test-time Optimisation
Test-time camera-pose alignment is necessary for evaluating novel views, especially on complex trajectories. The appendix compares several initialization and optimization strategies and adopts Neighbour + opt for all main results.
- All evaluated methods fail to align test poses on complex ScanNet and Tanks and Temples trajectories under their baseline alignment procedures.The comparison covers BARF, NeRFmm, and SC-NeRF.
- Neighbour + opt initializes each test pose from adjacent learned training poses before test-time photometric optimization.This procedure keeps the trained NeRF fixed during optimization and is used for the main-paper evaluations.
- Identity + opt can fall into local minima when the target pose is far from identity initialization.BARF’s initialization is effective for simple forward-facing scenes but not complex trajectories.
- Sim(3) + opt depends on precise alignment between learned poses and COLMAP poses, while incorrect pose estimates can distort the initialization.
- Sim(3) + no opt is biased toward COLMAP estimates, and misalignment can substantially reduce view-synthesis quality.This is the alignment strategy described for SC-NeRF.
A.4. Evaluation Metrics
The evaluation uses standard image-quality metrics for novel view synthesis and reports a comparison of test-time pose-alignment methods alongside LLFF-NeRF synthesis results. Depth evaluation is defined over valid pixels using estimated and ground-truth depth.
- Novel View Synthesis: PSNR, SSIM, and LPIPS measure novel view synthesis quality, with LPIPS computed using a VGG architecture.
- Pose Alignment: Table 8 compares different pose-alignment methods during test-time optimization on ScanNet 0079 00.
- Novel View Synthesis: Table 9 reports novel view synthesis results on the LLFF-NeRF dataset.
- Depth: Depth evaluation uses estimated depth d, ground-truth depth dgt, and the set V of valid pixels in a depth map.
Appendix B. Additional Results
Additional experiments report stronger pose, synthesis, and depth results across LLFF-NeRF, ScanNet, Tanks and Temples, and qualitative visualizations. The results include both quantitative comparisons and trajectory, rendering, and depth-map inspections.
- LLFF-NeRF Dataset: NoPe-NeRF performs better than NeRFmm on both pose accuracy and novel view synthesis quality on LLFF-NeRF.Both methods use normalized device coordinates.
- Depth Estimation: NoPe-NeRF’s depth estimation accuracy outperforms other baselines by a large margin on ScanNet scenes.Detailed evaluations are provided for four ScanNet scenes.
- Additional Visualizations: Additional Tanks and Temples and ScanNet visualizations show synthesized images, rendered depth maps, trajectories, and relative rotation errors.
- Quantitative Results: Tables 10–14 provide pose-accuracy and depth-map evaluations for LLFF-NeRF and multiple ScanNet scenes.
- Novel View Synthesis and Depth: The Tanks and Temples qualitative results show that NoPe-NeRF recovers details in both colour and geometry.
- Novel View Synthesis and Depth: The ScanNet qualitative results likewise show recovery of details in both colour and geometry.