PaperlayerView exact revision on arXiv

Source-linked AI summary

SPARF: Neural Radiance Fields from Sparse and Noisy Poses

Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, Federico Tombari

arXiv:2211.11738v3cs.CV

TL;DR

SPARF addresses novel-view synthesis from as few as three wide-baseline images with noisy camera poses, where standard NeRF requires dense views and accurate poses. It jointly refines poses and learns the radiance field using multi-view correspondence and depth-consistency objectives, achieving a new state of the art on three challenging datasets. Its applicability is limited to image collections with overlapping visible regions and depends on correspondence quality.

  • Problem

    Sparse wide-baseline images and noisy camera poses make NeRF overfit and leave pose-NeRF refinement under-constrained.

  • Method

    SPARF jointly trains the NeRF and camera poses using pixel correspondences for multi-view geometric consistency and pseudo-depth supervision for unseen viewpoints.

  • Results

    SPARF sets a new state of the art in the sparse-view regime on three challenging datasets.

  • Takeaways & Limitations

    The approach produces realistic novel-view renderings from few wide-baseline images with noisy camera pose estimates.

  • Takeaways & Limitations

    SPARF requires each input image to have visible regions overlapping with at least one other image, and its performance depends on the matching network’s quality.

Abstract

from arXiv · show

Neural Radiance Field (NeRF) has recently emerged as a powerful representation to synthesize photorealistic novel views. While showing impressive performance, it relies on the availability of dense input views with highly accurate camera poses, thus limiting its application in real-world scenarios. In this work, we introduce Sparse Pose Adjusting Radiance Field (SPARF), to address the challenge of novel-view synthesis given only few wide-baseline input images (as low as 3) with noisy camera poses. Our approach exploits multi-view geometry constraints in order to jointly learn the NeRF and refine the camera poses. By relying on pixel matches extracted between the input views, our multi-view correspondence objective enforces the optimized scene and camera poses to converge to a global and geometrically accurate solution. Our depth consistency loss further encourages the reconstructed scene to be consistent from any viewpoint. Our approach sets a new state of the art in the sparse-view regime on multiple challenging datasets.

1. Introduction

NeRF performs high-fidelity novel-view synthesis with dense views and accurate poses, but sparse wide-baseline images and noisy poses cause overfitting, degenerate geometry, and unreliable pose estimation. SPARF addresses this by jointly refining poses and learning the NeRF through multi-view geometric constraints and depth consistency.

  • NeRF’s high-fidelity view synthesis depends on dense input views and highly accurate camera poses.These requirements limit its usability in real-world settings such as AR/VR and autonomous driving.
  • Sparse inputs cause NeRF to overfit training images, producing inconsistent or degenerate reconstructions even with perfect poses.The resulting geometry can collapse toward the provided views and yield poor novel-view renderings.
  • COLMAP pose estimation degrades as the number of views decreases or image baselines increase, making accurate poses difficult for few wide-baseline images.
  • Sparse-view NeRF methods often assume perfect camera poses, although accurate pose estimation for few wide-baseline images is challenging.
  • Per-image photometric or silhouette signals are too weak in under-constrained sparse scenes to enforce a global geometrically accurate solution.The introduction identifies explicit relations between training images and scene geometry as crucial.
  • SPARF jointly trains the NeRF and camera poses using multi-view correspondences and a depth consistency loss, without assuming a scene or object-shape prior.Correspondences constrain pose and geometry across views, while depth supervision promotes consistency from unseen directions.

2. Related Work

Related work addresses sparse-view rendering through learned priors or per-scene regularization, and addresses pose refinement through joint NeRF-camera optimization. These approaches respectively face out-of-distribution generalization challenges or rely mainly on signals that are insufficient for sparse geometric constraints.

  • Sparse input novel-view rendering: Conditional radiance-field models use large-scale posed multi-view datasets to improve few-shot novel-view rendering.Their generalization to out-of-distribution novel views remains challenging.
  • Sparse input novel-view rendering: Per-scene methods regularize appearance or geometry to reduce NeRF overfitting with few input images.Examples include semantic embedding, ray entropy, distortion, depth smoothness, and normalizing-flow objectives.
  • Joint NeRF and pose refinement: BARF and NeRF-- jointly optimize radiance fields and noisy camera parameters using photometric loss as their only training signal.
  • Joint NeRF and pose refinement: Other pose-refinement methods modify activations or training schedules to ease optimization of noisy camera poses.SiNeRF and GARF use different activation functions, while GNeRF uses sequential training with a rough initial pose network.

3. Preliminaries

The preliminaries define camera transforms and projection operations, represent scenes with a NeRF, and describe volume rendering and photometric training. They establish the components later used for joint pose-NeRF refinement.

  • Camera pose and projection: A camera-to-world pose P_i consists of rotation and translation in SE(3), while K denotes the intrinsic matrix.
  • Camera pose and projection: The projection operator maps a camera-frame 3D point to a pixel, while backprojection maps a pixel and depth to a camera-frame 3D point.
  • Scene representation: A neural radiance field maps a 3D location and unit-norm viewing direction to RGB color and volume density.SPARF adopts the NeRF framework for scene representation and image formation.
  • Rendering: NeRF uses an MLP with positional encoding, and volume rendering samples points along each camera ray to predict pixel colors and depths.The framework trains coarse and fine MLPs, with the coarse network guiding sampling for the fine network.
  • Photometric training: Prior joint pose-NeRF approaches optimize the radiance field and camera pose estimates with a photometric loss over the training images.This strategy works well with dense views but fails in the sparse regime.

4. Method

SPARF jointly learns a NeRF and refines noisy camera poses for sparse wide-baseline views using correspondence-based multi-view geometry and depth consistency. The first objective links training images geometrically, while the second supplies pseudo-depth supervision for unseen viewpoints within a staged training framework.

  • Overview: SPARF targets as few as three sparse, wide-baseline input views with noisy camera pose estimates.Its setting assumes large viewpoint variations and no prior on scene or object shape.
  • Improving Geometry at Unobserved Views: Depth consistency complements correspondence supervision: correspondences learn accurate geometry, while pseudo-depth supervision encourages consistency from novel viewpoints.
  • Multi-View Correspondence Loss: The multi-view correspondence objective uses matched pixels, rendered depths, and current poses to minimize reprojection error across training views.Backprojected pixels should correspond to the same world-space 3D point, enforcing globally consistent geometry and poses.
  • Multi-View Correspondence Loss: Dense correspondence confidence weights select matches with w_p ≥ 0.95, while the Huber loss handles reprojection residuals.The dense map also implicitly imposes a smoothness prior on rendered depth.
  • Improving Geometry at Unobserved Views: The depth consistency loss projects training-view 3D points into an unseen view and compares their projected depth with the NeRF-rendered depth.A visibility mask handles occlusion and out-of-view projections.
  • Training strategy: The final objective combines photometric, correspondence, and depth-consistency losses, with joint pose training followed by pose freezing and coarse-and-fine NeRF training.The staged procedure is designed to obtain sharp geometry after pose exploration.

5. Experimental Results

SPARF is evaluated for three-view novel-view rendering on DTU, LLFF, and Replica, including noisy or identity pose initialization. Analyses show that coarse-to-fine positional encoding and multi-view geometry objectives improve reconstruction and pose refinement.

  • Experimental Settings: The evaluation uses DTU, LLFF, and Replica with three input views, reporting pose registration errors and PSNR, SSIM, and LPIPS for view synthesis.DTU uses wide-baseline object scenes; LLFF uses forward-facing scenes; Replica uses room-scale indoor videos.
  • Method Analysis: Coarse-to-fine positional encoding gives the best DTU results among the tested strategies, while standard high-frequency encoding produces degenerate geometry and novel views.The comparison uses three views and fixed ground-truth poses.
  • Method Analysis: Per-image depth supervision can yield locally accurate geometry while leaving image-specific NeRF and pose solutions disconnected; 3D-point supervision enforces a global geometric solution.The comparison uses ground-truth depth in sparse-view pose-NeRF training.
  • Method Analysis: Adding the multi-view correspondence loss produces drastically better performance across metrics, and depth consistency provides a further improvement to the best overall result.This ablation uses DTU with three views, fixed ground-truth poses, and coarse-to-fine positional encoding.
  • Method Analysis: Photometric and mask or silhouette objectives fail to register poses on DTU, whereas SPARF’s correspondence-based objective supports joint pose-NeRF refinement from noisy initialization.The DTU setup uses three views with initial noisy poses, including 15° average rotation and 70 translation error.
  • Comparison to SOTA with Noisy Poses: SPARF outperforms prior methods in sparse-view LLFF evaluation and achieves the best performance across both DTU and LLFF metrics with fixed ground-truth poses.BARF struggles with sparse LLFF views, while SPARF improves novel-view rendering using multi-view correspondence and depth consistency objectives.

6. Conclusion

SPARF jointly refines camera poses and trains a NeRF from few wide-baseline images with noisy pose estimates. Its multi-view geometry objectives establish state-of-the-art sparse-view results, subject to correspondence and overlap requirements.

  • 6. Conclusion: SPARF jointly trains a NeRF and refines camera poses from as few as three wide-baseline images with noisy pose estimates.The method does not assume a prior on scene or object shape.
  • 6. Conclusion: The approach integrates two objectives inspired by multi-view geometry and achieves state-of-the-art performance on three challenging datasets.The conclusion reports this outcome for sparse-view novel-view rendering.
  • Limitations and future work: SPARF applies only when each input image has visible regions overlapping with at least one other image.This is an explicit scope condition in the limitations discussion.
  • Limitations and future work: Performance depends on the quality of the matching network, motivating correspondence filtering and per-scene online refinement as future directions.The authors also identify joint refinement of camera intrinsics and distortion parameters as future work.

A.3. Training details

SPARF uses staged joint pose-NeRF training with correspondence and depth-consistency losses, coarse-to-fine positional encoding, and dataset-specific optimization schedules. Training choices balance pose optimization, geometry accuracy, and robustness to correspondence errors.

  • Staged training: Joint pose-NeRF training uses two stages: poses and the coarse MLP are trained first, then poses are frozen while coarse and fine MLPs are trained.The first stage occupies 30% of total training iterations.
  • Training procedure: At each iteration, the method combines photometric supervision, sampled multi-view correspondences, and depth consistency from a selected neighboring training view.Matches are computed between all view pairs before training begins.
  • Coarse-to-fine positional encoding: Coarse-to-fine positional encoding is activated from 40% to 70% of training during joint refinement and from 10% to 50% when poses are fixed.The joint-refinement schedule leaves positional encodings inactive during the earliest 40% of training.
  • Training schedules: With three input views, joint refinement trains for 100K iterations on all datasets, while fixed-pose training uses 50K iterations on DTU and Replica and 70K on LLFF.Using 100K iterations with fixed ground-truth poses can produce similar or worse results because of overfitting.
  • Loss weighting: The correspondence and depth-consistency loss weights are both set to λc = 10^-3 and λd = 10^-3 in the standard configuration.On DTU with fixed ground-truth poses, the authors instead use λc = 10^-4 and λd = 10^-3.
  • Loss weighting: Correspondence loss is emphasized early for pose optimization, then halved every 10K iterations after pose freezing so photometric supervision can correct geometry errors from unreliable matches.The depth-consistency loss is intended to contribute less than correspondence loss because it enforces viewpoint consistency after geometry is learned.

B.2. Metrics

SPARF evaluates pose registration, novel-view synthesis, and depth reconstruction using alignment-aware metrics. Additional analyses compare pose estimation and rendering under varying input views and initialization noise.

  • Pose registration: Optimized and ground-truth pose trajectories are aligned with a RANSAC-inspired Sim(3) procedure because few-view Umeyama alignment is unstable.The selected transformation minimizes average camera alignment error across sampled camera pairs.
  • Pose registration: Rotation error is the absolute angle between ground-truth and estimated rotations, while translation error is their Euclidean positional distance.Translation error has no units because pose positions are not metric on the evaluated datasets.
  • Novel-view rendering: Novel-view quality is measured with mean PSNR, SSIM, and AlexNet LPIPS after test-time photometric pose optimization.This optimization keeps the NeRF fixed and reduces the influence of camera misalignment on view-synthesis evaluation.
  • Depth evaluation: Depth quality is evaluated as the average absolute difference between aligned predicted and ground-truth depths over valid ground-truth regions.Predicted depth is first rescaled using the alignment scale.
  • Additional analyses: COLMAP can fail even with > 20 input views, while sparse-view pose estimates become noisy and SPARF produces lower final rotation and translation errors.Replacing standard matches with SuperPoint-SuperGlue improves COLMAP, but few-view estimation remains challenging.
  • Additional analyses: Figure 8 compares BARF and SPARF using pose-registration errors and masked-background PSNR across initial-noise levels with 3 DTU views.Rotation errors are reported in degrees and translation errors are multiplied by 100.

C.2. Robustness to pose initialization and failure cases

SPARF is tested under increasingly noisy pose and correspondence initialization, and its failure modes are tied to unreliable image matches. Dense matching is more robust than sparse matching on repetitive or homogeneous surfaces.

  • Robustness to pose initialization: SPARF is evaluated with synthetically perturbed rotations, translations, or both, using only its multi-view correspondence objective.The experiment uses a DTU scene with 3 input views and omits depth consistency and staged training.
  • Failure cases: SPARF likely fails when too few or inaccurate correspondences are extracted between input views.Its joint pose-NeRF optimization depends directly on the predicted matches.
  • Failure cases: Reliable correspondences are particularly difficult for nearly symmetric objects and scenes with many homogeneous surfaces.The DTU pumpkin example has both properties and yields no reliable matches from PDC-Net or SuperPoint-SuperGlue.
  • Correspondence predictors: Dense matching is more robust than sparse matching on repetitive and homogeneous surfaces, producing better DTU pose-refinement results in that setting.Sparse SuperPoint-SuperGlue matching struggles with repeatable keypoints, whereas PDC-Net can exploit smoothness in homogeneous regions.
  • Correspondence predictors: PDC-Net and SuperPoint-SuperGlue achieve similar performance on LLFF despite their different matching strategies.The comparison evaluates the correspondence predictors within the same joint pose-NeRF framework.
  • Robustness to pose initialization: SPARF remains robust to correspondence noise with standard deviation up to 6 pixels, but performance drops for highly erroneous matches.The test uses 3 DTU views with noisy poses and ground-truth matches corrupted by Gaussian noise.

C.4. Additional ablation study

Ablations show that correspondence supervision, staged training, and depth consistency each contribute to joint pose-NeRF refinement. The study also tests correspondence predictors, visibility masking, and broader initialization settings.

  • Joint pose-NeRF refinement: Staged training improves PSNR and SSIM by allowing the fine network to learn sharper geometry after camera poses are registered and frozen.Jointly training poses with coarse and fine networks can produce blurry surfaces during pose-space exploration.
  • Joint pose-NeRF refinement: Adding depth consistency slightly improves rendering performance, producing the best overall ablation results.The loss encourages scene geometry to remain consistent across viewpoints.
  • Correspondence predictors: PDC-Net and SuperPoint-SuperGlue are compared as alternative correspondence predictors for SPARF* on DTU and LLFF.The tables report joint pose-NeRF performance under each matching network.
  • Correspondence robustness: SPARF* evaluates photometric and correspondence losses without depth consistency or staged training under noisy-match perturbations.Figure 10 varies Gaussian noise added to ground-truth image matches on one DTU scene.
  • Depth consistency: The visibility mask is ablated for depth consistency on DTU with 3 views and fixed ground-truth poses.The mask excludes pixels whose virtual-view projections are occluded by the reconstructed scene.
  • Additional settings: Additional experiments vary initialization schemes, numbers of input views, and whether all 25 available training views are used.These settings extend evaluation beyond the main 3-view experiments.

D.1. Results on the DTU dataset

On DTU, SPARF refines noisy poses and improves novel-view rendering across initialization schemes and view counts. It also generalizes to all 25 views, although inconsistent illumination slightly reduces rendering metrics.

  • DTU results: SPARF jointly refines COLMAP-initialized poses and trains the NeRF on 14 DTU scenes, excluding scan30 where initialization fails.The evaluation compares fixed initial poses, refined poses, and fixed ground-truth poses.
  • Initialization schemes: PDC-Net gives initial rotation and translation errors of 0.75° and 3.87, compared with 1.34° and 6.84 for SuperPoint-SuperGlue.PDC-Net performs better on homogeneous regions because dense matching can rely on smoothness properties.
  • DTU results: SPARF narrows the rendering gap between fixed COLMAP poses and fixed ground-truth poses by jointly refining poses and the NeRF.Fixed incorrect poses cause artifacts, while refinement yields similar LPIPS and depth error to the fixed-ground-truth case.
  • Input-view count: With 3, 6, or 9 input views, SPARF outperforms all previous works on the evaluated DTU tasks.Excluding scan30 yields optimized rotation and translation errors below 1° and 1, respectively, with translation multiplied by 100.
  • Qualitative results: SPARF produces realistic novel views with accurate geometry from many viewing directions using only 3 noisy input views.The qualitative examples cover a large variety of scenes.
  • All-view evaluation: With all 25 DTU views, SPARF significantly outperforms BARF in pose registration and novel-view synthesis.PSNR and SSIM are slightly worse than with 9 views because some images have inconsistent illumination.

D.2. Results on the LLFF dataset

On LLFF, SPARF improves sparse-view novel-view synthesis from as few as 3 input views with identity poses, while its advantage diminishes when more views sufficiently constrain the scene.

  • With 2 or 3 views, BARF struggles to refine poses, whereas SPARF improves pose registration and novel-view synthesis.
  • Adding depth regularization helps RegBARF with 2 or 3 views but harms performance with 6 or 9 views.
  • SPARF produces realistic renderings and geometrically accurate scenes from 3 input views initialized with identity poses.
  • With all available LLFF training views, BARF and SPARF reach similar performance in the many-view regime.

D.3. Results on the Replica dataset

On Replica, SPARF improves novel-view renderings and scene geometry from three views under both COLMAP-based and synthetically noisy pose initialization.

  • From COLMAP-PDCNet poses with 0.39° rotation and 3.01 translation error, SPARF refines the errors to 0.15° and 0.76.
  • Some Replica scenes have initial translation errors up to 8, making pose refinement particularly beneficial for rendering quality.
  • With COLMAP-SP-SG initialization, SPARF achieves the best results and matches its perfect-pose accuracy after pose refinement.
  • SPARF produces the best Replica renderings, with fewer floaters and blurry surfaces and sharper, more accurate depth geometry.

E. Additional Results with Fixed GT Poses

With fixed ground-truth poses, SPARF remains highly competitive across DTU and LLFF view counts, outperforming standard NeRF-based alternatives especially in sparse settings.

  • On DTU, SPARF sets a new state of the art on all metrics for 3, 6, and 9 input views, except whole-image PSNR with 3 views.
  • Conditional models trained on DTU face LLFF as an out-of-distribution scenario, while SPARF outperforms the best conditional model with 3 views.
  • On LLFF with 3 input views, SPARF outperforms all other methods on every metric.
  • For 6 and 9 LLFF views, SPARF is slightly below MipNeRF and RegNeRF but substantially exceeds standard NeRF and DS-NeRF.
Loading 2211.11738v3…