PaperlayerView exact revision on arXiv

Source-linked AI summary

Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields

Dominic Maggio, Marcus Abate, Jingnan Shi, Courtney Mario, Luca Carlone

arXiv:2209.09050v1cs.RO

TL;DR

NeRF-based localization methods have required a good initial pose guess and substantial computation. Loc-NeRF combines a NeRF map with Monte Carlo localization, and reports better position and rotation accuracy than iNeRF and NeRF-Navigation, including real-time localization on a real-world robot.

  • Problem

    Existing NeRF-based localization methods require a good initial pose guess and incur high computational overhead, while classical visual localization is sensitive to keypoint-match quality and uses image-database maps.

    Source: §I · p. 1, §II · p. 2

  • Method

    Loc-NeRF uses a particle filter with a trained NeRF map for image-based updates and motion estimates for prediction.

    Source: §I · p. 1, §IV · p. 3, B. Update Step · p. 3

  • Results

    Loc-NeRF achieves higher position and rotation accuracy than iNeRF and lower average position and rotation errors than NeRF-Navigation, and demonstrates real-time localization on a real-world robot.

    Source: A. Single-image Pose Estimation:… · p. 4, B. Pose Tracking: Comparison with… · p. 5, §VI · p. 6

  • Takeaways & Limitations

    Loc-NeRF performs global localization without relying on an accurate initial pose guess and has demonstrated real-time localization on a real-world robotic platform.

    Source: §I · p. 2, §VI · p. 6

Abstract

from arXiv · show

We present Loc-NeRF, a real-time vision-based robot localization approach that combines Monte Carlo localization and Neural Radiance Fields (NeRF). Our system uses a pre-trained NeRF model as the map of an environment and can localize itself in real-time using an RGB camera as the only exteroceptive sensor onboard the robot. While neural radiance fields have seen significant applications for visual rendering in computer vision and graphics, they have found limited use in robotics. Existing approaches for NeRF-based localization require both a good initial pose guess and significant computation, making them impractical for real-time robotics applications. By using Monte Carlo localization as a workhorse to estimate poses using a NeRF map model, Loc-NeRF is able to perform localization faster than the state of the art and without relying on an initial pose estimate. In addition to testing on synthetic data, we also run our system using real data collected by a Clearpath Jackal UGV and demonstrate for the first time the ability to perform real-time global localization with neural radiance fields. We make our code publicly available at https://github.com/MIT-SPARK/Loc-NeRF.

I. INTRODUCTION

Vision-based localization is important across robotics and computer vision, but classical keypoint-based methods depend on matching quality and image-database maps. Loc-NeRF uses a NeRF map with Monte Carlo localization to estimate poses with poor or no initial guesses, including real-time tracking on real-world data.

I. INTRODUCTION

  • Classical camera-pose methods detect and match keypoints before estimating 6DoF poses, making them sensitive to match quality and dependent on image-database maps.The cited applications of vision-based localization include self-driving vehicles and robot manipulation.Source: §I · p. 1
  • Earlier NeRF-localization work includes iNeRF pose optimization and NeRF-Navigation across estimation and planning, while Loc-NeRF applies Monte Carlo localization to extract poses from a NeRF.The section describes the NeRF-localization literature as sparse.Source: §I · p. 1

Contributions

  • Loc-NeRF combines a NeRF map with particle-filter localization, using NeRF in the update step and visual-inertial odometry or robot dynamics for motion prediction.The approach estimates 6DoF poses and can adjust computational effort by changing the number of particles.Source: §I · pp. 1–2
  • Experiments show single-image pose estimation without an accurate initial guess, global localization, and real-time tracking with real-world data.The real-time experiment uses a Clearpath Jackal UGV; the figure shows NeRF renderings from the beginning, middle, and end of the experiment.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

Prior scene representations and localization methods span traditional geometric models, neural implicit representations, and estimation techniques. In RGB-only localization with neural implicit representations, existing approaches iNeRF and NeRF-Navigation require a good initial guess and incur high computational overhead.

II. RELATED WORK

  • Traditional point clouds lack surface information, meshes are challenging to estimate accurately from robot sensor data, and voxel-model accuracy is limited by voxel size.Classical localization also uses sparse or dense scene representations and estimation methods such as particle filters and Kalman filters.Source: §II · p. 2

Neural Implicit Shape Representations

  • Neural implicit representations include signed-distance and occupancy models, while NeRF uses view directions to synthesize novel, photorealistic views.Neural implicit methods have also been explored for localization and mapping, including iMAP, NICE-SLAM, and iSDF.Source: §II · p. 2
  • Methods for training NeRF from unknown or low-accuracy image poses can take several hours or over a day and target NeRF construction rather than real-time pose estimation with a trained NeRF.The section also describes work on large-scale and unbounded NeRF scenes, and computational enhancements to training and rendering.Source: §II · p. 2

Visual Localization

  • For RGB-only robot localization, iNeRF and NeRF-Navigation require a good initial pose guess and incur high computation overhead; Loc-NeRF uses a particle-filter backbone to relax that reliance.NeRF-Navigation's process-loss term supports tracking across multiple images, while the section notes that depth is used by iMAP, NICE-SLAM, and iSDF.Source: §II · p. 2

III. NERF PRELIMINARIES

NeRF represents a scene with a multilayer perceptron trained on posed RGB images. At inference, it estimates density and color along camera rays and uses volume rendering to predict pixel color.

III. NERF PRELIMINARIES

  • NeRF is trained from RGB images with known poses and a known camera model, and its MLP predicts density and RGB color from 3D position and viewing direction.At inference, the method renders novel views from the trained scene representation.Source: §III · p. 2
  • For each pixel, NeRF projects a camera ray, samples points along it, and uses volume rendering to estimate the pixel color C(r).Coarse samples are generated uniformly, with fine samples selected based on the estimated density of the coarse samples.Source: §III · pp. 2–3
  • The sampled depth z along ray r is bounded by z_near and z_far in the rendering description.The paper refers readers to [10] for a more detailed description.Source: §III · p. 3

IV. LOC-NERF: MONTE CARLO LOCALIZATION USING NEURAL RADIANCE FIELDS

Loc-NeRF estimates a robot's 6DoF pose from a trained NeRF map, RGB images, and motion estimates. It uses a particle filter to represent the pose posterior and updates its particles through prediction, update, and resampling.

IV. LOC-NERF: MONTE CARLO LOCALIZATION USING NEURAL RADIANCE FIELDS

  • Given a trained NeRF map, RGB images, and motion estimates, Loc-NeRF estimates the robot's 6DoF pose over time.The filter estimates the posterior conditioned on the map, images collected so far, and motion measurements collected so far.Source: §IV · p. 3
  • Loc-NeRF represents the pose posterior as a weighted set of particles, each carrying a pose and a corresponding weight.The implementation represents each particle's 3D pose as a 4×4 transformation matrix.Source: §IV · p. 3
  • At each time step, the particle filter updates its particle set through prediction, update, and resampling.The update follows receipt of new images and odometry measurements.Source: §IV · p. 3

A. Prediction Step

Loc-NeRF predicts particle poses from robot motion and updates their weights using how well NeRF-rendered views match the camera image. The update approximates image likelihood from randomly sampled pixels.

A. Prediction Step

  • The prediction step applies measured robot motion to each particle, then adds Gaussian noise to represent odometry error.The motion measurement can come from odometry or integrated robot dynamics.Source: A. Prediction Step · p. 3

B. Update Step

  • The update step uses a heuristic measurement likelihood to weight each particle according to its pose in the NeRF map.The likelihood models the chance of observing the camera image from a particle’s pose.Source: B. Update Step · p. 3
  • The weight update compares observed image intensities with NeRF-predicted intensities at randomly sampled pixels, assigning lower weights to mismatching particles.Weights are normalized to sum to 1.Source: B. Update Step · p. 3

C. Resampling Step

Loc-NeRF resamples particles according to their weights, adjusts prediction noise and particle count as the particle spread narrows, and returns a single pose from the weighted particle poses.

C. Resampling Step

  • Resampling draws particles with replacement according to their weights, retaining more likely pose hypotheses and discarding less likely ones.Source: C. Resampling Step · p. 3

D. Computational Enhancements and Pose Estimate

V. EXPERIMENTS

The experiments evaluate single-image pose estimation, pose estimation over time, and real-time pose tracking with a robot. The first two use benchmarks, while the third uses collected real-world data.

V. EXPERIMENTS

  • Single-image pose estimation on LLFF data is benchmarked against iNeRF, with either a poor initial guess or no initial guess.Source: §V · p. 4
  • Pose estimation over time uses synthetic Blender data and is benchmarked against NeRF-Navigation.Source: §V · p. 4
  • A full-system demonstration evaluates real-time pose tracking using data collected by a Clearpath Jackal UGV.Source: §V · p. 4

A. Single-image Pose Estimation: Comparison with iNeRF

In single-image tests, Loc-NeRF achieves higher position and rotation accuracy than iNeRF when performance is compared against NeRF forward passes. The section also evaluates global localization and the effect of annealing.

Setup

  • The iNeRF comparison tests 20 images across four LLFF scenes using shared pre-trained NeRF-Pytorch weights and a perturbed initial pose for iNeRF.Loc-NeRF starts with 300 particles, anneals to 100, and uses 64 randomly sampled pixels per particle.Source: A. Single-image Pose Estimation:… · p. 4

Results

  • Loc-NeRF achieves higher position and rotation accuracy than iNeRF when performance is plotted against NeRF forward passes.The comparison accounts for different computational costs per iteration; iNeRF’s average position error can increase over time as estimates diverge or reach local minima.Source: A. Single-image Pose Estimation:… · p. 4
  • Annealing most benefits position accuracy and speeds updates by reducing the particle count; on an RTX 5000 laptop, Loc-NeRF’s update rate rises from 0.6 Hz to 1.8 Hz.The reported rates correspond to 300 particles initially and 100 during annealing.Source: A. Single-image Pose Estimation:… · p. 4

Global localization

  • In global localization, Loc-NeRF converges to an accurate pose estimate, while iNeRF produces no valid result in that setting.The test randomizes position offsets within ±1 m per axis and yaw across [−180°, +180°], while initializing roll and pitch from ground truth.Source: A. Single-image Pose Estimation:… · p. 4

B. Pose Tracking: Comparison with NeRF-Navigation

In simulated drone trials, Loc-NeRF achieved lower average position and rotation errors than NeRF-Navigation and recovered from inaccurate pose estimates. NeRF-Navigation was more robust when NeRF rendering quality was lower, and its starting pose was assumed to be known.

B. Pose Tracking: Comparison with NeRF-Navigation

  • Both methods used the same simulated trajectory, sensor images, motion estimate, and forward-pass budget per image.Loc-NeRF used 200 particles with 64 sampled pixels and 24 update steps per image; NeRF-Navigation used 300 iterations with 1,024 pixels.Source: B. Pose Tracking: Comparison with… · p. 4
  • Across 18 simulated drone trials, Loc-NeRF achieved lower average position and rotation errors than NeRF-Navigation and recovered from inaccurate pose estimates.The comparison used the same number of forward passes per camera image.Source: B. Pose Tracking: Comparison with… · pp. 4–5
  • NeRF-Navigation’s process loss added robustness on trajectory segments where NeRF rendering quality was lower.Its starting pose was assumed to be well known, although Loc-NeRF does not require that assumption.Source: B. Pose Tracking: Comparison with… · p. 5

C. Full System Demonstration

The full system was run on a Clearpath Jackal using a camera and VINS-Fusion for motion prediction. Loc-NeRF’s update rate accelerated during annealing, and rendered views from estimated poses were compared with camera images.

C. Full System Demonstration

  • Loc-NeRF ran on a Clearpath Jackal with a RealSense D455 camera, while VINS-Fusion supplied the prediction step.The NeRF was pretrained using metric-scaled poses and images collected with a RealSense D455 camera carried by a person.Source: C. Full System Demonstration · p. 5
  • The prediction step ran at 15 Hz, while the update step accelerated from approximately 0.9 Hz with 400 particles to 2.5 Hz with 150 particles.The system used 32 sampled pixels per particle and annealing began after about six update steps.Source: C. Full System Demonstration · p. 6
  • Selected trajectory frames compare true camera images with NeRF-rendered images generated from Loc-NeRF’s pose estimates.The comparisons show update steps 20, 40, 60, and 100, at 13, 20, 28, and 44 seconds.Source: C. Full System Demonstration · p. 6, Figure 6 · p. 6

VI. CONCLUSION

The conclusion presents Loc-NeRF as Monte Carlo localization using a NeRF map, with support for poor initial guesses, global localization, and real-time localization on a real robot. Scaling to larger environments and reducing computation remain future-work directions.

VI. CONCLUSION

  • Loc-NeRF combines Monte Carlo localization with a NeRF map and uses existing techniques, such as visual-inertial navigation or robot dynamics, for prediction.Source: §VI · p. 6
  • The paper reports localization from poor initial guesses, global localization, and real-time operation on a real-world robotic platform.Source: §VI · p. 6
  • Future work includes scaling localization to larger environments and reducing computation through faster NeRF rendering.The conclusion also mentions adapting the number of particles and using bigger NeRF models.Source: §VI · p. 6
Loading 2209.09050v1…