PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Only Robot Navigation in a Neural Radiance World

Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, Mac Schwager

arXiv:2110.00168v2cs.RO

TL;DR

The paper addresses vision-only robot navigation in environments represented by pre-trained NeRFs, using an onboard RGB camera for localization. It combines differential-flatness trajectory optimization with an optimization-based pose filter in an online replanning loop, and demonstrates navigation by quadrotors and an omnidirectional ground robot in simulated scenes. The main execution limitation is computational speed, which the authors identify as depending on NeRF performance.

  • Problem

    The paper studies how a mobile robot with only a monocular camera can plan and track a collision-free path through an environment represented by a NeRF.

  • Method

    The method combines NeRF-density collision-aware trajectory optimization, an optimization-based dynamic-state filter, and receding-horizon online replanning.

  • Results

    The pipeline demonstrates robot navigation in simulated playground, church, and Stonehenge environments, including an omnidirectional ground robot turning through a narrow gap.

  • Takeaways & Limitations

    NeRFs can support both collision-aware planning and camera-based localization within a feedback navigation pipeline, including under mismatch with ground-truth mesh environments.

  • Takeaways & Limitations

    Execution speed is the limiting factor for the proposed method, motivating future use of faster NeRF representations.

Abstract

from arXiv · show

Neural Radiance Fields (NeRFs) have recently emerged as a powerful paradigm for the representation of natural, complex 3D scenes. NeRFs represent continuous volumetric density and RGB values in a neural network, and generate photo-realistic images from unseen camera viewpoints through ray tracing. We propose an algorithm for navigating a robot through a 3D environment represented as a NeRF using only an on-board RGB camera for localization. We assume the NeRF for the scene has been pre-trained offline, and the robot's objective is to navigate through unoccupied space in the NeRF to reach a goal pose. We introduce a trajectory optimization algorithm that avoids collisions with high-density regions in the NeRF based on a discrete time version of differential flatness that is amenable to constraining the robot's full pose and control inputs. We also introduce an optimization based filtering method to estimate 6DoF pose and velocities for the robot in the NeRF given only an onboard RGB camera. We combine the trajectory planner with the pose filter in an online replanning loop to give a vision-based robot navigation pipeline. We present simulation results with a quadrotor robot navigating through a jungle gym environment, the inside of a church, and Stonehenge using only an RGB camera. We also demonstrate an omnidirectional ground robot navigating through the church, requiring it to reorient to fit through the narrow gap. Videos of this work can be found at https://mikh3x4.github.io/nerf-navigation/ .

I. INTRODUCTION

The paper presents a vision-only navigation pipeline that uses a pre-trained NeRF for collision-aware planning and camera-based localization. It combines dynamically feasible trajectory optimization, state filtering, and online replanning, demonstrating navigation across several simulated environments and robot types.

  • Motivation and approach: The proposed pipeline uses a pre-trained NeRF to represent the environment for both trajectory planning and robot localization.NeRF density supports collision-aware planning, while rendered images support camera-based state estimation.
  • Contributions: The trajectory planner uses differential flatness to generate full, dynamically feasible trajectories that avoid collisions with the NeRF environment.
  • Contributions: The optimization-based filter estimates the robot’s full dynamic state by balancing dynamics prediction and NeRF-based photometric losses.
  • Contributions: An online replanning controller combines planning and filtering in feedback so robots can navigate using only an RGB camera.
  • Evaluation: The system is evaluated in high-fidelity simulations involving a playground, a church, and Stonehenge, with tests on quadrotor and ground robots.Evaluation uses ground-truth mesh models rather than trained NeRF models, demonstrating robustness to model mismatch.

II. RELATED WORK

Related work establishes neural implicit representations and prior NeRF-based localization, while positioning this paper’s estimator as recursive Bayesian state estimation with dynamics and uncertainty propagation.

  • A. Neural implicit representations: Neural implicit representations use neural networks to map query points to geometric or appearance quantities such as density, color, or texture.
  • B. NeRF-based localization: Prior work studied mapping and online NeRF construction, while single-image methods estimated camera pose using a pre-trained NeRF.
  • B. NeRF-based localization: This paper extends maximum likelihood pose estimation into recursive Bayesian estimation that incorporates system dynamics and propagates uncertainty across timesteps.

B. Trajectory optimization

The paper formulates collision-aware trajectory optimization directly over NeRF density, using a differentiable collision cost within a receding-horizon navigation loop.

  • B. Trajectory optimization: Collision avoidance in trajectory optimization is addressed for environments represented by NeRFs, where prior work largely focused on other geometric representations.
  • Receding-horizon pipeline: At each timestep, the planner optimizes from the current mean state estimate, executes the first control action, incorporates a camera image, and replans.
  • Collision representation: Unlike SDF-based methods, the approach uses NeRF density as a proxy for occupancy and checks collisions at a finite collection of robot-body points.The proxy assumes that terminating a mass particle is analogous to terminating a light ray.
  • Collision representation: The NeRF-based collision probability is included as a cost to minimize during trajectory optimization, rather than imposed as a chance constraint.The formulation uses a Boole’s-inequality bound over body-fixed points.

IV. TRAJECTORY PLANNING IN A NERF

The planner optimizes dynamically feasible flat-output trajectories that minimize collision risk in NeRF density fields and control effort. Unlike conventional convex planners, it directly optimizes dense waypoints while representing obstacles implicitly through NeRF density.

  • IV. TRAJECTORY PLANNING IN A NERF: NeRF density represents obstacles implicitly, so the planner minimizes integrated collision risk rather than imposing hard occupancy constraints.A NeRF query provides density rather than absolute occupancy, motivating a soft collision objective.
  • IV. TRAJECTORY PLANNING IN A NERF: Differential flatness represents robot states and inputs through flat outputs and their derivatives, enabling efficient trajectory optimization.For quadrotors, position and yaw are the flat outputs.
  • IV. TRAJECTORY PLANNING IN A NERF: Traditional flatness-based planners use polynomial trajectories between fixed waypoints, but obstacle-passing decisions must be hard-coded to preserve convexity.The proposed method instead optimizes waypoint locations directly in a fundamentally nonconvex problem.
  • IV. TRAJECTORY PLANNING IN A NERF: The trajectory cost combines a NeRF-based collision penalty with a control penalty, with rotations, traveled distances, and controls derived from waypoint dynamics.The collision term accounts for the robot’s body point cloud and the density encountered along the trajectory.
  • IV. TRAJECTORY PLANNING IN A NERF: Optimization begins from heuristic waypoints and uses first-order gradients to move the trajectory toward lower-density regions while balancing control effort.Initial waypoints can come from a straight line or A∗ on a coarse grid.

C. Initialization

Initialization uses a heuristic path between the current and goal poses, while the pose-filtering problem updates the robot’s belief after each executed action and image measurement.

  • C. Initialization: The planner initializes waypoint poses with a straight line or A∗ path on a coarse grid overlaid on the NeRF scene.These preliminary poses are subsequently refined by gradient-based optimization.
  • C. Initialization: After executing an action, the robot updates its pose belief using an onboard measurement and its most recent control action.This closes the loop between trajectory execution and state estimation.
  • C. Initialization: The filter adds a dynamics process loss to NeRF photometric alignment, estimating pose derivatives and improving robustness in low-gradient image regions.It also produces a state covariance for downstream robotics algorithms.

A. Optimization formulation

The pose filter combines dynamics prediction with NeRF-based photometric evidence to update the robot’s state estimate and posterior uncertainty from onboard images and actions.

  • A. Optimization formulation: Each timestep supplies a new image, previous action, prior state mean, and prior covariance to the estimator.The dynamics model is xt = f(xt−1, ut) with process noise covariance Qt.
  • A. Optimization formulation: The estimator selects feature-rich pixels and minimizes a weighted objective combining process consistency with photometric error between rendered and observed images.Feature detectors bias sampling toward higher-gradient image regions, while St denotes measurement noise covariance.
  • A. Optimization formulation: Minimizing the objective yields the updated state mean, while per-pixel outlier rejection reduces variance.The loss uses a weighted ℓ2 norm.
  • A. Optimization formulation: The posterior covariance is obtained from the relationship between the Hessian of a Gaussian loss and covariance.This provides uncertainty alongside the filtered state estimate.

B. Performance enhancing optimization details

The system improves pose optimization by recursively projecting updates onto SE(3), then combines the planner and filter in a receding-horizon loop that replans after disturbances.

  • B. Performance enhancing optimization details: Recursive SE(3) optimization projects back onto the manifold after every gradient step rather than only after the full optimization.The two procedures differ because exponential-map updates on SE(3) do not generally commute.
  • B. Performance enhancing optimization details: The authors qualitatively observe that recursive SE(3) descent converges quicker and more smoothly than tangent-plane optimization under noisy photometric losses.The implementation uses the LieTorch library.
  • B. Performance enhancing optimization details: In Stonehenge, replanning after a vertical disturbance changes the route from below the arch to above it.The figure depicts the initial plan in blue, noisy execution in white, and the replanned trajectory in green.
  • B. Performance enhancing optimization details: The online pipeline initializes a plan, executes its first action, filters a new camera image, and hot-starts replanning from the updated posterior mean.The loop continues until the robot reaches the goal state.

VII. EXPERIMENTS

The experiments evaluate the planner, estimator, and complete navigation pipeline in high-fidelity simulated mesh environments using trained NeRF models and ground-truth geometry for assessment.

  • Experiments use high-fidelity simulated mesh environments to evaluate the trajectory planner, state estimator, and complete online replanning pipeline.NeRFs are trained from rendered mesh images, while robot observations and evaluation use the underlying ground-truth environment.
  • The trajectory optimizer generates aggressive, dynamically feasible trajectories for quadrotors and omnidirectional robots while avoiding collisions.
  • NeRF collision loss correlates with ground-truth mesh intersection volume during playground trajectory optimization.This supports using the NeRF density-based objective to reduce actual collision overlap.

B. Planner - Comparison to prior work

The planner is compared with minimum-snap and RRT baselines, showing smooth, low-effort, low-collision trajectories and applicability to a tight-space omnidirectional robot maneuver.

  • B. Planner - Comparison to prior work: Minimum-snap uses the same A*-generated waypoints but cannot optimize their locations based on the NeRF.
  • B. Planner - Comparison to prior work: RRT requires converting the NeRF to a mesh and uses a spherical collision model because it plans positions without robot orientation.
  • B. Planner - Comparison to prior work: The comparison evaluates all three planners on Stonehenge trajectories spanning different obstacle configurations, speeds, and complexities.Failure is defined as collision with the ground-truth mesh.
  • B. Planner - Comparison to prior work: The proposed planner yields low control effort, low NeRF collision cost, and a low failure rate compared with minimum-snap and RRT.Minimum-snap clips a ground-truth column in the example, while RRT's erratic trajectory has high control effort.
  • C. Planner - Omnidirectional robot in tight space: An omnidirectional couch-shaped robot turns to fit through a narrow gap while avoiding collision using the NeRF-based collision penalty.This demonstrates the planner on a difficult kinematic planning problem beyond quadrotor navigation.

D. Estimator - Comparison to prior work

The estimator is evaluated against a dynamically informed iNeRF baseline, while the broader pipeline compares feedback replanning with open-loop execution under noisy dynamics.

  • D. Estimator - Comparison to prior work: The estimator comparison evaluates a dynamically informed iNeRF baseline and the proposed full filter under identical actions and initial states.The baseline optimizes photometric loss after dynamics-based propagation, whereas the full filter also uses the proposed process formulation.
  • D. Estimator - Comparison to prior work: Across 100 trials, the proposed filter outperforms dynamically informed iNeRF on almost every metric without under-performing.The filter also provides a finite state covariance for identifying low-fidelity NeRF regions.
  • E. Online Replanning: The complete pipeline is tested on planned trajectories in the playground and Stonehenge scenes with additive dynamics noise.
  • E. Online Replanning: Feedback replanning reaches the goal with collision-free trajectories, whereas open-loop execution causes collisions and divergence under disturbance.The feedback controller replans when state estimates deviate significantly from the original trajectory.

F. Performance and Timing

The navigation pipeline combines trajectory planning and state estimation, but its current computational cost limits real-time deployment. Future work targets faster NeRF representations and tighter perception-control integration.

  • Performance and Timing: Initial trajectory optimization typically requires 20s over 2500 iterations, while subsequent online updates require 2s over 250 iterations at Δt = 0.1s.These timings were measured on an AMD Ryzen 9 5900X CPU with an Nvidia RTX 3090 GPU.
  • Performance and Timing: The state estimator typically runs for 4s over 300 gradient steps, including 0.25s for Hessian computation.The reported timing depends on the number of optimization iterations.
  • Performance and Timing: The proposed planner and pose filter form a full online trajectory planning and replanning pipeline for collision-free navigation.The pipeline combines discrete-time differential-flatness trajectory optimization with vision-based state estimation.
  • Performance and Timing: The method’s execution speed is limited by the underlying NeRF representation, motivating future work on faster NeRF implementations.The authors hope to leverage reported orders-of-magnitude improvements in NeRF performance.
  • Performance and Timing: The authors aim to integrate perception and control more actively by directing cameras toward informative views and using estimator uncertainty to reduce collision risk.These extensions are described as ongoing work rather than demonstrated capabilities of the current system.
  • Performance and Timing: Future work includes representing movable objects with multiple NeRFs, improving pixel subsampling, and deploying the method on quadrotors in real scenes.The current work includes simulation-based demonstrations, while real-scene quadrotor implementation remains a future objective.
Loading 2110.00168v2…