PaperlayerView exact revision on arXiv

Source-linked AI summary

Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting Maps

Timothy Chen, Ola Shorinwa, Joseph Bruno, Aiden Swann, Javier Yu, Weijia Zeng, Keiko Nagami, Philip Dames, Mac Schwager

arXiv:2403.02751v3cs.RO

TL;DR

Splat-Nav addresses navigation in maps whose limited resolution can omit scene details useful for planning and localization. It combines safe corridor planning with RGB-based pose estimation in GSplat maps, and reports safe simulation planning and hardware performance comparable to motion capture or VIO without manual frame alignment.

  • Problem

    Explicit scene representations can be limited in resolution, leaving out details that may be valuable for planning and localization.

    Source: §I · p. 1

  • Method

    Splat-Nav combines Splat-Plan, which builds safe corridors from GSplat geometry, with Splat-Loc, which estimates pose from monocular RGB observations using GSplat-rendered data and PnP.

    Source: §I · p. 1, §IV · p. 5

  • Results

    Simulations show Splat-Plan is always safe against full collision geometry while point-cloud planner variants sometimes collide or fail; across 124 hardware flights, Splat-Nav performed as well as motion capture or onboard VIO without manual frame alignment.

    Source: §I · p. 2

  • Takeaways & Limitations

    Splat-Nav supports safe closed-loop replanning and open-vocabulary goal specification in GSplat maps.

    Source: §I · p. 1, §VII · p. 15

    Trajectory safety depends on the quality of the pre-constructed GSplat map, and Splat-Plan cannot address obstacles missing from that scene.

    Source: §VIII · p. 16

Abstract

from arXiv · show

We present Splat-Nav, a real-time robot navigation pipeline for Gaussian Splatting (GSplat) scenes, a powerful new 3D scene representation. Splat-Nav consists of two components: 1) Splat-Plan, a safe planning module, and 2) Splat-Loc, a robust vision-based pose estimation module. Splat-Plan builds a safe-by-construction polytope corridor through the map based on mathematically rigorous collision constraints and then constructs a B\'ezier curve trajectory through this corridor. Splat-Loc provides real-time recursive state estimates given only an RGB feed from an on-board camera, leveraging the point-cloud representation inherent in GSplat scenes. Working together, these modules give robots the ability to recursively re-plan smooth and safe trajectories to goal locations. Goals can be specified with position coordinates, or with language commands by using a semantic GSplat. We demonstrate improved safety compared to point cloud-based methods in extensive simulation experiments. In a total of 126 hardware flights, we demonstrate equivalent safety and speed compared to motion capture and visual odometry, but without a manual frame alignment required by those methods. We show online re-planning at more than 2 Hz and pose estimation at about 25 Hz, an order of magnitude faster than Neural Radiance Field (NeRF)-based navigation methods, thereby enabling real-time navigation. We provide experiment videos on our project page at https://chengine.github.io/splatnav/. Our codebase and ROS nodes can be found at https://github.com/chengine/splatnav.

I. INTRODUCTION

Splat-Nav combines planning and RGB-based localization in Gaussian Splatting maps, where the ellipsoidal scene geometry supports safe trajectory planning. Simulations and hardware trials compare its safety, performance, and alignment requirements with existing methods.

I. INTRODUCTION

  • Explicit scene representations can be limited in resolution and omit details useful for planning and localization, while NeRF rendering requires neural-network inference that is described as impractical for real-time planning.GSplats represent scenes with Gaussian ellipsoids and provide geometrically consistent collision geometry for planning.Source: §I · p. 1
  • Splat-Nav combines Splat-Loc, which estimates pose from monocular RGB observations using GSplat-rendered data and PnP, with Splat-Plan, which routes smooth trajectories through safe corridors to positional or language-conditioned goals.The system supports open-loop trajectory generation and closed-loop replanning; localization estimates can be fused with existing estimates or used to steer motion toward safer paths.Source: §I · pp. 1–2, Figure 1 · p. 2
  • Splat-Plan is always safe against full collision geometry in simulation, while four point-cloud planner variants sometimes collide or fail to find trajectories.It achieves similar or better path lengths with similar computation time; Splat-Loc is reported as more accurate, faster, and less failure-prone than localization baselines.Source: §I · p. 2
  • Across 124 hardware flights, Splat-Nav performed as well as motion capture or onboard VIO without requiring manual alignment to the GSplat frame.Both Splat-Plan and Splat-Loc operate natively in the same GSplat map.Source: §I · p. 2

II. RELATED WORK

  • Prior planning work includes grid, sampling-based, and trajectory-optimization methods; point-cloud methods can form safe polytopes, while onboard reactive sensing is described as myopic and prone to local optima.NeRF planners offer high-level planning approaches, and the authors identify their work as the first planning algorithm suitable for GSplat scene representations.Source: §II · pp. 2–3
  • Prior localization approaches include filters, learning-based methods, and NeRF pose tracking; photometric-loss methods can have small convergence regions and require repeated rendering computations.The authors position their PnP-based localization algorithm as addressing these challenges; existing GSplat SLAM works use photometric loss and do not address safe trajectory planning and control.Source: §II · p. 3

III. 3D GAUSSIAN SPLATTING

Gaussian Splatting represents scenes with anisotropic 3D Gaussian primitives that support fast rendering and collision-geometry extraction. Splat-Plan uses ellipsoidal collision geometry to construct safe polytopic corridors and plan trajectories through them.

III. 3D GAUSSIAN SPLATTING

  • Each Gaussian uses a mean, covariance, opacity, and spherical-harmonics coefficients; its covariance is parameterized by rotation and diagonal scaling.The anisotropic covariance and adaptive splitting and merging of Gaussians support compact representations, while avoiding NeRF-style volumetric ray-marching enables real-time rendering.Source: §III · pp. 3–4
  • 0.031 Chamfer distance for the GSplat mesh (3M vertices) versus 0.081 for the NeRF point cloud (4M points) in Stonehenge; the GSplat collision geometry is extracted instantaneously.The paper attributes the NeRF geometry’s poorer collision quality partly to view inconsistency and unremoved outliers.Source: §III · p. 4, Figure 2 · p. 4
  • GSplat primitives can be extracted rapidly, while their full ellipsoidal geometry covers surfaces that may be sparse in point clouds derived from Gaussian means.The paper notes that sparse mean-based point clouds can leave a sim-to-real gap for planning near those areas.Source: §III · p. 4, §IV · pp. 4–5

IV. PLANNING WITH SAFE POLYTOPES

  • Splat-Plan builds safe polytopic corridors using ellipsoid-intersection tests, then plans Bézier curves within those corridors.Its safety guarantee depends on the map containing the obstacles; the authors limit planning to pre-trained maps and static scenes.Source: §IV · pp. 4, 5, 7
  • Splat-Plan’s collision test uses ellipsoids to check whether the robot and map geometry intersect, with a shared-basis formulation to avoid matrix inversion.The authors use GPU parallelization and bisection for efficient collision checking; the bisection estimate converges at rate 1/2^(k+1) after k iterations.Source: §IV · pp. 5–6
  • Supporting hyperplanes around the seed path define safe polytopes, and manageability ensures the seed object is enclosed by the free-space partition.The resulting connected corridor admits a feasible trajectory that stays within it.Source: §IV · pp. 6–7

1) Seed Path:

Splat-Plan converts ellipsoidal collision geometry into a conservative occupancy grid, then uses graph search to seed a path.

1) Seed Path:

  • The planner bounds each ellipsoid, subdivides boxes larger than a grid cell, and bins their vertices to over-approximate collision geometry.This procedure supports batch GPU operations.Source: 1) Seed Path · p. 7
  • A kernel derived from the robot sphere and a 3D max-pooling operation account for robot-body extent in the occupancy grid.The resulting grid marks where the robot can be centered safely or unsafely.Source: 1) Seed Path · p. 7
  • Dijkstra search on the final grid produces a seed path as an ordered set of connected line segments.The authors describe uniform-grid search as optimal and typically faster than random-tree methods when grid conversion is inexpensive.Source: 1) Seed Path · p. 7

2) Collision Set:

Splat-Plan narrows collision checking to map ellipsoids near each seed-path segment by selecting candidates from a segment-aligned bounding box.

2) Collision Set:

  • A bounding box oriented along each seed segment selects nearby ellipsoids for collision checks instead of testing every primitive in the scene.The box’s clearance from the segment is at least the maximum stopping distance plus the robot radius.Source: 2) Collision Set · p. 7
  • The planner tests whether an ellipsoid intersects the box by transforming the plane and ellipsoid into a plane and an origin-centered sphere, then checking signed distance.A transformed plane’s signed distance from the origin must be less than 1.Source: 2) Collision Set · p. 7

3) Polytope Generation:

Splat-Plan constructs corridor polytopes by adding collision constraints around seed-path segments while limiting redundant constraints and optimization cost.

3) Polytope Generation:

  • The planner greedily adds a halfspace for the remaining ellipsoid with the smallest K(s*) and removes ellipsoids that violate each new halfspace.Using every ellipsoid could overconstrain free space and slow spline optimization.Source: 3) Polytope Generation · p. 8
  • Manageability lets the planner use fewer polytopes than seed-path segments by extending each polytope until a later segment is no longer contained.More polytopes can yield smoother paths with fewer pinch points, at the cost of higher spline-optimization computation.Source: 3) Polytope Generation · p. 8

4) Spline Optimization:

Splat-Plan fits piecewise Bézier trajectories through a safe polytope corridor, minimizing path length while enforcing corridor and continuity constraints. Its safety guarantee follows from constraining Bézier control points to the corridor, subject to the stated path-search and optimization conditions.

4) Spline Optimization:

  • The optimizer fits P Bézier curves through P corridor polytopes, minimizing path length while constraining endpoints and continuity between adjacent curves.Without dynamics constraints, the optimization reduces to a quadratic program that can be solved in real time.Source: 4) Spline Optimization · p. 8
  • Constraining Bézier control points to corridor polytopes keeps the entire curves inside the corridor by the convex hull property, guaranteeing safety in the continuum.The paper states that Splat-Plan is sound and complete when a smooth safe path exists in the GSplat environment.Source: 4) Spline Optimization · p. 8
  • The stated convergence argument requires increasingly fine occupancy-grid resolution, enough corridor-construction iterations, and a sufficiently expressive Bézier parametrization.With insufficient conditions, path search or corridor construction can fail, or the curve optimization can be infeasible.Source: 4) Spline Optimization · pp. 8–9
  • In hardware demonstrations, each spline segment is timed from its seed-path length and desired maximum speed, while remaining constrained to the safety corridor.The curves are reparameterized in metric time, with continuity enforced between segments.Source: 4) Spline Optimization · p. 9

V. MONOCULAR POSE ESTIMATION

Splat-Loc estimates a robot’s pose in a GSplat map from monocular RGB, using rendered map observations and camera-tracking tools. It can initialize from an independent pose estimate or the previous estimate, and also provides a global initialization procedure when a good initial pose is unavailable.

V. MONOCULAR POSE ESTIMATION

  • Splat-Loc estimates pose from a monocular RGB image and can operate standalone or alongside an independent pose-estimation system.It assumes camera calibration is known; estimates may also be used to initialize or correct an independent system’s poses.Source: §V · p. 9, Problem Formulation · p. 9
  • Given a coarse initial pose, Splat-Loc renders a map image and local point cloud, matches image features, and solves a reprojection-error PnP problem for the pose update.The method uses rendered depth and camera intrinsics to project matched rendered-image keypoints into 3D; RANSAC removes outlier matches before Levenberg–Marquardt optimization.Source: Problem Formulation · pp. 9–10
  • When the initial pose is unavailable, Splat-Loc performs a one-time global initialization by registering a scene point cloud to a camera-frame point cloud.It augments FPFH descriptors with RGB color, finds putative correspondences, and uses RANSAC to remove outliers.Source: Global Initialization · p. 10

VI. EXPERIMENTS

The experiments evaluate Splat-Nav in simulation and real-world scenes, with ablations comparing its algorithms against existing methods.

VI. EXPERIMENTS

  • The evaluation examines navigation performance in real-world scenes on hardware and in simulation.The authors also conduct ablations against existing methods.Source: §VI · p. 10
  • The experiments include ablation studies comparing the proposed algorithms with existing methods.Source: §VI · p. 10
  • The evaluation examines the effectiveness of the Splat-Nav navigation pipeline for GSplat maps.Source: §VI · p. 10

A. Simulation Results

Across the simulation evaluations, Splat-Loc variants are assessed against pose-estimation baselines and Splat-Plan against planning alternatives. Splat-Plan’s trajectories are safe relative to the GSplat collision geometry, and it returns trajectories without failures in the reported tests.

1) Test Environments:

  • The pose-estimation benchmarks cover Stonehenge and three real-world scenes: Statues, Flightroom, and Old Union.Stonehenge image-pose pairs are rendered from a mesh; the other scenes use video frames processed with COLMAP.Source: 1) Test Environments · p. 10

2) Splat-Loc Evaluations:

  • Splat-Loc is compared with GS-Loc, ICP, Colored-ICP, and feature-matching variants, across trials with varying initial pose errors.The study evaluates rotation error, translation error, computation time per frame, and success rate; its initial pose is available and perturbed from ground truth.Source: 2) Splat-Loc Evaluations · p. 11
  • On Statues, Splat-Loc-Glue has mean rotation error below 0.06° and translation error below 4 mm, with mean computation time below 42 ms.It has the lowest mean rotation and translation errors and fastest mean computation time among the compared methods; GS-Loc is least accurate and slowest.Source: 2) Splat-Loc Evaluations · p. 11, Table I · p. 11
  • With initial errors δR = 30° and δt = 0.5 m on Stonehenge, Splat-Loc-SIFT and Splat-Loc-Glue achieve average rotation errors below 0.5° and translation errors below 5 mm.Splat-Loc-Glue achieves a perfect success rate, while Splat-Loc-SIFT has a lower success rate; ICP and Colored-ICP are inaccurate and have substantial error variance.Source: 2) Splat-Loc Evaluations · p. 11, Table II · p. 11

3) Splat-Plan Evaluations:

  • Splat-Plan is evaluated against point-cloud, sampling-based, and NeRF-based planners across eight dense and sparse GSplat scenes.Each scene has 100 start-goal pairs; four Safe Flight Corridor variants test point-cloud-based planning choices.Source: 3) Splat-Plan Evaluations · pp. 11–12, Table III · p. 11
  • Splat-Plan’s paths are described as smooth and non-conservative, with comparable execution times to SFC methods.In the scene with the most Gaussians, Old Union, GPU memory use is about 3.1 GB: 1.6 GB for GSplat and 1.5 GB for the occupancy grid.Source: 3) Splat-Plan Evaluations · p. 12
  • Splat-Plan’s trajectories remain safe relative to the GSplat collision geometry, and it has a 0 failure rate while other methods fail except NeRF-Nav.The authors note that no ground-truth mesh was available for many real-world scenes and use the GSplat ellipsoidal geometry as a high-quality approximation.Source: 3) Splat-Plan Evaluations · p. 12, Table IV · p. 12

B. Hardware Results

Hardware tests evaluate Splat-Nav’s localization and planning across language-specified goals and three control schemes. In the controlled Maze setting, Splat-Loc achieved accuracy comparable to VIO, while one keyboard-goal flight was an exception to otherwise collision-free flights.

1) Test Environment:

  • The Maze semantic GSplat’s rendered image was photorealistic, and semantic relevancy aligned spatially with queried objects for language-based goals.The tested object prompts were keyboard, beachball, phonebook, and microwave; their locations required turns or elevation maneuvers and were initially out of view.Source: 1) Test Environment · p. 12, 4) Goal Specification · pp. 12–13

2) Hardware:

  • Hardware tests used a Modal AI drone, approximated as a 0.5 m-diameter sphere, with OptiTrack used only for evaluation.The drone measures 29 cm × 20 cm × 10 cm.Source: 2) Hardware · p. 12

5) Control Schemes:

  • Open-loop used one initial trajectory, while Closed-loop VIO and Splat-Loc replanned; each goal/control combination was repeated 10 times.The Splat-Loc scheme transforms Splat-Plan trajectories into the VIO control frame.Source: 5) Control Schemes · p. 13

6) Splat-Loc Evaluations:

7) Splat-Plan Evaluations:

  • All 120 flights were collision-free except one Splat-Loc keyboard-goal flight, where poor GSplat coverage near the scene edge was implicated.The authors attribute the exception to few features and poor map quality where training images did not cover the scene edge.Source: 7) Splat-Plan Evaluations · pp. 13–14, 6) Splat-Loc Evaluations · p. 13
  • Splat-Plan and four SFC variants were compared across four scenes; the figure reports superior safety and success rates with competitive path length and computation time.The comparison includes dense and sparse scenes and assesses distance to GSplat, failure rate, path length, polytope volume, and computation time.Source: Figure 5 · p. 14
  • Across four scenes and 100 shared start–end pairs per scene, Splat-Plan was safer than SFC, RRT*, and NeRF-Nav and never failed in eight settings.The figure compares dense and sparse scenes using safety, path length, and computation-time measures.Source: Figure 6 · p. 14

8) Fast Control:

The paper reports real-time Splat-Nav operation and collision-free endurance flights, while identifying limits tied to pre-constructed, static scenes, map quality, visual features, and GPU memory. Its tests did not cover online mapping or dynamic scenes.

8) Fast Control:

  • Fast-control tests increased vmax to 1.5 m/s, the speed at which onboard VIO could no longer track waypoints accurately enough to avoid collision.The resulting flights combined these speeds with a cluttered environment.Source: 8) Fast Control · p. 15

9) Closed-loop Endurance:

  • The re-planning pipeline demonstrated collision-free flight over minutes, setting another goal whenever the drone reached one.The endurance pipeline was left to execute continually.Source: 9) Closed-loop Endurance · p. 15

VII. CONCLUSION

  • Splat-Nav combines guaranteed-safe Splat-Plan planning above 2 Hz with Splat-Loc pose estimation at 25 Hz, and the paper reports hardware and simulation evaluations.The conclusion characterizes planning as faster than 2 Hz and pose estimation as 25 Hz.Source: §VII · p. 15

VIII. LIMITATIONS AND FUTURE WORK

  • Testing used pre-constructed, static scenes; safety also depends on map quality, informative features, and access to about 10 GB of GPU memory.The authors identify online GSplat mapping, dynamic scenes, feature-poor regions, and onboard compute resources as limitations or future-work areas.Source: §VIII · pp. 15–16

APPENDIX A PROOF OF PROPOSITION 1

The appendix derives a shared eigenbasis for two covariance matrices to simplify matrix inversion, then proves that a linearized collision constraint defines a supporting hyperplane for the safe set.

APPENDIX A PROOF OF PROPOSITION 1

  • A shared generalized eigenbasis diagonalizes both covariance matrices, reducing matrix inversion to reciprocals of eigenvalues.The proof uses Cholesky decomposition of Σb, eigendecomposes the resulting symmetric positive-definite matrix, and recovers the original eigenbasis.Source: Appendix A · pp. 16–17

APPENDIX B PROOF OF PROPOSITION 2

  • The safety test for each collision ellipsoid is approximated as a polytope using a quadratic constraint on the test point.The robot position is represented by the test point, and the construction applies the collision test to each ellipsoid in the test set.Source: Appendix B · p. 17
  • Linearizing the ellipsoid constraint at an exterior point yields a supporting hyperplane for the safety constraint.The proof selects a point outside the ellipsoid along the ray from its center through the robot center, then linearizes the constraint there.Source: Appendix B · p. 17

APPENDIX C PROOF OF COROLLARY 5

The appendix proves that the hyperplane constraint contains the segment between safe endpoints, including interior and boundary optima. It also reports the performance of Splat-Loc's global pose initialization in two scenes.

A. Optimum in the Interior

  • For an interior optimum, the full line segment remains in the hyperplane and is not in collision.The argument uses the safe point at the unconstrained minimum and orthogonality of the segment direction to the ellipsoid normal.Source: Appendix C · p. 17, A. Optimum in the Interior · p. 18

B. Optimum on the Boundary

  • For a boundary optimum, the line segment still remains within the polytope, completing the proof of Corollary 5.The boundary case sets the closest point on the segment as the starting point and considers an unconstrained optimum outside the segment.Source: B. Optimum on the Boundary · p. 18

APPENDIX D GLOBAL POSE INITIALIZATION OF SPLAT-LOC

  • Global point-cloud alignment succeeds in 80% of both Statues and Stonehenge trials, runs at approximately 20–30 Hz, and yields about 0.3 deg and 1 cm pose errors when successful.These estimates are less accurate than the recursive pose estimates; Table VII reports initialization-module performance.Source: Appendix D · p. 18, Table VII · p. 18

Related papers

Loading 2403.02751v3…