Source-linked AI summary
Orbeez-SLAM: A Real-time Monocular Visual SLAM with ORB Features and NeRF-realized Mapping
Chi-Ming Chung, Yang-Che Tseng, Ya-Ching Hsu, Xiang-Qian Shi, Yun-Hung Hua, Jia-Fong Yeh, Wen-Chin Chen, Yi-Ting Chen, Winston H. Hsu
TL;DR
Visual SLAM needs to adapt to new scenes without pre-training while producing dense maps in real time, but prior systems have limitations in depth dependence, input modality, or speed. Orbeez-SLAM combines visual odometry with fast NeRF mapping to support monocular RGB operation and reports faster runtime with strong rendering outcomes.
Problem
Visual SLAM lacks a broadly suitable solution that is pre-training-free, real-time, dense, and adaptable to new scenes.
Method
Orbeez-SLAM combines ORB-SLAM2 visual odometry with an instant-ngp-based NeRF map, jointly estimating poses and updating the NeRF online.
Results
360 ∼800 times faster than NICE-SLAM, Orbeez-SLAM also outperforms NICE-SLAM variants on rendered-image PSNR and can provide superior RGB results without depth supervision.
Takeaways & Limitations
Orbeez-SLAM provides a pre-training-free dense-map SLAM system that works with monocular cameras for spatial-AI applications.
Takeaways & Limitations
Effective use of dense maps in downstream tasks remains outside the paper’s scope and is left for future work.
Abstract
from arXiv · showhide
A spatial AI that can perform complex tasks through visual signals and cooperate with humans is highly anticipated. To achieve this, we need a visual SLAM that easily adapts to new scenes without pre-training and generates dense maps for downstream tasks in real-time. None of the previous learning-based and non-learning-based visual SLAMs satisfy all needs due to the intrinsic limitations of their components. In this work, we develop a visual SLAM named Orbeez-SLAM, which successfully collaborates with implicit neural representation and visual odometry to achieve our goals. Moreover, Orbeez-SLAM can work with the monocular camera since it only needs RGB inputs, making it widely applicable to the real world. Results show that our SLAM is up to 800x faster than the strong baseline with superior rendering outcomes. Code link: https://github.com/MarvinChung/Orbeez-SLAM.
I. INTRODUCTION
Orbeez-SLAM targets a pre-training-free, real-time visual SLAM system that adapts to new scenes and produces dense maps. It combines visual odometry with fast NeRF mapping to support monocular operation and reports strong speed, tracking, and reconstruction results.
- Motivation: Spatial AI needs visual SLAM that adapts to new scenes without pre-training and generates fine-grained maps in real time.Traditional systems emphasize localization accuracy but provide crude maps, while Tandem’s depth module requires pre-training.
- Related limitations: NeRF-SLAM baselines require RGB-D inputs and converge slowly because they optimize poses through neural rendering without visual odometry.instant-ngp addresses training speed using multiresolution hash encoding and CUDA.
- Proposed approach: Orbeez-SLAM combines ORB-SLAM2 visual odometry with instant-ngp-based NeRF for online, real-time mapping without pre-training.Visual odometry improves early camera pose estimation, allowing monocular operation without depth supervision.
- Evaluation: Orbeez-SLAM produces dense scene information, including depth and color, while being evaluated on indoor scenes against state-of-the-art baselines.The reported evaluation covers speed, camera tracking, and reconstruction quality.
II. RELATED WORKS
Related work spans explicit and implicit scene representations, feature-based and direct SLAM, and NeRF-based mapping. Orbeez-SLAM addresses reported gaps in adaptation, monocular input, and real-time dense mapping by combining visual odometry with fast NeRF.
- Implicit representations: Implicit neural representations provide continuous scene representations that are not coupled to input spatial dimension or resolution.NeRF is an implicit representation used successfully for novel-view synthesis.
- NeRF-SLAM: Prior NeRF-SLAM systems use SDF or NeRF maps but require RGB-D inputs and suffer from slow convergence.These limitations motivate real-time monocular NeRF-SLAM trained from scratch at the target scene.
- Visual SLAM systems: Traditional feature-based SLAM matches image features and minimizes reprojection error, whereas direct SLAM uses pixel intensities and photometric error.Traditional systems can rapidly compute accurate locations but lack fine-grained scene information.
- Orbeez-SLAM: Orbeez-SLAM combines visual-odometry guidance with a fast NeRF implementation to avoid pre-training and achieve real-time inference for novel scenes.The method is designed to provide dense maps for complicated spatial-AI tasks.
III. PRELIMINARIES
NeRF represents a scene as a continuous function mapping 3D position and viewing direction to color and density. It renders ray colors by integrating these quantities, approximated through discrete sampled points and accelerated by skipping unoccupied voxels.
- NeRF representation: NeRF reconstructs a 3D scene from 2D images by learning a continuous function of 3D location and viewing direction.The function outputs color and volume density and is approximated with an MLP.
- Volume rendering: Ray color is computed by integrating transmittance, volume density, and color along the ray between near and far bounds.The ray is parameterized as r(t) = o + td.
- Discrete rendering: NeRF approximates ray rendering by sampling N discrete positions between the near and far bounds and applying quadrature.The sampled points estimate the continuous rendering integral.
- Sampling strategy: The skip voxel strategy ignores unoccupied voxels and samples voxels intersecting the surface.This focuses computation on positions contributing more strongly to the rendered ray color.
B. Density grid
The density grid accelerates ray rendering by storing NeRF query results and skipping unoccupied regions. Orbeez-SLAM extends this strategy with accumulated ray-scanning information to guide surface triangulation and reject noise.
- Density-grid acceleration: Coarse-to-fine ray sampling requires frequent NeRF queries, motivating density grids that store query results for faster voxel skipping.Only positions likely to intersect surfaces are sampled.
- Surface identification: Orbeez-SLAM stores voxel sampling counts in addition to density predictions to identify voxels frequently blocking cast rays.These voxels are treated as more likely to contain surfaces.
IV. METHODOLOGY
Orbeez-SLAM combines VO-based feature tracking with NeRF mapping to estimate poses, refine sparse maps, and generate dense scene representations. Tracking and mapping run concurrently, while reprojection, bundle-adjustment, and photometric objectives optimize the system.
- System overview: Orbeez-SLAM uses VO for camera-pose estimation, triangulation and bundle adjustment for sparse mapping, and NeRF training for dense maps.Updated poses and map points are passed to NeRF, which generates dense maps for downstream tasks.
- Optimization: Pose estimation minimizes reprojection error between observed pixels and projections of 3D map points.The projection uses camera intrinsics and extrinsics, and camera poses are optimized by minimizing this error.
- Optimization: Bundle adjustment jointly optimizes keyframe camera poses and observable map-point positions after VO triangulation.The optimized keyframes and map points are then passed to NeRF.
- Optimization: NeRF regression minimizes photometric error between predicted and observed ray colors.Rays are formed from keyframe camera parameters and pixel coordinates, with sampling concentrated near surfaces using skip voxels.
- System overview: Tracking and mapping run concurrently, and a frame becomes a keyframe only when tracking is sufficient and mapping is not busy.The tracking process estimates poses, while mapping refines poses and maintains maps.
C. Ray-casting triangulation
Ray-casting triangulation converts frequently sampled density-grid voxels into dense NeRF map points. A threshold-based counter rejects insufficiently observed voxels and supports dense point-cloud generation.
- Ray-casting triangulation: Only voxels scanned frequently enough are triangulated into map points to reject noise.The implementation uses 64 as the practical sampling threshold based on visualization quality.
- Ray-casting triangulation: The method counts ray samples per density-grid voxel and treats frequently sampled voxels as likely surfaces.A voxel receives a counter increment when its weight exceeds the surface-candidate threshold.
- Ray-casting triangulation: The resulting map points form the dense point cloud shown in the system pipeline.These points are not optimized in the bundle-adjustment objective.
V. EXPERIMENTS
Experiments use TUM RGB-D, Replica, and ScanNet, three benchmarks that provide images, depth data, and camera trajectories for evaluating visual SLAM.
- Benchmarks: Experiments evaluate Orbeez-SLAM on TUM RGB-D, Replica, and ScanNet benchmarks.These benchmarks provide extensive images, depths, and camera trajectories and are widely used in prior work.
Baselines.
The evaluation compares Orbeez-SLAM with learning-based and traditional SLAM baselines using trajectory, reconstruction, and rendering metrics. The setup emphasizes consistent GT-trajectory evaluation while excluding mesh reconstruction.
- Baselines: Baselines include DI-Fusion, iMAP, iMAP*, NICE-SLAM, BAD-SLAM, Kintinuous, and ORB-SLAM2.The comparison separates learning-based from traditional SLAM methods.
- Evaluation setting: All methods are evaluated in the depth version because monocular SLAM cannot recover scene scale without depth, while Orbeez-SLAM is also demonstrated with monocular cameras.Previous NeRF-SLAMs require depth supervision, whereas Orbeez-SLAM does not use it for rendering.
- Metrics: ATE measures aligned trajectory RMSE, while Depth L1 and PSNR evaluate estimated depth and rendered-image quality.Depth L1 compares estimated and ground-truth depth, and PSNR assesses rendered versus ground-truth images along the GT trajectory.
- Evaluation setting: Evaluation omits meshes because post-processing NeRF into meshes is not unified across methods.The authors instead use GT trajectories, depth, and PSNR to assess geometry and radiance.
- Evaluation setting: GT-trajectory evaluation uses a consistent standard despite different numbers of sampled keyframes and tests novel-view performance from keyframe subsets.The metrics can reveal incorrect localization even when a model reproduces seen keyframes.
Implementation Details.
Orbeez-SLAM is implemented by combining ORB-SLAM2 and instant-ngp, with ablations evaluating camera-pose optimization choices.
- Implementation Details: Orbeez-SLAM follows ORB-SLAM2 and instant-ngp implementations, retaining loop closing while preserving keyframes for NeRF processing.The system is implemented in C++ and CUDA on an Intel i7-9700 CPU with an NVIDIA RTX 3090 GPU.
- Implementation Details: Runtime comparison is reported in frames per second on the TUM RGB-D benchmark against NICE-SLAM.The table presents fps with higher values preferred.
- Implementation Details: Ablation results show that optimizing camera poses with reprojection error alone outperforms using both reprojection and photometric errors.The study is conducted on Replica.
B. Quantitative Results
Orbeez-SLAM is evaluated for tracking, depth reconstruction, color rendering, runtime, and qualitative reconstruction across small- and large-scale scenes.
- Quantitative Results: Orbeez-SLAM outperforms deep-learning baselines on TUM tracking and achieves the best average results across ScanNet scenes.Its tracking performance is only slightly below ORB-SLAM2 on TUM while also providing a dense NeRF map.
- Quantitative Results: Orbeez-SLAM beats all NICE-SLAM variants on PSNR and surpasses NICE-SLAM in Depth L1 when NICE-SLAM renders without ground-truth depth.NICE-SLAM achieves the best Depth L1 when ground-truth depth is supported during rendering.
- Quantitative Results: The rendering comparison visualizes RGB and depth outputs for Orbeez-SLAM and NICE-SLAM under monocular and RGB-D settings.Orbeez-SLAM does not use depth for NeRF rendering in the RGB-D setting, whereas depth is used for tracking.
- Quantitative Results: NeRF-rendered results are shown at the beginning of training, after tracking, and after full loss convergence across TUM, Replica, and ScanNet.The qualitative results are strong on TUM and Replica but fail on the large ScanNet scene.
- Quantitative Results: Orbeez-SLAM is 360 ∼800 times faster than NICE-SLAM on the reported runtime comparison.The comparison is conducted on the TUM RGB-D benchmark.
C. Ablation Study
The ablation and qualitative studies examine pose-loss choices, depth supervision, and scene scale as factors affecting Orbeez-SLAM’s results.
- C. Ablation Study: Camera poses guided only by reprojection error perform better than poses guided by both reprojection and photometric errors.The authors attribute the difference to the slower convergence of the photometric loss during real-time inference.
- C. Ablation Study: Orbeez-SLAM provides superior RGB results without depth supervision, while NICE-SLAM produces better depth results because it accesses ground-truth depth.In the RGB-D setting, ground-truth depth is used only for Orbeez-SLAM tracking.
- C. Ablation Study: Orbeez-SLAM produces excellent rendered results on TUM and Replica but fails on the large-scale ScanNet case.The authors identify large-scale scenes as more challenging and leave improvement for future work.
- C. Ablation Study: Orbeez-SLAM combines ORB features, visual odometry, and fast NeRF mapping to provide pre-training-free monocular SLAM with dense maps.The paper leaves the effective use of dense maps in downstream tasks outside its scope.