Source-linked AI summary
NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields
Antoni Rosinol, John J. Leonard, Luca Carlone
TL;DR
Real-time 3D reconstruction from monocular images remains difficult, while dense monocular SLAM can provide poses, depth maps, and uncertainty estimates for neural radiance-field fitting. The proposed pipeline combines dense monocular SLAM with hierarchical volumetric neural radiance fields and produces photometrically and geometrically accurate scene reconstructions in real time.
Problem
Real-time 3D reconstruction from monocular images remains one of the most difficult computer vision problems.
Method
The pipeline combines dense monocular SLAM, which estimates poses and dense depth maps with uncertainty, with hierarchical volumetric neural radiance fields.
Results
The approach produces photometrically and geometrically accurate scene reconstructions in real time from casually taken monocular video.
Takeaways & Limitations
Dense monocular SLAM provides suitable pose, depth, and uncertainty information for optimizing a hierarchical volumetric neural radiance field on the fly.
Takeaways & Limitations
The approach requires approximately 11Gb of GPU memory, which can be prohibitively large for low-compute robotics applications such as drones.
Abstract
from arXiv · showhide
We propose a novel geometric and photometric 3D mapping pipeline for accurate and real-time scene reconstruction from monocular images. To achieve this, we leverage recent advances in dense monocular SLAM and real-time hierarchical volumetric neural radiance fields. Our insight is that dense monocular SLAM provides the right information to fit a neural radiance field of the scene in real-time, by providing accurate pose estimates and depth-maps with associated uncertainty. With our proposed uncertainty-based depth loss, we achieve not only good photometric accuracy, but also great geometric accuracy. In fact, our proposed pipeline achieves better geometric and photometric accuracy than competing approaches (up to 179% better PSNR and 86% better L1 depth), while working in real-time and using only monocular images.
1. Introduction
The paper addresses real-time, geometrically and photometrically accurate 3D reconstruction from monocular images by combining dense monocular SLAM with hierarchical volumetric NeRFs and uncertainty-weighted depth supervision.
- Motivation: Monocular 3D reconstruction is difficult because images lack explicit scene-depth measurements.Deep-learning methods have improved monocular depth and SLAM, but accurate geometric and photometric mapping from casual monocular video remains unavailable in real time.
- Motivation: NeRFs provide photometrically accurate representations but are slow to infer and originally require ground-truth poses.Their density-based surfaces can also produce floater geometry without depth supervision.
- Approach: The proposed pipeline combines dense monocular SLAM with hierarchical volumetric NeRFs to reconstruct scenes from RGB streams without input poses or depths.Dense SLAM supplies pose estimates, depth maps, and uncertainty estimates for fitting the radiance field on the fly.
- Results: Real-time reconstruction from monocular RGB streams is demonstrated using the proposed pipeline.Figure 1 depicts the input image, depth uncertainty, uncertainty-thresholded point cloud, and a same-view rendered neural radiance field.
- Results: The approach achieves state-of-the-art performance on the Replica dataset for monocular approaches.The supplied passage reports this as an overall contribution without providing a numerical benchmark value.
2. Related Work
Related work spans dense monocular SLAM, accelerated neural radiance fields, and pose-free reconstruction. The paper combines these directions by replacing volumetric TSDF mapping with a hierarchical neural radiance field and weighting dense-SLAM supervision by uncertainty.
- Dense SLAM: Dense SLAM methods address computational complexity and ambiguous depth through latent representations, learned upsampling, or alternative sensors.RGB-D, Lidar, and stereo simplify depth estimation but require additional sensing or calibration.
- Dense SLAM: The paper replaces volumetric TSDF mapping with a hierarchical volumetric neural radiance field.This representation is intended to improve photometric accuracy and reconstruction completeness while allowing joint pose and map optimization.
- Neural Radiance Fields: Radiance-field research improves reconstruction speed using spatial partitioning, spherical harmonics, and hash-based hierarchical volumetric representations.Instant-NGP demonstrates real-time neural radiance-field training with a hash-based hierarchical representation.
- SLAM with NeRFs: The method uses dense SLAM poses and depth maps as probabilistic supervisory signals, weighting them by their marginal covariance.The uncertainty information is used to fit the radiance field rather than discarded.
- SLAM with NeRFs: Compared with related monocular hierarchical-map systems, the approach combines dense-SLAM pose estimation with depth supervision weighted by depth uncertainty.The paper contrasts this with direct image-alignment losses, ORB-SLAM initialization, and depth measured by RGB-D cameras.
3. Methodology
The pipeline runs dense monocular SLAM and hierarchical volumetric NeRF mapping in parallel, using SLAM poses, depths, and uncertainty to supervise radiance-field optimization. An uncertainty-weighted depth loss addresses noisy dense depths while real-time tracking and mapping support online reconstruction.
- Tracking: Droid-SLAM supplies dense depth maps, camera poses, and optical-flow weights for keyframes from sequential monocular images.Its tracking module computes dense optical flow and solves dense bundle adjustment using inverse-depth maps.
- Tracking: Dense bundle adjustment uses a block-sparse Hessian, Schur complement, and Cholesky factorization to solve camera poses and depths efficiently.The reduced camera system is solved first, after which depths are recovered and induced optical flow is fed back to the tracking network.
- Tracking: Marginal covariances for dense depths and poses are computed from the block-partitioned Hessian and passed with tracking outputs to mapping.These covariances provide the uncertainty information used to weight supervision.
- Mapping: The NeRF is supervised by RGB images and dense depths, with the depth loss weighted by depth marginal covariance to reduce the effect of noisy outliers.Dense monocular depth maps can contain large outliers and otherwise bias radiance-field reconstructions.
- Mapping: Rendered depths are produced by sampling points along each pixel ray, evaluating volume densities with an MLP, and alpha-compositing the samples.The rendered depth is the expected ray termination distance, while the color image uses analogous volumetric rendering.
- System: Tracking and mapping run continuously in parallel: tracking optimizes bundle-adjustment reprojection error, while mapping optimizes all received keyframes and renders the reconstruction.The tracking thread sends poses, images, depths, and depth covariances whenever it generates a new keyframe.
4. Results
The evaluation compares geometric and photometric reconstruction across datasets, mapping methods, and depth-loss settings. The proposed uncertainty-weighted approach achieves strong accuracy while operating in real time from monocular inputs.
- Datasets: Evaluation uses Cube-Diorama for controlled ablations and Replica for comparisons with related reconstruction methods.Cube-Diorama provides ground-truth poses, depths, and images; Replica contains five offices and three apartments rendered along 2000-frame trajectories.
- Methods for Evaluation: The comparison spans TSDF-Fusion, σ-Fusion, iMAP, Nice-SLAM, and the proposed approach using geometric L1 depth error and photometric PSNR.iMAP and Nice-SLAM are also evaluated with and without Replica ground-truth depth supervision, while TSDF methods and the proposed method use dense monocular SLAM outputs.
- Geometric and Photometric Accuracy: Up to 179% higher PSNR in office-1 and 86% better L1 accuracy in room-2 are achieved over Nice-SLAM.The strongest combined improvement occurs in office-1, with 179% better PSNR and 80% better L1.
- Geometric and Photometric Accuracy: The proposed reconstructions outperform the compared map representations, including σ-Fusion despite using the same pose, depth, and uncertainty estimates.Figure 3 compares the tracking pointcloud, σ-Fusion TSDF, Nice-SLAM, and the proposed radiance-field reconstruction.
- Depth Loss Ablation: Uncertainty weighting makes reconstruction robust to noisy poses and depths, whereas noisy poses without depth fail to converge within 60 seconds.Ground-truth pose and depth provide the ideal case; the proposed approach reaches strong results using noisy estimates weighted by uncertainty.
- Depth Loss Ablation: After 120 seconds, unweighted noisy depths worsen PSNR by 4dB and L1 depth by 7cm, while covariance weighting avoids this bias.Without depth, the radiance field reaches 7.8cm L1 error after 500 seconds; raw dense depths reduce this to 4.1cm but worsen PSNR by 3dB.
- Runtime: The pipeline reconstructs scenes in real time at 10 frames per second while processing 640 × 480 images.Tracking averages 15 frames per second and mapping averages 10 frames per second, using parallel threads and custom CUDA kernels.
5. Limitations
The approach has substantial GPU-memory demands from dense tracking correlation volumes and hierarchical mapping grids, though two strategies could reduce this burden.
- ∼11Gb of GPU memory is required because the pipeline uses dense correlation volumes for tracking and hierarchical volumetric grids for mapping.The memory requirement can be prohibitively large for low-compute robotics applications such as drones.
- Correlation volumes can be computed on the fly to reduce the need to store dense correlation volumes.
- Volumetric information can be streamed to the CPU for inactive regions while a GPU sliding window loads the region of interest.
6. Conclusion
The paper presents dense monocular SLAM as a source of pose and depth information for real-time, photometrically and geometrically accurate NeRF reconstruction. It also points toward photometrically accurate mapping for metric-semantic SLAM and high-level scene understanding.
- Dense monocular SLAM provides pose estimates and depth-maps weighted by marginal covariance estimates to optimize a hierarchical hash-based volumetric neural radiance field.
- The resulting approach generates photometrically and geometrically accurate scene reconstructions in real-time from casually taken monocular video.
- The approach could extend metric-semantic SLAM with photometrically accurate representations.
- The approach could serve as a mapping engine for high-level scene understanding, including 3D Dynamic Scene Graphs.