Source-linked AI summary
NICER-SLAM: Neural Implicit Scene Encoding for RGB SLAM
Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R. Oswald, Andreas Geiger, Marc Pollefeys
TL;DR
Dense RGB SLAM must handle ambiguous geometry without depth sensors, while prior neural implicit systems either depend on RGB-D input or separate tracking from mapping. NICER-SLAM jointly optimizes camera poses and a hierarchical implicit map using monocular cues, optical flow, warping, and adaptive SDF-to-density transformation, and reports strong mapping, tracking, and novel view synthesis across synthetic and real-world datasets. Its current pipeline is not optimized for real-time operation and performs no loop closure.
Problem
Prior dense neural implicit SLAM methods rely on RGB-D sensors or separate tracking pipelines and do not produce high-fidelity dense 3D reconstruction from monocular RGB alone.
Method
NICER-SLAM jointly optimizes camera poses and a hierarchical neural implicit map from RGB video, adding monocular geometric cues, optical flow, warping loss, and adaptive SDF-to-density transformation.
Results
NICER-SLAM demonstrates strong mapping, tracking, and novel view synthesis performance on synthetic and real-world datasets, competitive with recent RGB-D SLAM methods.
Takeaways & Limitations
The system reconstructs high-fidelity dense maps and learns high-quality scene colors robustly in large indoor scenes using only RGB input.
Takeaways & Limitations
The pipeline is not optimized for real-time operation and currently performs no loop closure, leaving tracking performance potentially improvable.
Abstract
from arXiv · showhide
Neural implicit representations have recently become popular in simultaneous localization and mapping (SLAM), especially in dense visual SLAM. However, previous works in this direction either rely on RGB-D sensors, or require a separate monocular SLAM approach for camera tracking and do not produce high-fidelity dense 3D scene reconstruction. In this paper, we present NICER-SLAM, a dense RGB SLAM system that simultaneously optimizes for camera poses and a hierarchical neural implicit map representation, which also allows for high-quality novel view synthesis. To facilitate the optimization process for mapping, we integrate additional supervision signals including easy-to-obtain monocular geometric cues and optical flow, and also introduce a simple warping loss to further enforce geometry consistency. Moreover, to further boost performance in complicated indoor scenes, we also propose a local adaptive transformation from signed distance functions (SDFs) to density in the volume rendering equation. On both synthetic and real-world datasets we demonstrate strong performance in dense mapping, tracking, and novel view synthesis, even competitive with recent RGB-D SLAM systems.
1. Introduction
NICER-SLAM addresses the limitations of dense RGB SLAM by jointly optimizing camera tracking and neural implicit mapping from monocular RGB video. It combines hierarchical SDF encoding with monocular cues, optical flow, warping, and adaptive density transformation to support high-quality reconstruction and rendering.
- Motivation: Existing dense visual SLAM methods commonly require RGB-D sensors, while monocular systems are limited to small scenes and restricted camera motion.RGB-D methods also struggle when depth sensors are unavailable or for unobserved regions.
- Motivation: RGB-only neural implicit SLAM remains fragmented because prior systems separate camera tracking from mapping and do not produce dense 3D scene models.Those systems use visual odometry for tracking and neural radiance fields for mapping, evaluating rendered depth and color instead.
- Challenges: RGB-only SLAM faces depth ambiguity, harder surface reconstruction, and slower optimization convergence, especially in textureless or sparsely observed regions.These challenges require stronger geometric priors for both mapping and tracking.
- Approach: NICER-SLAM introduces end-to-end joint optimization of camera poses and a hierarchical neural implicit scene representation from monocular RGB video.The representation models SDFs with coarse-to-fine feature grids for detailed geometry and high-fidelity renderings.
- Approach: The system adds monocular geometric cues, optical flow, an RGB warping loss, and a locally adaptive SDF-to-density transformation to improve geometry consistency and indoor-scene fitting.The contributions include different geometric and motion regularizations alongside the adaptive transformation.
- Results: NICER-SLAM reports strong mapping, tracking, and novel view synthesis performance on synthetic and real-world datasets, competitive with recent RGB-D SLAM methods.The system is also designed to produce high-quality novel view synthesis.
2. Related Work
Prior dense SLAM systems use view-centric or world-centric maps, while neural implicit SLAM improves representation flexibility but often remains limited by scene size, camera motion, or RGB-D dependence. NICER-SLAM uses hierarchical latent codes in multi-resolution voxel grids for dense geometry and plausible unobserved-region estimation.
- Dense Visual SLAM: Dense SLAM represents scenes either with view-centric keyframe depth maps or world-centric surfels, occupancies, and TSDF voxel grids.NICER-SLAM belongs to the world-centric category but stores latent codes rather than explicit surfaces.
- Dense Visual SLAM: NICER-SLAM’s multi-resolution voxel-grid latent codes support high-quality geometry at low grid resolutions and plausible estimates for unobserved regions.This representation anchors full-scene geometry in uniform world coordinates.
- Neural Implicit-based SLAM: Neural implicit SLAM has shown promise for joint scene and pose optimization, but early methods generally handle only small objects or small camera movements.Recent works relax those constraints but rely mainly on external SLAM systems for tracking.
- Neural Implicit-based SLAM: iMAP and NICE-SLAM unified neural implicit mapping with camera tracking, with NICE-SLAM scaling to larger indoor environments through hierarchical feature grids and tiny MLPs.Follow-up methods target efficiency, optimization speed, inertial measurements, or alternative shape representations.
- Positioning: Figure 2 depicts an RGB-only pipeline that jointly estimates camera poses and a hierarchical scene representation using rendered colors, depths, normals, warping, and optical flow.A concurrent system emphasizes camera tracking, whereas NICER-SLAM focuses on high-quality 3D reconstruction and novel view synthesis.
- Neural Implicit-based SLAM: Most neural implicit SLAM follow-ups still require RGB-D inputs, limiting use when depth sensors are unavailable or in outdoor scenes.This dependence remains a central distinction between those systems and RGB-only approaches.
3. Method
NICER-SLAM jointly optimizes camera tracking and a hierarchical neural implicit scene representation from RGB video. It combines differentiable volume rendering with adaptive geometric representations and multiple RGB-based constraints for dense reconstruction and tracking.
- System Overview: NICER-SLAM jointly estimates camera poses, scene geometry, and colors through end-to-end optimization from RGB video.The representation supports rendering color, depth, and normals for joint mapping and tracking.
- Hierarchical Neural Implicit Representations: Hierarchical coarse-to-fine feature grids and MLP decoders represent SDF geometry, combining scene layout modeling with high-frequency geometric details.The coarse representation uses a 32 × 32 × 32 voxel grid, while the fine representation models residual SDF values across multiple resolutions.
- Hierarchical Neural Implicit Representations: Color prediction enables RGB-based optimization and on-the-fly novel-view image rendering.Color is encoded with a separate multi-resolution feature grid and decoder.
- Volume Rendering: Differentiable volume rendering converts sampled SDF values to densities and aggregates color, depth, and normals along each camera ray.The rendering process samples points x_i = o + t_i v and uses transmittance, alpha values, and inter-sample distances for aggregation.
- Volume Rendering: A locally adaptive SDF-to-density transformation assigns local β values because a single global β is sub-optimal for complicated sequential indoor scenes.The local values are derived from voxel-wise point-sample counts through an empirically fitted exponential transformation.
- End-to-End Joint Mapping and Tracking: RGB rendering, warping, and optical-flow losses provide geometric, photometric, and regional constraints for joint mapping and tracking under RGB-only ambiguity.The warping loss projects rendered depth from one frame into nearby keyframes, while optical flow supplies regional smoothness priors.
4. Experiments
Experiments on synthetic Replica and challenging real-world 7-Scenes evaluate reconstruction, tracking, novel view synthesis, and design choices. NICER-SLAM produces detailed geometry and strong rendering while remaining competitive in tracking and benefiting from its monocular geometric cues and SDF-based representation.
- Experimental Setup: NICER-SLAM is evaluated on synthetic Replica and low-resolution, motion-blurred 7-Scenes data against RGB-D, monocular, and MVS baselines.Baselines include NICE-SLAM, Vox-Fusion, COLMAP, DROID-SLAM, and a tracking variant without global bundle adjustment.
- Mapping and Tracking: On Replica, NICER-SLAM significantly outperforms RGB-only baselines in scene geometry and remains competitive with RGB-D methods.The reported geometry evaluation uses Accuracy, Completion, Completion Ratio, and Normal Consistency.
- Mapping and Tracking: On Replica, camera tracking reaches 1.88 cm average ATE RMSE versus 1.95 cm for NICE-SLAM without ground-truth depth input.DROID-SLAM achieves the strongest tracking performance overall, while NICER-SLAM remains on par with NICE-SLAM.
- Novel View Synthesis: NICER-SLAM produces better novel-view synthesis than all baselines on Replica, including methods using additional depth inputs.The evaluated viewpoints are far from the input views, requiring extrapolation; traditional methods leave missing regions, whereas neural implicit baselines tend to oversmooth renderings.
- 7-Scenes: On 7-Scenes, NICER-SLAM produces sharper, more detailed geometry and is more robust than RGB-only baselines in textureless and reflective regions.RGB-D methods still outperform RGB-only methods overall for tracking on imperfect RGB input, while predicted monocular geometric priors improve NICER-SLAM’s robustness.
- Ablation Study: Ablations show that combined losses, multi-resolution color grids, coarse SDF grids, and locally adaptive SDF-to-density transformation improve mapping and tracking.Replacing SDFs with occupancy while keeping the architecture identical produces less accurate geometry on a randomly selected Replica scene.
5. Conclusions
NICER-SLAM jointly optimizes neural implicit map representations and camera poses, using monocular cues to reconstruct high-fidelity dense maps and scene colors in large indoor scenes. The pipeline improves mapping and novel view synthesis but is not yet real-time and currently lacks loop closure.
- NICER-SLAM jointly optimizes neural implicit map representations and camera poses for dense RGB SLAM.
- Additional monocular depths, normals, and optical flows support accurate, robust reconstruction of high-fidelity 3D dense maps and scene colors in large indoor scenes.
- The pipeline is not yet optimized for real-time operation.
- Because the current pipeline performs no loop closure, its tracking performance could be further improved.