Source-linked AI summary
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, Marc Pollefeys
TL;DR
Existing neural implicit SLAM methods can produce over-smoothed reconstructions and struggle with large scenes because global fully connected representations lack local observation information. NICE-SLAM combines hierarchical feature grids with pretrained implicit decoders, and experiments report competitive mapping and tracking alongside detailed, scalable reconstruction. It remains limited in predictive range, lacks loop closures, and retains a performance gap relative to learning-based approaches.
Problem
Existing neural implicit SLAM methods produce over-smoothed reconstructions and have difficulty scaling to large scenes because their fully connected architectures lack local observation information.
Method
NICE-SLAM uses hierarchical feature grids for scene geometry and appearance, pretrained neural implicit decoders, and local feature updates within the viewing frustum.
Results
NICE-SLAM demonstrates competitive mapping and tracking performance while providing fine-detailed mapping, high tracking accuracy, faster speed, and less computation than a single large MLP representation.
Takeaways & Limitations
Hierarchical local updates make neural implicit SLAM more scalable and efficient while supporting geometry extrapolation into small unobserved regions that stabilizes camera tracking.
Takeaways & Limitations
Predictive ability is restricted to the coarse representation scale, NICE-SLAM does not perform loop closures, and a performance gap to learning-based approaches remains.
Abstract
from arXiv · showhide
Neural implicit representations have recently shown encouraging results in various domains, including promising progress in simultaneous localization and mapping (SLAM). Nevertheless, existing methods produce over-smoothed scene reconstructions and have difficulty scaling up to large scenes. These limitations are mainly due to their simple fully-connected network architecture that does not incorporate local information in the observations. In this paper, we present NICE-SLAM, a dense SLAM system that incorporates multi-level local information by introducing a hierarchical scene representation. Optimizing this representation with pre-trained geometric priors enables detailed reconstruction on large indoor scenes. Compared to recent neural implicit SLAM systems, our approach is more scalable, efficient, and robust. Experiments on five challenging datasets demonstrate competitive results of NICE-SLAM in both mapping and tracking quality. Project page: https://pengsongyou.github.io/nice-slam
1. Introduction
Dense visual SLAM must support real-time operation, prediction in unobserved regions, scalability, and robustness, while prior neural implicit approaches struggle with detailed reconstruction and large scenes. NICE-SLAM addresses these challenges with hierarchical local feature grids and pretrained implicit decoders, achieving competitive mapping and tracking.
- Dense visual SLAM requires real-time operation, prediction for unobserved regions, scalability to large scenes, and robustness to noisy or missing observations.
- Prior systems trade off capabilities: traditional methods scale but lack plausible unobserved-region geometry, while learning-based methods predict better but usually work only in small scenes.
- iMAP’s single global MLP is difficult to update locally from partial observations, limiting its scalability to multi-room scenes.
- NICE-SLAM represents geometry and appearance with hierarchical feature grids and pretrained decoders, optimizing only grid features within the viewing frustum.
- Local updates are a prerequisite for large-scale neural implicit SLAM, and NICE-SLAM’s hierarchical grid-based encoding supports them.
- Evaluations on various datasets demonstrate competitive performance in both mapping and tracking.
2. Related Work
Dense SLAM methods use view-centric or world-centric maps, with world-centric systems commonly relying on surfels or voxel grids. NICE-SLAM builds on iMAP’s neural implicit formulation while addressing its limitations through learnable latent embeddings and a pretrained continuous decoder.
- Dense visual SLAM separates mapping and tracking, using either view-centric keyframe geometry or world-centric representations such as surfels and voxel grids.
- World-centric voxel-grid methods commonly store occupancies or TSDF values, whereas NICE-SLAM stores implicit latent geometry codes in a voxel-grid representation.
- iMAP uses a single MLP to compactly represent an entire RGB-D scene, but limited model capacity degrades detailed geometry and camera tracking in larger scenes.
- NICE-SLAM combines learnable latent embeddings with a pretrained continuous implicit decoder to provide a scalable alternative to iMAP.
3. Method
NICE-SLAM represents scene geometry and appearance with hierarchical feature grids and decoders, then jointly estimates the scene and camera through differentiable rendering losses. Its coarse-to-fine representation supports local updates, unobserved-region geometry prediction, and alternating mapping and tracking.
- System Overview: The system renders depth and color from a hierarchical scene representation and camera pose, then estimates both by backpropagating reconstruction losses through a differentiable renderer.Mapping updates the scene representation, while tracking updates the camera pose.
- Hierarchical Scene Representation: Geometry uses coarse, mid, and fine feature grids with corresponding decoders, while a separate feature grid and decoder model scene color.The fine-level decoder predicts a residual occupancy offset from the mid-level occupancy.
- Hierarchical Scene Representation: Fixed pretrained geometry decoders provide inductive biases, whereas color features and their decoder are jointly optimized during reconstruction.Joint color optimization improves tracking performance, but color remains locally consistent and may require global postprocessing for whole-scene visualization.
- Depth and Color Rendering: For each camera ray, sampled points are queried for occupancy and color, whose termination weights produce coarse and fine depth estimates plus a rendered color image.The method uses stratified and importance sampling near the observed depth.
- Mapping and Tracking: Mapping minimizes L1 geometric and photometric re-rendering losses over the current frame and selected keyframes, using staged optimization of the feature grids.Tracking optimizes camera pose through depth-variance and photometric objectives, while dynamic-object pixels with losses above 10× the frame median are filtered.
- Keyframe Selection: Keyframes with visual overlap are selected for geometry optimization, enabling local updates while keeping geometry outside the current view static and reducing the optimized parameters.The active set includes K−2 overlapping keyframes, the most recent keyframe, and the current frame.
4. Experiments
NICE-SLAM is evaluated across five datasets and multiple settings for reconstruction, tracking, runtime, robustness, geometry prediction, and architectural choices. The results show detailed and scalable reconstruction, accurate tracking, efficiency gains, robustness to dynamic objects, and benefits from hierarchical design choices.
- Experimental Setup: NICE-SLAM is evaluated on Replica, ScanNet, TUM RGB-D, Co-Fusion, and a self-captured large multi-room apartment.Baselines include TSDF-Fusion, DI-Fusion, and an iMAP re-implementation.
- Mapping and Tracking: On Replica, NICE-SLAM significantly outperforms baseline methods on almost all reconstruction metrics while maintaining reasonable memory consumption.Qualitatively, it produces sharper geometry and fewer artifacts.
- Mapping and Tracking: On TUM RGB-D and ScanNet, NICE-SLAM improves camera tracking over implicit-SLAM baselines and produces sharper, more detailed geometry on large scenes.State-of-the-art traditional tracking methods still outperform implicit representations on TUM RGB-D, but NICE-SLAM reduces that gap.
- Performance Analysis: NICE-SLAM requires 1/4 the FLOPs of iMAP and is over 2× faster for tracking and 3× faster for mapping under the reported sampling settings.Its FLOPs remain unchanged for very large scenes, unlike iMAP’s single-MLP representation.
- Performance Analysis: On Co-Fusion, NICE-SLAM achieves 1.6cm ATE RMSE versus 7.8cm for iMAP∗, demonstrating improved tracking robustness to dynamic objects.The method masks pixels associated with the moving object during optimization.
- Performance Analysis: Coarse-level scene priors enable NICE-SLAM to predict unobserved regions and fill holes, while hierarchical optimization adds geometric detail and improves convergence.Ablations also report that local bundle adjustment and learned color representation improve tracking accuracy and robustness.
5. Conclusion
NICE-SLAM combines neural implicit representations with hierarchical grid-based scene representations to support detailed, efficient dense visual SLAM. Its predictive geometry can fill small holes and stabilize tracking, while predictive ability remains limited to coarse-representation scale and loop closures are absent.
- NICE-SLAM combines neural implicit representations with hierarchical grid-based scene representations for dense visual SLAM.
- Fine-detailed mapping, high tracking accuracy, faster speed, and lower computation result from tiny MLPs, multiresolution feature grids, and local scene updates.
- The network fills small holes and extrapolates geometry into unobserved regions, which stabilizes camera tracking.
- Predictive ability is restricted to the scale of the coarse representation, and the method does not perform loop closures.
- A performance gap remains between NICE-SLAM and learning-based approaches.
– Supplementary Material – NICE-SLAM: Neural Implicit Scalable Encoding for SLAM
The supplementary material provides implementation details and additional experimental analyses for NICE-SLAM. The listed contributors and acknowledgments accompany these materials.
- The listed authors include Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, and additional collaborators.
- The supplementary material includes implementation details and parameters.
- The supplementary material includes additional experiments and ablations.
A. Implementation Details
NICE-SLAM uses hierarchical feature grids, pretrained decoders, and selective local optimization to balance reconstruction quality, efficiency, and predictive geometry. Supplementary details cover initialization, visualization, pretraining, optimization settings, and design ablations.
- Feature selection: Feature selection updates only features fully inside the current viewing frustum, preserving reconstructed geometry and reducing optimization parameters.
- Feature-grid initialization: The fine-level grid is initialized so its residual decoder output is zero, enabling a smooth coarse-to-fine energy transition.
- Feature-grid levels: A 3-level feature grid provides a balance between reconstruction quality, computational efficiency, real-time capability, and memory consumption.The reported memory consumption is 12 MB for Replica scenes.
- Feature-grid design: The coarse level is disconnected from mid- and fine-level residuals because its voxel size exceeds 1 meter, so updating it would affect a large area.
- Scene reconstruction: The scene uses hierarchical grids for implicit reconstruction, with marching cubes producing visualization meshes and coarse predictions covering partially observed voxels.
- Pretraining: The encoder-decoder is pretrained on the Synthetic Indoor Scene Dataset using a Point Cloud Encoder and 32-dimensional feature grids.
- Online optimization: NICE-SLAM tracks every frame and usually optimizes geometry every fifth frame, with TUM RGB-D using geometry optimization every frame.
B.1. Frame Loss Robustness
Under extreme frame loss, NICE-SLAM recovers camera pose and scene geometry more effectively than iMAP*. Its coarse-level geometric representation improves prediction capability during the missing-frame interval.
- After skipping 100 frames, NICE-SLAM recovers the camera pose using 300 iterations, whereas iMAP* struggles even with 1500 iterations.
- The coarse-level geometric representation improves prediction capability during extreme frame loss.
B.3. Frustum Feature Selection
The ablation evaluates frustum feature selection during camera motion across previously reconstructed geometry. Fixing border features prevents significant reconstruction artifacts.
- Frustum Feature Selection: Without fixing the border features, significant artifacts appear in the reconstruction.The ablation runs with and without frustum feature selection on ScanNet scene0000.
- Frame Loss: The robustness experiment reports results after frame loss at frame 2000, evaluated at frame 2100.The missing frames correspond to the straight line in the middle of the trajectory.
B.4. More Results on Replica Dataset [45]
Additional Replica results compare average metrics across five runs under visible-region evaluation and best metrics across five runs over all regions. Visualizations cover office-0 and office-4 under multiple rendering settings.
- Quantitative Results: Average Replica metrics are computed over five consecutive runs while excluding regions outside all camera viewing frustums.Best metrics over five runs are also reported when considering all regions.
- Quantitative Results: The re-implemented iMAP* has similar performance to the original iMAP across the reported Replica results.
- Visual Results: Additional visualizations use different rendering settings to highlight performance differences.
B.5. More Results on ScanNet [13]
The ScanNet results include tracking and reconstruction ablations, Replica-style visual comparisons, and a reconstruction process visualization. NICE-SLAM’s local map updates are associated with temporally more stable and less noisy geometry than iMAP* in the cited process comparison.
- Tracking: The tracking ablation uses ATE RMSE in centimeters as its evaluation metric.
- Frustum Feature Selection: The frustum feature-selection ablation examines reconstruction on ScanNet scene0000 while the camera scans other scene regions.The cutout is previously reconstructed geometry that should remain constant, and the mesh is visualized with vertex normals.
- Replica Visualizations: The Replica visual comparison includes office-0 and office-4 rendered with ambient color, shadow, and grey-normal settings.The caption states that NICE-SLAM produces high-quality geometry and colors.
- Quantitative Results: Replica reconstruction results are presented as average metrics over five runs and best metrics over five runs.The corresponding tables report average and best reconstruction results.
- Reconstruction Process: NICE-SLAM’s local map updates produce geometry that is temporally more stable and often less noisy than iMAP* during ScanNet reconstruction.