SNI-SLAM: Semantic Neural Implicit SLAM

1. Introduction

• Prior semantic NeRF work jointly learned geometry and semantics but required hours of offline training, impractical for real-time semantic SLAM.Existing NeRF-SLAM systems mostly establish RGB maps, whose color information is not directly suitable for downstream tasks such as navigation.Source: §1 · p. 2
• Appearance, geometry, and semantic information are interrelated, while view-varying appearance and semantic multi-view consistency can affect each other's representation.The paper identifies processing modalities independently and cross-modal effects on appearance details as two challenges.Source: §1 · p. 2
• Cross-attention feature collaboration uses appearance, semantic, and geometry characteristics to mutually enhance the modalities.The authors contrast this with MSeg3D, which fuses geometry and semantics but does not use appearance as another modality.Source: §1 · p. 2
• A one-way correlation approach lets information from one modality enhance others without changing its original representation or receiving reverse influence.The approach addresses the unexplored effect of semantic optimization on appearance and geometric expression in Semantic-NeRF.Source: §1 · p. 2
• SNI-SLAM is a dense RGB-D NeRF-based semantic SLAM system for real-time 3D semantic segmentation, with hierarchical semantic encoding and feature-level optimization.The authors report state-of-the-art performance against existing NeRF-based SLAM in mapping, tracking, and semantic segmentation on Replica and ScanNet.Source: §1 · p. 2

2. Related Work

• Explicit semantic SLAM methods such as SLAM++ [36] and Kimera represent semantic scenes, but require substantial storage and offer insufficient detail.The paper turns to neural implicit representation for high-fidelity semantic SLAM with minimal storage.Source: §2 · p. 2
• The SNI-SLAM overview shows RGB-D input and modality features fused by cross-attention, followed by feature supervision, decoding, mapping, and pose optimization.Interpolated scene features support feature-loss construction and generation of RGB, depth, and semantics.Source: Figure 2 · p. 3
• Prior neural implicit SLAM systems model color and geometry, while NIDS-SLAM [11] does not integrate semantics with geometry and appearance.The paper proposes cross-attention fusion of all three feature types to improve mapping, tracking, and semantic segmentation.Source: §2 · p. 3

3.1. Cross-Attention based Feature Fusion

• Geometry features derived from sampled 3D coordinates, pretrained semantic features, and environment-specific appearance features provide the inputs for cross-modal fusion.The semantic features come from a pretrained segmentation network, while an updated appearance MLP transforms them into environment-specific features.Source: §3.1 · p. 4
• Cross-attention combines geometry, appearance, and semantic features, while concatenating the enhanced appearance feature with its original preserves fine-grained detail.Geometry guides attention to produce fused semantic features; the resulting multimodal representation supports more accurate feature representation.Source: §3.1 · p. 4

3.2. Hierarchical Semantic Mapping

• Coarse-to-fine semantic features capture scene layout and finer details, and the paper reports that this hierarchy improves implicit semantic modeling.Features from two spatial resolutions are concatenated for each coordinate; feature planes save storage space compared with voxel grids.Source: §3.2 · p. 4, Figure 3 · p. 4
• A collaborative decoder uses geometry, appearance, and semantic features to produce SDF, RGB, and semantic values for rendering color, depth, and semantics.Its one-way feature correlation is designed to support information exchange without mutual interference; the semantic decoder uses a 256-dimensional hidden layer.Source: §3.2 · pp. 4–5

3.3. Loss Functions

• Feature loss directly supervises intermediate features to guide updates to the feature planes and MLP, helping preserve important scene details.The extracted features serve as supervision for interpolated features from the feature planes.Source: §3.3 · p. 5
• The objective combines free-space, truncated-region, semantic, feature, color, and depth losses, with the complete loss defined as a weighted sum.Semantic loss uses cross-entropy, and its rendering gradient is detached to prevent interference with geometry and appearance optimization.Source: §3.3 · p. 5

4.1. Experimental Results

• Up to 32% relative increase in tracking accuracy is reported on Replica, alongside the highest reconstruction accuracy among compared NeRF-based SLAM methods.Table 1 averages results over eight Replica scenes, with five independent runs per scene.Source: Table 1 · p. 6, §4.1 · p. 7
• Qualitative reconstructions show more accurate detailed geometry and higher completion, especially in areas with limited observations.The comparison highlights narrow object details and edges; the paper attributes fine-grained reconstruction to combining appearance, geometry, and semantic representations.Source: Figure 4 · p. 6, §4.1 · p. 7
• SNI-SLAM outperforms NIDS-SLAM on Replica segmentation, with up to 10% increase on mIoU, and improves ScanNet tracking accuracy by 10%.The ScanNet comparison uses tracking RMSE; the Replica semantic comparison evaluates four scenes with ground-truth labels.Source: §4.1 · p. 7, Table 3 · p. 7, Table 2 · p. 7
• On TUM RGB-D, which lacks semantic ground truth, SNI-SLAM uses DEVA-generated 2D labels and reports up to 41% improvement over other NeRF-based SLAM methods.Table 4 reports tracking RMSE on five TUM RGB-D scenes.Source: §4.1 · p. 7, Table 4 · p. 7, §4 · p. 7

4.2. Runtime Analysis

• Runtime analysis reports semantic mapping with only a slight runtime increase and similar parameter counts versus existing NeRF-based SLAM methods; SNI-SLAM is faster than NIDS-SLAM with half its parameters.Table 5 compares runtime and memory on Replica.Source: Table 5 · p. 7, §4.2 · p. 8

4.3. Ablation Study

• Coarse-to-fine hierarchical semantic mapping improves semantic mapping and tracking accuracy by representing overall categories alongside local semantic details.The ablation describes single-layer representations as prone to mis-segmenting large areas such as walls.Source: §4.3 · p. 8, Table 6 · p. 8
• Feature loss helps preserve small objects and edge-region details, and its geometry, appearance, and semantic feature supervision can reduce boundary mis-segmentation.The ablation contrasts feature-level supervision with RGB, depth, and semantic losses that supervise information limited to one dimension.Source: §4.3 · p. 8, Figure 5 · p. 8
• Cross-attention feature fusion lets appearance cues guide semantic features, helping distinguish a TV screen from its background when segmentation is mistaken.The paper describes this as leveraging complementary geometry, appearance, and semantic features.Source: §4.3 · p. 8, Figure 5 · p. 8

• SNI-SLAM uses neural implicit representation to improve dense visual mapping and tracking accuracy while providing semantic mapping of the whole scene.Source: §5 · p. 8
• Cross-attention lets appearance, geometry, and semantic features potentially promote one another through cross-learning.Source: §5 · p. 8
• Coarse-to-fine semantic representation models scene semantics at multiple levels while preserving overall precision and intricate details.Source: §5 · p. 8
• The decoder fuses interpolated features from feature planes to produce more accurate decoding results.Source: §5 · p. 8