Source-linked AI summary
Point-SLAM: Dense Neural Point Cloud-based SLAM
Erik Sandström, Yue Li, Luc Van Gool, Martin R. Oswald
TL;DR
Dense RGBD SLAM needs a scene representation that supports tracking and mapping without the redundancy of separate representations. Point-SLAM uses an adaptively growing neural point cloud and RGBD re-rendering optimization, achieving stronger or competitive performance across tracking, mapping, and rendering evaluations while balancing resource use.
Problem
Existing dense SLAM systems often use different representations for tracking and mapping, creating undesirable data redundancy and independence.
Method
Point-SLAM iteratively grows a neural point cloud with density adapted to input information and alternates mapping and tracking using an RGBD-based re-rendering loss.
Results
Point-SLAM substantially improves reconstruction and rendering accuracy while remaining competitive in tracking, runtime, and memory usage.
Takeaways & Limitations
A single data-driven neural point representation can support dense RGBD mapping and tracking while balancing scene accuracy with computational and memory demands.
Takeaways & Limitations
The system is more sensitive to motion blur and specularities, while its density adaptation and several hyperparameters remain heuristic rather than test-time adaptive.
Abstract
from arXiv · showhide
We propose a dense neural simultaneous localization and mapping (SLAM) approach for monocular RGBD input which anchors the features of a neural scene representation in a point cloud that is iteratively generated in an input-dependent data-driven manner. We demonstrate that both tracking and mapping can be performed with the same point-based neural scene representation by minimizing an RGBD-based re-rendering loss. In contrast to recent dense neural SLAM methods which anchor the scene features in a sparse grid, our point-based approach allows dynamically adapting the anchor point density to the information density of the input. This strategy reduces runtime and memory usage in regions with fewer details and dedicates higher point density to resolve fine details. Our approach performs either better or competitive to existing dense neural RGBD SLAM methods in tracking, mapping and rendering accuracy on the Replica, TUM-RGBD and ScanNet datasets. The source code is available at https://github.com/eriksandstroem/Point-SLAM.
1. Introduction
Dense visual SLAM supports applications requiring detailed scene maps, but conventional systems often separate tracking and mapping representations. Point-SLAM addresses this with a data-adaptive neural point representation that supports both tasks while balancing accuracy, runtime, and memory.
- Dense visual SLAM provides maps for AR, VR, navigation, collision detection, occlusion reasoning, and scene interpretation.
- Separate representations for tracking and dense mapping create data redundancy and independence between camera tracking and the estimated dense map.
- Point-SLAM anchors neural features in an iteratively growing point cloud and uses the same representation for mapping and tracking.
- Its dynamic point-density strategy adapts feature placement to input information density, trading reconstruction accuracy against computational and memory demands.
- The approach shows benefits across datasets in tracking, rendering, and mapping accuracy.
2. Related Work
Related work spans grid-, point-, and network-based scene representations for dense reconstruction and SLAM. Point-SLAM builds on point representations while adding neural implicit features, adaptive density, and volumetric rendering for online RGBD SLAM.
- Dense Visual SLAM and Mapping: Dense reconstruction methods commonly use TSDFs, voxel grids or hashing, octrees, point or surfel clouds, and learned scene representations.
- Scene Representations: Point sets offer flexible scene encoding but make neighborhood search harder because they lack an inherent connectivity structure.
- Scene Representations: Point-SLAM models scene space as unordered points with optimizable features, using neural implicit features and volumetric alpha compositing for rendering.
3. Method
Point-SLAM uses an adaptively generated neural point cloud as the sole dense RGBD SLAM representation, supporting tracking and mapping through neural rendering and re-rendering losses. Point density follows image information, while point features and decoder networks produce geometry and color for rendering.
- Neural Point Cloud Representation: The neural point cloud is incrementally expanded as new scene areas are explored and uses geometric and color feature descriptors anchored at 3D locations.Points are added from RGBD observations and the representation converges to a bounded set when no new scene parts are visited.
- Neural Point Cloud Representation: Image-gradient-dependent search radii allocate denser points to detailed regions and coarser points elsewhere, controlling compression and memory usage.Lower and upper radius bounds rl and ru determine the dynamic resolution, while intermediate gradients interpolate between them.
- Neural Point Cloud Representation: During point addition, pixels are sampled uniformly and from high-gradient regions, then unprojected; rays without nearby neighbors receive three depth-dependent points.The three points are centered at the measured depth and offset to model expected depth-camera noise.
- Rendering: Rendering samples five points around the sensor depth, decodes occupancy and color with MLPs, and aggregates them along rays using volume rendering.Unlike voxel-based frameworks that also sample empty space, this approach concentrates samples near the observed surface; NICE-SLAM uses 48 samples in the cited example.
- Rendering: For missing-depth pixels, ray marching provides hole filling within a bounded depth interval but cannot fill arbitrarily large holes reliably.The interval runs from 30cm to 1.2Dmax and uses 25 samples.
- Mapping: Mapping minimizes a re-rendering loss combining geometric and color L1 terms while optimizing point features and decoder parameters, with keyframes regularizing the process.Depth-only optimization initializes color optimization, after which color loss is included for the remaining 60% of iterations; an exposure MLP handles significant inter-frame exposure changes.
4. Experiments
Experiments evaluate Point-SLAM against dense neural RGBD SLAM baselines on Replica, TUM-RGBD, and ScanNet, covering reconstruction, tracking, rendering, runtime, and memory. Results show strong reconstruction, tracking, and rendering performance, with competitive runtime and dataset-dependent sensitivity to motion blur and specularities.
- Experimental setup: Experiments compare Point-SLAM with NICE-SLAM, Vox-Fusion, and ESLAM on Replica, plus evaluations on TUM-RGBD and ScanNet.The datasets include synthetic indoor scenes and real-world RGBD data.
- Reconstruction: Point-SLAM outperforms all compared methods on Replica geometric reconstruction metrics, with average depth L1 improvements of 85%, 82%, and 63% over NICE-SLAM, Vox-Fusion, and ESLAM.The authors attribute finer detail recovery and memory conservation to adaptive point density near surfaces and details.
- Tracking: Point-SLAM achieves better Replica tracking on average and outperforms existing dense neural RGBD methods on TUM-RGBD, while remaining competitive on ScanNet.The ScanNet evaluation uses exposure compensation, and the dataset is described as more complex because of motion blur and specularities.
- Rendering: Point-SLAM improves rendering over existing dense neural RGBD SLAM methods and produces more accurate high-frequency details in full-resolution renderings.The adaptive neural point-cloud density is reported to increase rendering fidelity by allocating more points to detailed regions.
- Efficiency: Point-SLAM’s memory usage falls between NICE-SLAM and Vox-Fusion while its runtime is competitive in the Replica Office 0 evaluation.Runtime measurements used an RTX 2080 Ti for Point-SLAM and an RTX 3090 for Vox-Fusion.
- Limitations: The authors identify sensitivity to motion blur and specularities, and propose optimizing point locations, learning density adaptation, and adapting hyperparameters at test time.These limitations are presented as future directions rather than evaluated fixes.
5. Conclusion
Point-SLAM uses a neural point cloud for both dense mapping and tracking, with data-driven feature anchoring and dynamic resolution balancing accuracy, memory, and compute.
- Point-SLAM uses a neural point cloud as the shared scene representation for dense mapping and tracking.
- Data-driven feature anchoring aligns features with surface locations and adjusts feature density to the input information density.
- The resulting system balances memory and compute usage with the accuracy of the estimated 3D scene representation.
- Experiments report higher reconstruction and rendering accuracy, with competitive tracking, runtime, and memory usage.
A. Videos
The supplementary material includes an introductory video describing Point-SLAM's method, key results, and online reconstruction process relative to NICE-SLAM and Vox-Fusion.
- The video introduces Point-SLAM's method and its most important results.
- It visualizes the online reconstruction process of Point-SLAM.
- The visualization compares Point-SLAM with NICE-SLAM and Vox-Fusion.
B. Method
The supplementary method section specifies the dynamic-resolution threshold, gradient-histogram basis, exposure-network architecture, and dataset-specific parameter configuration fields.
- Dynamic Resolution Strategy: The upper bound for the color gradient magnitude threshold is set to gu = 0.15 for all evaluated datasets.
- Dynamic Resolution Strategy: Figure 7 plots cumulative pixel histograms by gradient magnitude and search radius as a function of pixel gradient magnitude.
- Exposure Network Architecture: The exposure network uses one hidden layer with 128 neurons, an 8-dimensional latent input, and a 12-dimensional output reshaped into affine transformation parameters.
- Parameter Configurations: The parameter table records mapping frequency, keyframe-window size, and optimization iterations for the tested datasets.
C. Implementation Details
The implementation uses PyTorch and Adam, evaluates on GPUs with up to 12 GB memory, and varies tracking and optimization learning rates across datasets and stages.
- The pipeline is implemented in PyTorch 1.12 and Python 3.10.
- Training uses Adam with betas = (0.9, 0.999), eps = 1e-08, and zero weight decay.
- Results are gathered using Nvidia GPUs with a maximum memory of 12 GB.
- Tracking learning rates are 0.002 on Replica and TUM-RGBD and 0.0005 on ScanNet, while geometry and color optimization use separate rates.
D. Evaluation Metrics
The evaluation uses geometric reconstruction, tracking-trajectory, and depth-rendering metrics, with explicit thresholds and alignment procedures.
- Reconstruction: F-score measures reconstruction quality as the harmonic mean of mesh Precision and Recall.Precision and Recall use a distance threshold τ = 0.01 m.
- Reconstruction: Precision measures predicted-mesh points near the ground-truth mesh, while Recall measures ground-truth points near the predicted mesh.Both quantities use the same τ = 0.01 m threshold.
- Reconstruction: Meshes are aligned with iterative closest point before Precision and Recall are computed.
- Depth: Replica depth L1 error is reported for both rendered depth maps and reconstructed meshes after TSDF fusion and Marching cubes.
- Tracking: ATE RMSE measures translation error between estimated and ground-truth trajectories after alignment with Horn’s closed-form solution.
E. More Experiments
Additional experiments examine adaptive point density, adaptive mapping iterations, and qualitative performance across Replica, ScanNet, and TUM-RGBD.
- Dynamic Search Radius Visualization: Dynamic point density reduces the point count from 66K to 54K while preserving dense sampling in texture-rich regions.Sparsification occurs mainly in low-texture areas such as sofas and walls, while rich textures remain resolved.
- Adaptive Mapping Ablation: Mapping speed increases by a factor of four compared with the main-paper results when the adaptive lower bound is reduced from 0.95.
- Adaptive Mapping Ablation: Rendering, tracking, and reconstruction metrics remain virtually unchanged until the adaptive lower bound drops to 0.8.
- Additional ScanNet Results: On additional ScanNet scenes, Point-SLAM performs better on average than the baseline methods.The reported comparison uses ATE RMSE across three runs, with failed runs averaged over successful runs.
- TUM-RGBD and ScanNet Results: Qualitative TUM-RGBD results show high-quality renderings and textured and untextured meshes.
- Replica Results: Replica reconstructions are described as more precise on average, while adaptive density increases rendering fidelity by encoding more high-frequency details.