PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM

Hengyi Wang, Jingwen Wang, Lourdes Agapito

arXiv:2304.14377v1cs.CV

TL;DR

Neural RGB-D SLAM must reconcile accurate, coherent surface reconstruction with real-time camera tracking, but coordinate and parametric representations each have important limitations. Co-SLAM combines one-blob coordinate encoding, sparse hash-grid features, depth-guided sampling, and global bundle adjustment. It achieves high-fidelity reconstruction, accurate tracking, plausible hole filling, efficient memory use, and real-time operation across several datasets.

  • Problem

    Coordinate-based neural SLAM offers coherence but converges slowly, whereas parametric encodings are fast but struggle with smoothness and hole filling.

  • Method

    Co-SLAM jointly represents scenes with one-blob coordinate encoding and sparse hash-grid features, and performs global bundle adjustment using rays from all past keyframes.

  • Results

    Co-SLAM achieves state-of-the-art camera tracking and 3D reconstruction while running at 15-17Hz on Replica and Synthetic RGB-D and 12-13Hz on ScanNet and TUM.

  • Takeaways & Limitations

    The joint representation supports high-fidelity maps, accurate tracking, plausible hole filling, and efficient memory use in dense real-time neural RGB-D SLAM.

Abstract

from arXiv · show

We present Co-SLAM, a neural RGB-D SLAM system based on a hybrid representation, that performs robust camera tracking and high-fidelity surface reconstruction in real time. Co-SLAM represents the scene as a multi-resolution hash-grid to exploit its high convergence speed and ability to represent high-frequency local features. In addition, Co-SLAM incorporates one-blob encoding, to encourage surface coherence and completion in unobserved areas. This joint parametric-coordinate encoding enables real-time and robust performance by bringing the best of both worlds: fast convergence and surface hole filling. Moreover, our ray sampling strategy allows Co-SLAM to perform global bundle adjustment over all keyframes instead of requiring keyframe selection to maintain a small number of active keyframes as competing neural SLAM approaches do. Experimental results show that Co-SLAM runs at 10-17Hz and achieves state-of-the-art scene reconstruction results, and competitive tracking performance in various datasets and benchmarks (ScanNet, TUM, Replica, Synthetic RGBD). Project page: https://hengyiwang.github.io/projects/CoSLAM

1. Introduction

Neural RGB-D SLAM seeks robust camera tracking and dense, high-fidelity reconstruction while balancing learned scene coherence against real-time computation. Co-SLAM combines coordinate and sparse parametric encodings with global bundle adjustment to target this balance.

  • Real-time RGB-D SLAM must jointly track camera motion and reconstruct dense surfaces from sensor measurements.
  • Coordinate-based representations capture high-frequency geometry and coherence but require long training, limiting real-time neural SLAM.Sequential systems may sparsify ray sampling and reduce tracking iterations, causing oversmoothed reconstructions and potential tracking errors.
  • Parametric embeddings converge orders of magnitude faster and represent local detail, but lack smoothness and struggle to fill unobserved regions.NICE-SLAM exemplifies a multi-resolution feature-grid approach that can capture local detail but cannot perform plausible hole filling.
  • Co-SLAM jointly uses one-blob coordinate encoding and hash-grid sparse features to combine coherence priors, fast optimization, local detail, and improved hole filling.
  • Global bundle adjustment samples rays from all past keyframes, improving pose-estimation robustness while avoiding optimization over only a small selected subset.Co-SLAM reports requiring a fraction of NICE-SLAM’s iterations for similar errors and operating at 15-17Hz on Replica and Synthetic RGB-D and 12-13Hz on ScanNet and TUM.

2. Related Work

Prior SLAM systems combine geometric mapping with camera tracking, while neural implicit methods trade between expressive coordinate-based representations and faster parametric encodings. Neural SLAM systems such as iMAP and NICE-SLAM address this trade-off with different representations and efficiency strategies.

  • Traditional dense visual SLAM tracks camera poses and incrementally builds maps using depth and color, with later systems improving scalability through efficient data structures.
  • Neural implicit representations encode scene geometry and appearance in network parameters, with coordinate-based MLPs offering expressive reconstruction but lengthy training.
  • Parametric encodings increase parameter size to accelerate training, motivating sparse structures such as octrees, tri-planes, hash grids, and sparse voxel grids.
  • iMAP uses an MLP for joint tracking and mapping, while NICE-SLAM uses a multi-level feature grid to reduce computational overhead and improve scalability.iMAP reports 10 Hz tracking and 2 Hz mapping, whereas NICE-SLAM’s local feature-grid updates fail to achieve plausible hole filling.

3. Method

Co-SLAM jointly optimizes a hybrid neural scene representation and camera poses for real-time RGB-D tracking and reconstruction. It combines multi-resolution hash-grid features with coordinate encoding, depth-guided sampling, smoothness regularization, and global bundle adjustment over stored keyframe pixels.

  • 3.1. Joint Coordinate and Parametric Encoding: Co-SLAM maps world coordinates to RGB and truncated signed distance values using shallow MLP decoders with a shared multi-resolution hash-grid representation.The geometry decoder predicts SDF values and features, while a color decoder predicts RGB values.
  • 3.1. Joint Coordinate and Parametric Encoding: One-blob encoding is injected into the hash-grid to provide fast convergence, efficient memory use, and hole filling for online SLAM.The hybrid representation combines coordinate-based smoothness and coherence with sparse parametric local detail.
  • 3.2. Depth and Color Rendering: Depth-guided sampling supplements uniform ray samples with near-surface samples around valid depth measurements, avoiding slower importance sampling.For valid depth rays, additional samples are drawn within [d − ds, d + ds].
  • 3.3. Tracking and Bundle Adjustment: Tracking and mapping minimize color, depth, SDF, and feature-smoothness losses to align rendered predictions with RGB-D observations and regularize the scene.Approximate SDF supervision is applied within the truncation region, while free-space loss constrains samples farther from the surface.
  • 3.3. Tracking and Bundle Adjustment: Camera poses are initialized with a constant-speed motion model and then optimized per frame using sampled current-frame pixels.The pose is represented as a camera-to-world transformation parameterized by ξt.
  • 3.3. Tracking and Bundle Adjustment: Global bundle adjustment jointly optimizes the scene representation and camera poses using rays sampled from a large global keyframe pixel set rather than selected full keyframes.Co-SLAM stores around 5% of pixels per keyframe, samples Ng rays globally, and alternates scene and pose optimization.

4. Experiments

Co-SLAM is evaluated across synthetic and real RGB-D datasets for reconstruction, tracking, runtime, memory, and ablations. Results show high-quality completion and detail, competitive or improved tracking, real-time operation, and benefits from joint encoding and global bundle adjustment.

  • Experimental setup: Co-SLAM is evaluated on Replica, NeuralRGBD, ScanNet, and TUM RGB-D using reconstruction, tracking, runtime, and memory metrics.The evaluation includes 8 Replica scenes, 7 NeuralRGBD scenes, 6 ScanNet scenes, and 3 TUM scenes.
  • Reconstruction results: 15 FPS on Synthetic RGB-D accompanies fine-detail capture and complete, smooth reconstructions despite noisy depth.The method recovers structures such as wine bottles and chair legs, while baselines are less detailed, noisier, or unable to fill holes.
  • Tracking results: 6−12 Hz on ScanNet accompanies better tracking than NICE-SLAM with fewer tracking and mapping iterations.ScanNet reconstructions also show smoother results and finer details.
  • Tracking results: 13 FPS on TUM RGB-D provides competitive tracking, while 6.7 FPS with more tracking iterations gives the best tracking among neural SLAM methods.Classic SLAM methods still outperform Co-SLAM on tracking, but Co-SLAM improves reconstruction fidelity and completeness.
  • Runtime and memory: Above 15Hz is achieved in the default setting, while challenging ScanNet and TUM scenarios run at 5−13Hz with lower memory than NICE-SLAM.Further hash-table compression still lets Co-SLAM outperform iMAP while operating in real time and maintaining accurate reconstruction.
  • Ablations: The full joint encoding produces higher accuracy and better completion than either one-blob or hash encoding alone.The combined representation remains more robust than a hash-based feature grid without coordinate encoding, including under hash-table compression.
  • Ablations: Global bundle adjustment achieves the lowest average ATE of 8.75 cm, compared with 9.54 cm for GBA-10, 9.69 cm for LBA, and 16.81 cm without BA.Sampling rays from all keyframes is best overall even when strategies use the same total number of rays.

5. Conclusion

Co-SLAM combines joint coordinate and parametric encoding with global bundle adjustment for real-time RGB-D SLAM. It achieves high-fidelity mapping, accurate tracking, plausible hole filling, and efficient memory use, but remains dependent on RGB-D input quality.

  • Conclusion: The system combines joint coordinate and parametric encoding with global bundle adjustment for dense real-time neural RGB-D SLAM.The representation uses tiny MLPs and targets joint mapping and camera-pose optimization.
  • Conclusion: Co-SLAM achieves high-fidelity mapping, accurate tracking, plausible hole filling, and efficient memory use.
  • Limitations: Co-SLAM is sensitive to illumination changes and inaccurate depth measurements because it relies on RGB-D sensor inputs.The paper also identifies information-guided pixel sampling and loop closure as possible improvements.

Supplementary Material Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for

The supplementary material is associated with the Co-SLAM paper and its authors from University College London. The supplied metadata identifies the arXiv version as dated 27 April 2023.

  • Supplementary material: Co-SLAM is authored by Hengyi Wang, Jingwen Wang, and Lourdes Agapito at University College London.
  • Supplementary material: The paper is identified as arXiv:2304.14377v1, dated 27 Apr 2023.
  • Supplementary material: The listed contact addresses use the University College London domain.

1. Implementation details

Co-SLAM uses dataset-specific settings for real-time tracking, mapping, bundle adjustment, and scene representation. Its evaluation combines mesh culling with 3D and 2D reconstruction metrics, including trajectory comparisons with and without alignment.

  • Default setting: Co-SLAM uses 1024 tracking pixels, 10 tracking iterations, 2048 mapping pixels, and bundle adjustment every 5 frames.Each camera ray uses 32 regular and 11 depth-guided samples; first-frame mapping uses 200 iterations.
  • Dataset-specific settings: ScanNet uses 4cm finest resolution, 96 regular samples, 21 depth-guided samples, and λsmooth = 1e−3.
  • Dataset-specific settings: TUM uses 20 bundle-adjustment iterations, a 5cm truncation distance, and a camera-parameter tracking learning rate of 1e−2.
  • Mesh culling: The proposed culling strategy combines frustum and occlusion criteria with virtual-view simulation to remove artifacts while preserving completeness.Frustum-only culling leaves artifacts inside scene bounds, whereas frustum-plus-occlusion can remove occluded regions.
  • Evaluation metrics: Reconstruction quality is evaluated using Accuracy, Completion, Completion Ratio, and Depth L1 after uniformly sampling 200000 points from each mesh.Depth L1 additionally uses 1000 virtual views sampled inside a room cube.
  • Tracking evaluation: ScanNet tracking reports ATE RMSE both with and without trajectory alignment, complementing the reconstruction evaluation protocol.

2. Additional Experimental Results

Additional experiments show consistently strong reconstruction and tracking across synthetic and real-world scenes, while ablations examine memory, smoothness, and global bundle-adjustment choices. Co-SLAM also achieves comparable reconstruction with substantially faster training in batch-mode optimization.

  • Synthetic datasets: Co-SLAM consistently improves Completion and remains competitive in Accuracy across Replica and Synthetic RGB-D scenes.
  • Real-world scenes: Co-SLAM achieves smoother, higher-quality reconstruction than NICE-SLAM on an apartment sequence in 40 minutes versus 10 hours.
  • ScanNet camera tracking: Co-SLAM achieves overall better and more robust ScanNet tracking when trajectories are evaluated both with and without alignment.Without alignment, its estimated trajectory is associated with less distorted reconstruction.
  • More ablation studies: One hash grid achieves similar ATE and reconstruction quality to separate color and geometry grids while reducing computational cost and memory storage.
  • More ablation studies: Global bundle adjustment with pose optimization improves pose-estimation performance and robustness even when sampled ray batches have limited overlap.
  • Batch-mode optimisation: In batch-mode optimization, the joint encoding achieves competitive reconstruction performance with significantly faster training than coordinate- or parametric-encoding alternatives.
Loading 2304.14377v1…