PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM

Hengyi Wang, Jingwen Wang, Lourdes Agapito

arXiv:2304.14377v1cs.CV

TL;DR

Neural RGB-D SLAM must reconcile accurate, coherent surface reconstruction with real-time camera tracking, but coordinate and parametric representations each have important limitations. Co-SLAM combines one-blob coordinate encoding, sparse hash-grid features, depth-guided sampling, and global bundle adjustment. It achieves high-fidelity reconstruction, accurate tracking, plausible hole filling, efficient memory use, and real-time operation across several datasets.

  • Problem

    Coordinate-based neural SLAM offers coherence but converges slowly, whereas parametric encodings are fast but struggle with smoothness and hole filling.

    Source: §1 · p. 2

  • Method

    Co-SLAM jointly represents scenes with one-blob coordinate encoding and sparse hash-grid features, and performs global bundle adjustment using rays from all past keyframes.

    Source: §1 · p. 2

  • Results

    Co-SLAM achieves state-of-the-art camera tracking and 3D reconstruction while running at 15-17Hz on Replica and Synthetic RGB-D and 12-13Hz on ScanNet and TUM.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The joint representation supports high-fidelity maps, accurate tracking, plausible hole filling, and efficient memory use in dense real-time neural RGB-D SLAM.

    Source: §5 · p. 8

Abstract

from arXiv · show

We present Co-SLAM, a neural RGB-D SLAM system based on a hybrid representation, that performs robust camera tracking and high-fidelity surface reconstruction in real time. Co-SLAM represents the scene as a multi-resolution hash-grid to exploit its high convergence speed and ability to represent high-frequency local features. In addition, Co-SLAM incorporates one-blob encoding, to encourage surface coherence and completion in unobserved areas. This joint parametric-coordinate encoding enables real-time and robust performance by bringing the best of both worlds: fast convergence and surface hole filling. Moreover, our ray sampling strategy allows Co-SLAM to perform global bundle adjustment over all keyframes instead of requiring keyframe selection to maintain a small number of active keyframes as competing neural SLAM approaches do. Experimental results show that Co-SLAM runs at 10-17Hz and achieves state-of-the-art scene reconstruction results, and competitive tracking performance in various datasets and benchmarks (ScanNet, TUM, Replica, Synthetic RGBD). Project page: https://hengyiwang.github.io/projects/CoSLAM

1. Introduction

Neural RGB-D SLAM seeks robust camera tracking and dense, high-fidelity reconstruction while balancing learned scene coherence against real-time computation. Co-SLAM combines coordinate and sparse parametric encodings with global bundle adjustment to target this balance.

  • Real-time RGB-D SLAM must jointly track camera motion and reconstruct dense surfaces from sensor measurements.Source: §1 · p. 1
  • Coordinate-based representations capture high-frequency geometry and coherence but require long training, limiting real-time neural SLAM.Sequential systems may sparsify ray sampling and reduce tracking iterations, causing oversmoothed reconstructions and potential tracking errors.Source: §1 · pp. 1–2
  • Parametric embeddings converge orders of magnitude faster and represent local detail, but lack smoothness and struggle to fill unobserved regions.NICE-SLAM exemplifies a multi-resolution feature-grid approach that can capture local detail but cannot perform plausible hole filling.Source: §1 · p. 2
  • Co-SLAM jointly uses one-blob coordinate encoding and hash-grid sparse features to combine coherence priors, fast optimization, local detail, and improved hole filling.Source: §1 · p. 2
  • Global bundle adjustment samples rays from all past keyframes, improving pose-estimation robustness while avoiding optimization over only a small selected subset.Co-SLAM reports requiring a fraction of NICE-SLAM’s iterations for similar errors and operating at 15-17Hz on Replica and Synthetic RGB-D and 12-13Hz on ScanNet and TUM.Source: §1 · p. 2

2. Related Work

Prior SLAM systems combine geometric mapping with camera tracking, while neural implicit methods trade between expressive coordinate-based representations and faster parametric encodings. Neural SLAM systems such as iMAP and NICE-SLAM address this trade-off with different representations and efficiency strategies.

  • Traditional dense visual SLAM tracks camera poses and incrementally builds maps using depth and color, with later systems improving scalability through efficient data structures.Source: §2 · pp. 2–3
  • Neural implicit representations encode scene geometry and appearance in network parameters, with coordinate-based MLPs offering expressive reconstruction but lengthy training.Source: §2 · p. 3
  • Parametric encodings increase parameter size to accelerate training, motivating sparse structures such as octrees, tri-planes, hash grids, and sparse voxel grids.Source: §2 · p. 3
  • iMAP uses an MLP for joint tracking and mapping, while NICE-SLAM uses a multi-level feature grid to reduce computational overhead and improve scalability.iMAP reports 10 Hz tracking and 2 Hz mapping, whereas NICE-SLAM’s local feature-grid updates fail to achieve plausible hole filling.Source: §2 · p. 3

3. Method

Co-SLAM jointly optimizes a hybrid neural scene representation and camera poses for real-time RGB-D tracking and reconstruction. It combines multi-resolution hash-grid features with coordinate encoding, depth-guided sampling, smoothness regularization, and global bundle adjustment over stored keyframe pixels.

3.1. Joint Coordinate and Parametric Encoding

  • Co-SLAM maps world coordinates to RGB and truncated signed distance values using shallow MLP decoders with a shared multi-resolution hash-grid representation.The geometry decoder predicts SDF values and features, while a color decoder predicts RGB values.Source: §3 · p. 3, §3.1 · p. 3
  • One-blob encoding is injected into the hash-grid to provide fast convergence, efficient memory use, and hole filling for online SLAM.The hybrid representation combines coordinate-based smoothness and coherence with sparse parametric local detail.Source: §3.1 · p. 3

3.2. Depth and Color Rendering

  • Depth-guided sampling supplements uniform ray samples with near-surface samples around valid depth measurements, avoiding slower importance sampling.For valid depth rays, additional samples are drawn within [d − ds, d + ds].Source: §3.2 · p. 4

3.3. Tracking and Bundle Adjustment

  • Tracking and mapping minimize color, depth, SDF, and feature-smoothness losses to align rendered predictions with RGB-D observations and regularize the scene.Approximate SDF supervision is applied within the truncation region, while free-space loss constrains samples farther from the surface.Source: §3.3 · p. 4
  • Camera poses are initialized with a constant-speed motion model and then optimized per frame using sampled current-frame pixels.The pose is represented as a camera-to-world transformation parameterized by ξt.Source: §3.3 · p. 5
  • Global bundle adjustment jointly optimizes the scene representation and camera poses using rays sampled from a large global keyframe pixel set rather than selected full keyframes.Co-SLAM stores around 5% of pixels per keyframe, samples Ng rays globally, and alternates scene and pose optimization.Source: §3 · p. 3, Figure 3 · p. 4, §3.3 · p. 5

4. Experiments

Co-SLAM is evaluated across synthetic and real RGB-D datasets for reconstruction, tracking, runtime, memory, and ablations. Results show high-quality completion and detail, competitive or improved tracking, real-time operation, and benefits from joint encoding and global bundle adjustment.

Experimental setup

  • Co-SLAM is evaluated on Replica, NeuralRGBD, ScanNet, and TUM RGB-D using reconstruction, tracking, runtime, and memory metrics.The evaluation includes 8 Replica scenes, 7 NeuralRGBD scenes, 6 ScanNet scenes, and 3 TUM scenes.Source: §4.1 · p. 5

Reconstruction results

  • 15 FPS on Synthetic RGB-D accompanies fine-detail capture and complete, smooth reconstructions despite noisy depth.The method recovers structures such as wine bottles and chair legs, while baselines are less detailed, noisier, or unable to fill holes.Source: §4.2 · p. 6

Tracking results

  • 6−12 Hz on ScanNet accompanies better tracking than NICE-SLAM with fewer tracking and mapping iterations.ScanNet reconstructions also show smoother results and finer details.Source: §4.2 · p. 6, Table 3 · p. 6
  • 13 FPS on TUM RGB-D provides competitive tracking, while 6.7 FPS with more tracking iterations gives the best tracking among neural SLAM methods.Classic SLAM methods still outperform Co-SLAM on tracking, but Co-SLAM improves reconstruction fidelity and completeness.Source: §4.2 · pp. 6–7, Table 4 · p. 6

Runtime and memory

  • Above 15Hz is achieved in the default setting, while challenging ScanNet and TUM scenarios run at 5−13Hz with lower memory than NICE-SLAM.Further hash-table compression still lets Co-SLAM outperform iMAP while operating in real time and maintaining accurate reconstruction.Source: §4.3 · pp. 7–8

Ablations

  • The full joint encoding produces higher accuracy and better completion than either one-blob or hash encoding alone.The combined representation remains more robust than a hash-based feature grid without coordinate encoding, including under hash-table compression.Source: §4.4 · p. 8
  • Global bundle adjustment achieves the lowest average ATE of 8.75 cm, compared with 9.54 cm for GBA-10, 9.69 cm for LBA, and 16.81 cm without BA.Sampling rays from all keyframes is best overall even when strategies use the same total number of rays.Source: §4.4 · p. 8

5. Conclusion

Co-SLAM combines joint coordinate and parametric encoding with global bundle adjustment for real-time RGB-D SLAM. It achieves high-fidelity mapping, accurate tracking, plausible hole filling, and efficient memory use, but remains dependent on RGB-D input quality.

Conclusion

  • The system combines joint coordinate and parametric encoding with global bundle adjustment for dense real-time neural RGB-D SLAM.The representation uses tiny MLPs and targets joint mapping and camera-pose optimization.Source: §5 · p. 8
  • Co-SLAM achieves high-fidelity mapping, accurate tracking, plausible hole filling, and efficient memory use.Source: §5 · p. 8

Limitations

  • Co-SLAM is sensitive to illumination changes and inaccurate depth measurements because it relies on RGB-D sensor inputs.The paper also identifies information-guided pixel sampling and loop closure as possible improvements.Source: §5 · p. 8

Supplementary Material Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for

The supplementary material is associated with the Co-SLAM paper and its authors from University College London. The supplied metadata identifies the arXiv version as dated 27 April 2023.

Supplementary material

1. Implementation details

Co-SLAM uses dataset-specific settings for real-time tracking, mapping, bundle adjustment, and scene representation. Its evaluation combines mesh culling with 3D and 2D reconstruction metrics, including trajectory comparisons with and without alignment.

Default setting

  • Co-SLAM uses 1024 tracking pixels, 10 tracking iterations, 2048 mapping pixels, and bundle adjustment every 5 frames.Each camera ray uses 32 regular and 11 depth-guided samples; first-frame mapping uses 200 iterations.Source: §1.1 · p. 11

Dataset-specific settings

  • ScanNet uses 4cm finest resolution, 96 regular samples, 21 depth-guided samples, and λsmooth = 1e−3.Source: §1.1 · p. 11
  • TUM uses 20 bundle-adjustment iterations, a 5cm truncation distance, and a camera-parameter tracking learning rate of 1e−2.Source: §1.1 · p. 11

Mesh culling

  • The proposed culling strategy combines frustum and occlusion criteria with virtual-view simulation to remove artifacts while preserving completeness.Frustum-only culling leaves artifacts inside scene bounds, whereas frustum-plus-occlusion can remove occluded regions.Source: §1.2 · pp. 11–12, Figure 1 · p. 11

Evaluation metrics

  • Reconstruction quality is evaluated using Accuracy, Completion, Completion Ratio, and Depth L1 after uniformly sampling 200000 points from each mesh.Depth L1 additionally uses 1000 virtual views sampled inside a room cube.Source: §1.3 · p. 12

Tracking evaluation

  • ScanNet tracking reports ATE RMSE both with and without trajectory alignment, complementing the reconstruction evaluation protocol.Source: Table 3 · p. 12, Figure 2 · p. 12

2. Additional Experimental Results

Additional experiments show consistently strong reconstruction and tracking across synthetic and real-world scenes, while ablations examine memory, smoothness, and global bundle-adjustment choices. Co-SLAM also achieves comparable reconstruction with substantially faster training in batch-mode optimization.

Synthetic datasets

Real-world scenes

  • Co-SLAM achieves smoother, higher-quality reconstruction than NICE-SLAM on an apartment sequence in 40 minutes versus 10 hours.Source: §2.2 · p. 12

ScanNet camera tracking

  • Co-SLAM achieves overall better and more robust ScanNet tracking when trajectories are evaluated both with and without alignment.Without alignment, its estimated trajectory is associated with less distorted reconstruction.Source: §2.3 · p. 13

More ablation studies

  • One hash grid achieves similar ATE and reconstruction quality to separate color and geometry grids while reducing computational cost and memory storage.Source: §2.4 · p. 13, Table 6 · p. 14
  • Global bundle adjustment with pose optimization improves pose-estimation performance and robustness even when sampled ray batches have limited overlap.Source: §2.4 · pp. 13–14, Table 8 · p. 14

Batch-mode optimisation

  • In batch-mode optimization, the joint encoding achieves competitive reconstruction performance with significantly faster training than coordinate- or parametric-encoding alternatives.Source: §2.5 · p. 14, Table 7 · p. 14

Related papers

Loading 2304.14377v1…