PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Patch Visual SLAM

Lahav Lipson, Zachary Teed, Jia Deng

arXiv:2408.01654v1cs.CV

TL;DR

Deep SLAM systems can be costly in memory and struggle to maintain consistent runtime on a single GPU. DPV-SLAM extends DPVO with loop closure for monocular SLAM, and the authors report accuracy comparable to or better than the state of the art on several datasets while using less compute and running faster.

  • Problem

    Deep-network SLAM can require substantial GPU memory, and storing dense features for global optimization makes memory use grow with video length.

    Source: §1 · pp. 1–2

  • Method

    DPV-SLAM extends DPVO to monocular SLAM with proximity-based loop closure and an optional classical loop-closure backend.

    Source: §1 · p. 2, §3.3 · p. 8

  • Results

    DPV-SLAM is comparable to or better than the state of the art on several datasets while running faster and using less compute.

    Source: §6 · p. 14

  • Takeaways & Limitations

    DPV-SLAM is presented as an efficient, robust monocular SLAM system that maintains a relatively consistent frame rate using a single GPU.

    Source: §6 · p. 14

    DPV-SLAM requires a GPU, provides only sparse 3D reconstruction, and limits global bundle adjustment to 1000 key frames because its cost scales quadratically with pose variables.

    Source: §5 · p. 14

Abstract

from arXiv · show

Recent work in visual SLAM has shown the effectiveness of using deep network backbones. Despite excellent accuracy, however, such approaches are often expensive to run or do not generalize well zero-shot. Their runtime can also fluctuate wildly while their frontend and backend fight for access to GPU resources. To address these problems, we introduce Deep Patch Visual (DPV) SLAM, a method for monocular visual SLAM on a single GPU. DPV-SLAM maintains a high minimum framerate and small memory overhead (5-7G) compared to existing deep SLAM systems. On real-world datasets, DPV-SLAM runs at 1x-4x real-time framerates. We achieve comparable accuracy to DROID-SLAM on EuRoC and TartanAir while running 2.5x faster using a fraction of the memory. DPV-SLAM is an extension to the DPVO visual odometry system; its code can be found in the same repository: https://github.com/princeton-vl/DPVO

1 Introduction

Deep SLAM systems can be costly in GPU memory and unstable in runtime because frontend and backend operations compete for GPU access. DPV-SLAM extends DPVO with loop closure designed for a single GPU, aiming to retain accurate, efficient monocular SLAM across domains.

1 Introduction

  • Deep-network SLAM can require 24G of GPU memory, while separate frontend and backend CUDA operations on one GPU run sequentially and can drop from ≈30hz to <1hz.Dense feature maps retained for global optimization also cause GPU memory use to grow quickly with video length.Source: §1 · pp. 1–2
  • DPV-SLAM extends DPVO to full SLAM with proximity-based loop closure and a separate classical loop-closure mechanism.Its shared scene graph mixes odometry and low-cost loop-closure factors; CUDA-accelerated block-sparse bundle adjustment supports global optimization, while classical image retrieval and pose-graph optimization run on the CPU.Source: §1 · pp. 2–3
  • DPV-SLAM performs similarly to DROID-SLAM on EuRoC, achieves 4x lower error than DPVO there (0.105→0.024), and outperforms DROID-SLAM and DPVO on KITTI by significant margins.Source: §1 · p. 2
  • On EuRoC, DPV-SLAM achieves 4x lower error than DPVO (0.105→0.024), performs similarly to DROID-SLAM, and is reported to run 2.5x faster, though the cited excerpt does not name the speed baseline.Source: §1 · p. 2
  • The introduction reports zero catastrophic failures across tested settings and says DPV-SLAM performs well across domains without retraining.The comparison is against methods reporting both indoor and outdoor results without per-domain retraining.Source: §1 · p. 2

2 Related Work

Related work frames zero-shot cross-domain generalization as a longstanding SLAM challenge and distinguishes DPV-SLAM's evaluation focus from systems specialized for particular settings. It also situates DPVO's sparse-flow efficiency and its challenges in extending odometry to full SLAM.

2 Related Work

  • Zero-shot cross-domain generalization remains a challenge; this work uses synthetic-only training and reports better generalization and/or runtime.The authors distinguish this setting from in-domain systems developed with a particular test setting in mind, and do not claim to outperform KITTI-specialized systems.Source: §2 · p. 3
  • DPV-SLAM uses mid-term loop closure, or optionally both mid-term and long-term loop closure.Mid-term closure detects loops using current pose and depth estimates and updates them with bundle adjustment; long-term closure uses visual place recognition and pose-graph optimization.Source: §2 · p. 4
  • DPVO's sparse optical flow yields memory and runtime savings that can be spent elsewhere in the design, allowing accuracy similar to DROID-SLAM's frontend at lower cost and faster inference.The approach is harder to adapt to full SLAM because of its large per-frame storage requirement, and DPVO shares DROID-SLAM's outdoor performance issues.Source: §2 · p. 4
  • DPVO's design insight is that sparse-flow savings can offset initial accuracy loss when invested in other design choices.SLAM systems offer accuracy-speed-memory trade-offs through choices such as keypoints, optimization windows, image resolution, and factor-graph connectivity.Source: §2 · p. 4

3 Approach

DPV-SLAM extends DPVO’s patch-graph odometry with loop closure while preserving sparse optical-flow updates and bundle adjustment. It combines geometry-based proximity closure with an optional classical retrieval-based backend.

3.1 DPVO Preliminaries

  • DPVO predicts sparse optical-flow residuals for patch-to-frame edges, then optimizes patch depths and camera poses by bundle adjustment.Its patch graph represents patches with image coordinates and inverse depth; directed edges connect patches to frames.Source: §3.1 · pp. 4–5

3.2 Proximity Loop Closure

  • Proximity loop closure adds long-range patch-graph edges for nearby revisits, updates their optical flow, and performs global bundle adjustment on poses and depths.DPV-SLAM runs its PyTorch/CUDA operations in one process on one device; its CUDA block-sparse implementation supports efficient global optimization.Source: Figure 1 · p. 5, §3.2 · pp. 7–8

0.6 GB / 1K Frames

  • Uni-directional edges from stored old-frame patches to recent frames reduce dense-feature storage while retaining global constraints for optimization.The system stores patch features for previous time-steps, adding approximately 0.6G per 1K frames; CUDA block-sparse bundle adjustment handles global optimization.Source: Figure 2 · p. 6, §3.2 · pp. 7–8, Figure 3 · p. 7

3.3 Classical Loop Closure

  • The optional classical backend retrieves candidate image pairs, estimates relative 7DOF drift from triangulated 3D keypoints, and optimizes keyframe similarities in a pose graph.It uses off-the-shelf features and RANSAC+Umeyama for point-cloud alignment; CPU pose-graph optimization runs in parallel with negligible runtime overhead.Source: §3.3 · pp. 8–10, Figure 5 · p. 9

4 Experiments

DPV-SLAM is evaluated on four indoor and outdoor datasets, with comparisons focused on accuracy, speed, memory, and cross-domain performance. Results include comparable EuRoC accuracy to DROID-SLAM at higher speed and lower memory, plus strong TartanAir and KITTI performance.

Evaluation setup

  • Experiments use KITTI, TUM-RGBD, EuRoC-MAV, and TartanAir; DPV-SLAM and DPV-SLAM++ both use proximity loop closure, while DPV-SLAM++ adds classical loop closure.Experiments ran five times with median results, and timing measurements used an RTX-3090.Source: §4 · p. 10

KITTI

  • DPV-SLAM++ achieves the second-lowest average error among reported KITTI methods while averaging 39 FPS; proximity-only loop closure is insufficient under significant scale drift.The KITTI comparison also notes that DROID-SLAM performs exceptionally indoors but struggles on KITTI, while classical methods show the converse pattern.Source: §4 · p. 11, Table 2 · p. 12

TUM-RGBD

  • On TUM-RGBD, DPV-SLAM performs similarly to DROID-SLAM-based systems, with reported ATE ranges of 0.054–0.076 versus 0.035–0.114; those systems use 7.2–8.5G versus 4.0–6.0G.The benchmark covers indoor handheld motion and is challenging because of rolling shutter effects and motion blur.Source: §4 · p. 11, Table 1 · p. 12

EuRoC-MAV

  • 0.024 vs 0.022 ATE: DPV-SLAM performs marginally worse than DROID-SLAM on EuRoC, but runs 2.5x faster at 50 FPS versus 20 FPS and uses 5G versus 20G GPU memory.Compared with base DPVO, its EuRoC error is 0.105→0.024, with speed changing from 60 to 50 FPS and memory from 4G to 5G.Source: §4 · pp. 12–13, Table 3 · p. 13

TartanAir

  • 0.16 vs 0.24 ATE: DPV-SLAM outperforms DROID-SLAM on TartanAir; it runs at 27 FPS versus 8 FPS.The table reports results on the TartanAir monocular benchmark.Source: §4 · p. 13, Table 4 · p. 13

Runtime

  • 50-FPS and 39-FPS average speeds on EuRoC and KITTI correspond to 2.5x and 3.9x real-time, respectively, with loop closure producing lower-speed modes.On EuRoC, speed drops from 58 to 42 FPS during loop closure; on KITTI, infrequent loop closure briefly dips below real-time to 5–10 FPS before returning to 40 FPS.Source: Figure 9 · p. 14

5 Limitations

DPV-SLAM requires a GPU and produces only a sparse 3D reconstruction. Its global bundle-adjustment cost grows quadratically with pose variables, so the method limits its range to 1000 key frames; outdoor scale drift and occasional false-positive retrievals also remain concerns.

Scope and scaling

  • Global bundle adjustment scales quadratically with pose variables, so its range is limited to 1000 key frames.DPV-SLAM also requires a GPU and provides only a sparse 3D reconstruction.Source: §5 · p. 14

Remaining failure modes and cost

  • Outdoor scale drift sometimes occurs, image retrieval can produce occasional false positives, and classical loop closure adds 2G of GPU memory overhead.The scale-drift caveat is shared with most monocular systems.Source: §5 · p. 14

6 Conclusion

DPV-SLAM is presented as an efficient monocular SLAM system that generalizes across domains. Across four datasets, it is reported as robust and comparable to or better than state of the art on several datasets while using less compute.

Conclusion

  • DPV-SLAM is reported as robust across domains and comparable to or better than state of the art on several datasets while using less compute.The evaluation covers EuRoC, TartanAir, TUM-RGBD, and KITTI.Source: §6 · p. 14

A Bundle-Adjustment Efficiency

Block-sparse bundle adjustment is more efficient for moderate-to-large patch graphs, while dense optimization is better for very small, highly connected graphs. DPV-SLAM switches between the implementations according to graph size.

A Bundle-Adjustment Efficiency

  • Block-sparse bundle adjustment is much more efficient for moderate-to-large patch graphs, while dense optimization is better for very small, highly connected graphs.In the block-sparse version, Cholesky decomposition of the pose block in the Schur complement is the memory and runtime bottleneck.Source: A Bundle-Adjustment Efficiency · p. 18
  • For the KITTI experiment, the number of depth variables and edges remains constant as the number of pose variables changes.Source: A Bundle-Adjustment Efficiency · p. 18
  • The implementation switches between dense and block-sparse optimization on the fly depending on patch-graph size.Source: Figure 10 · p. 18
Loading 2408.01654v1…