PaperlayerView exact revision on arXiv

Source-linked AI summary

Pow3R-SLAM: Real-Time RGB-D SLAM with 3D Reconstruction Priors

Christopher Kolios, Ishaan Mehta, Sasa Janjic, Yeganeh Bahoo, Sajad Saeedi

arXiv:2609.38054v1cs.ROcs.CV

TL;DR

RGB-D SLAM must contend with sensor-depth holes and inconsistencies, yet omitting depth discards metric information. Pow3R-SLAM uses sensor depth as a prior for two-view pointmaps and for metric scale and backend geometry. Across the reported benchmarks, it improves accuracy and reconstruction while running faster than MASt3R-SLAM, though self-similar scenes remain challenging.

  • Problem

    Depth-based SLAM can inherit holes and boundary inconsistencies from sensor measurements, while systems that omit depth discard a key metric measurement.

    Source: §I · p. 1

  • Method

    Pow3R-SLAM uses sensor depth to condition Pow3R’s two-view pointmaps, set their metric scale, and anchor keyframe geometry in global optimization.

    Source: §I · p. 1

  • Results

    Across 24 TUM, 7-Scenes, and Replica sequences, Pow3R-SLAM runs 1.6× faster than MASt3R-SLAM, with 15% lower mean keyframe trajectory error and 30% lower Chamfer distance on 7-Scenes and Replica.

    Source: §VI · p. 8

  • Takeaways & Limitations

    Ablations identify depth conditioning as the main source of accuracy gains, and show that it remains effective with only 3% of depth pixels supplied to the network.

    Source: §VI · p. 8

    Repeating, self-similar structures are a challenging failure case: confident but incorrect correspondences can pull the backend toward a wrong solution.

    Source: §V · p. 8

Abstract

from arXiv · show

We present Pow3R-SLAM, a real-time RGB-D simultaneous localization and mapping (SLAM) system that uses Pow3R for tracking and mapping. Inspired by MASt3R-SLAM, a recent work on monocular SLAM using two-view 3D reconstruction priors, we extend the work to incorporate depth as a prior on the network's prediction, rather than as geometry to fuse. Where traditional RGB-D SLAM systems struggle with sparsity in the depth images, Pow3R utilizes the available depth to give a better-conditioned pointmap, while inferring the depths in empty regions from the two-view photometric, depth, and intrinsic data. Evaluated against MASt3R-SLAM following its protocol on 24 sequences from TUM, 7-Scenes, and Replica, Pow3R-SLAM runs 1.6x faster in wall time, has 15% lower mean trajectory error, a 3.1x lower unscaled error, and produces denser maps, with a 30% lower Chamfer distance. We also introduce a hybrid variant that runs 2.1x faster than MASt3R-SLAM at 25.3 frames per second (FPS), while maintaining improved tracking and mapping accuracy. Against ORB-SLAM3 in RGB-D mode, Pow3R-SLAM is more accurate on TUM, 7-Scenes, and ETH3D-SLAM, and completes every TUM sequence. While Pow3R-SLAM can struggle on a small set of self-similar scenes, its overall performance shows that adding depth as a prior for two-view 3D reconstruction SLAM can be beneficial. A project webpage is available at: https://ChrisKolios.github.io/Pow3R-SLAM , and code will be made open-source upon acceptance.

I. INTRODUCTION

RGB-D SLAM can inherit defects from sparse or inconsistent sensor depth, while ignoring depth discards a metric measurement. Pow3R-SLAM instead uses sensor depth as a prior for two-view pointmaps and builds tracking and mapping around Pow3R.

I. INTRODUCTION

  • Sensor-depth holes and boundary errors can contaminate fused maps, while systems that omit depth lose a key metric measurement.Depth sensors leave holes on some surfaces and can be inconsistent at object boundaries.Source: §I · p. 1
  • Pow3R can condition two-view geometry on depth and intrinsics, and infer depth where measurements are absent.The passage contrasts its pointmap reconstruction of a controller with a back-projected depth map that cannot fill it.Source: §I · p. 1
  • Sensor depth enters Pow3R-SLAM at three sites: network conditioning, pointmap scale fixing, and keyframe-geometry anchoring during global optimization.The map remains the network output, which fills holes and regularizes surfaces.Source: §I · p. 1
  • Pow3R-SLAM rebuilds loop retrieval and per-correspondence confidence from Pow3R features, and adds map-only keyframes and a hybrid ICP mode.The hybrid mode tracks alternate frames by ICP on sensor depth, halving network cost.Source: §I · pp. 1–2
  • The evaluation covers accuracy, reconstruction, runtime, component ablations, and sensitivity to depth quality against MASt3R-SLAM and ORB-SLAM3.The stated evaluation spans four benchmarks.Source: §I · p. 2

II. RELATED WORK

Prior SLAM methods use depth as geometry or supervision, while reconstruction-prior SLAM uses learned two-view or scene models; the paper extends this line with an RGB-D system. Its setup follows MASt3R-SLAM conventions and assumes calibrated pinhole imagery.

II. RELATED WORK

  • Existing RGB-D methods use depth for fusion, tracking, supervision, or dense optimization, while reconstruction-prior SLAM uses two-view or scene models for tracking and mapping.The authors identify depth as an input not previously investigated in reconstruction-prior-based SLAM.Source: §II · p. 2

III. METHODS

  • The method overview identifies sensor depth and hybrid ICP as components of Pow3R-SLAM.The passage points to Fig. 2 for the overview.Source: §III · p. 2

A. Preliminaries

  • Pow3R’s two-view inputs may include depth, intrinsics, and relative pose, but this RGB-D system supplies depth and intrinsics while solving for relative pose.The system maintains MASt3R-SLAM’s Sim(3) pose formulation.Source: A. Preliminaries · p. 3
  • The calibrated setup assumes a known pinhole camera, equal intrinsics across frames, and fixed calibration for registering depth to RGB.Images are undistorted first, and intrinsics are adjusted when images are resized for Pow3R.Source: A. Preliminaries · p. 3

B. Known Depth and Calibration

Pow3R-SLAM uses sensor depth in three distinct ways: it conditions the network, restores metric scale, and anchors geometry in backend optimization. The conditioning uses depth and camera-ray inputs; scale is estimated from valid depth ratios and bounded to reject outliers.

B. Known Depth and Calibration

  • Depth conditioning supplies normalized depth maps, validity masks, and dense camera-ray images to Pow3R’s first encoder block as patch tokens.The depth and ray inputs are cut into the same 16×16 patch grid as the image.Source: 1) cond · p. 3
  • Metric scale is estimated from valid sensor-to-predicted depth ratios, using their geometric mean when at least 50 pixels are valid.Ratios more than 10× from their median are clipped to that bound; fewer than 50 valid pixels leaves the pass unscaled.Source: 2) scale · p. 3
  • Before each backend solve, sensor depths replace pointmap depths along known rays wherever sensor measurements exist, acting as a soft constraint.The substituted depths are not clamped in the solution.Source: 3) anchor · p. 3

C. Pointmap Matching and Fusion

Pow3R-SLAM adopts projective matching and updates keyframe geometry using the most confident prediction at each pixel. Its keyframe-addition criterion follows MASt3R-SLAM, while retaining a single scale per forward pass avoids averaging scale error into the keyframe.

C. Pointmap Matching and Fusion

  • Projective matching follows MASt3R-SLAM, but Pow3R-SLAM omits per-pixel feature refinement because Pow3R has no such features.Matches are rejected when their 3D outlier distance exceeds 0.1 m.Source: C. Pointmap Matching and Fusion · p. 3
  • A new keyframe is added when the smaller matched-pixel fraction or distinct matched-pixel coverage fraction falls below 0.333.The method uses the same keyframe criteria as MASt3R-SLAM.Source: C. Pointmap Matching and Fusion · p. 3
  • Each keyframe pixel keeps its most confident prediction rather than a running weighted average, because each pass has its own scale estimate.Averaging predictions would average scale error into the keyframe.Source: C. Pointmap Matching and Fusion · p. 3

D. Tracking

Tracking estimates each incoming frame’s pose relative to the current keyframe from one network pass, then optimizes matched-point reprojection error. A lightly weighted log-depth term observes scale, while matching and factorization checks determine when to trigger relocalization.

D. Tracking

  • Each incoming frame is tracked against the current keyframe with one network pass that supplies matched points for relative-pose estimation.The incoming point is transported into keyframe coordinates and compared with the keyframe’s canonical point at the matched pixel.Source: D. Tracking · p. 3
  • With no per-match confidence, tracking sets q to 1 and rejects outliers through matching tests rather than a confidence gate.The cited tests include a ray-residual convergence check and 0.1 m 3D rejection.Source: D. Tracking · p. 4
  • Tracking minimizes reprojection error with a lightly weighted log-depth term that observes scale and prevents depth degeneracy under pure rotation.The relative pose is optimized with Gauss–Newton for up to 50 iterations.Source: D. Tracking · p. 4
  • Tracking declares a frame lost and triggers relocalization if fewer than 5% of pixels survive matching or the normal equations fail to factorize.A bidirectional edge is added to the previous keyframe when a new keyframe is added.Source: D. Tracking · p. 4

E. Loop Closure, Relocalization, and Backend

The system adapts MASt3R-SLAM’s loop retrieval and backend to Pow3R, adds a stricter relocalization acceptance rule, and anchors sensor depth during global optimization. Pow3R confidences are calibrated to preserve the existing match gate.

1) Loop Closure:

  • Loop closure uses an ASMK retrieval head refitted to Pow3R encoder tokens, then adds a loop edge when the minimum projective match fraction reaches 0.25.The retrieval pass is unconditioned and computed once per keyframe; the preceding-keyframe edge is always added.Source: 1) Loop Closure · p. 4

2) Relocalization:

  • Relocalization accepts a retrieved frame only when every candidate clears a 0.3 match-fraction floor, then initializes its pose from the first candidate and runs a backend solve.The retrieval database uses the same score threshold as MASt3R-SLAM.Source: 2) Relocalization · p. 4

3) Backend:

  • Before each backend solve, sensor depths replace pointmap depths along observed rays in a backend-only anchored copy; anchoring is disabled when scale is disabled.The first keyframe pose remains fixed, and each new-keyframe solve uses at most ten iterations.Source: 3) Backend · p. 4
  • Pow3R confidences are mapped to match MASt3R-SLAM’s confidence distribution, preserving its Qmn > 1.5 gate without threshold retuning or retraining.The two confidence factors downweight low-confidence geometry, while the gate carries most of the weight’s practical effect.Source: 3) Backend · p. 4

F. Map-Only Keyframes

Map-only keyframes decouple map density from the keyframe rate used for pose estimation. They add geometry to the final map without modifying trajectories.

F. Map-Only Keyframes

  • Frames with 0.333 ≤ ωk < 0.45 can be stored as map-only keyframes after at least five input frames, up to 200 per sequence.These keyframes carry geometry but no estimator state; their stored data include a rescaled pointmap, confidence, image, and relative pose.Source: F. Map-Only Keyframes · p. 4
  • At export, map-only pointmaps are recomposed with their keyframes’ final poses and unioned with keyframe maps, densifying the map without modifying trajectories.Backend corrections and loop closures therefore propagate to their exported world poses.Source: F. Map-Only Keyframes · p. 4

G. Hybrid ICP Variant

The hybrid variant replaces network tracking on alternate frames with depth-based point-to-plane ICP, with safeguards that return tracking to the network when the ICP pose or fit is inadequate. The supplied results passages identify evaluation scope and table conventions but do not provide benchmark values.

G. Hybrid ICP Variant

  • The hybrid variant replaces the network call on every second frame with point-to-plane ICP on sensor depth, associating valid depth pixels to the keyframe pointmap.ICP frames never become keyframes, enter the backend, or add map points.Source: G. Hybrid ICP Variant · p. 5
  • ICP uses a 2 cm Huber kernel and coarse-to-fine SE(3) optimization, falling back to network tracking when pose or fit checks fail.Fallback conditions include a non-finite pose, a finest-level cost increase above 5%, or fewer than 30% of associable pixels being inliers.Source: G. Hybrid ICP Variant · p. 5

IV. RESULTS

  • Pow3R-SLAM evaluates localization on TUM RGB-D, 7-Scenes, Replica, and ETH3D-SLAM, and geometry on 7-Scenes and Replica.7-Scenes depth and color are unregistered, so color images are resampled into the depth camera using a shared calibration that does not use ground truth or trajectory error.Source: §IV · p. 5

ATE ROOT-MEAN SQUARE ERROR (RMSE) (M) ON TUM RGB-D

CALIBRATED MONOCULAR RESULT AS REPORTED IN [7]. FRESH: OUR

IV. RESULTS

  • Runtime results use two exclusive repeats with half-range reported, and the single-threaded front end waits for the backend to drain.Non-blocking mode changes wall time by under 5%.Source: CALIBRATED MONOCULAR RESULT AS… · p. 5

A. Pose Estimation

Pow3R-SLAM has the lowest mean trajectory error on TUM in both modes and on registered 7-Scenes images. On 7-Scenes it beats MASt3R-SLAM on six of seven sequences, with stairs as the exception.

1) TUM RGB-D:

  • Pow3R-SLAM records the lowest mean trajectory error on TUM in both modes, with mean 0.027 vs. 0.030 m for MASt3R-SLAM; ORB-SLAM3 loses track on 3/9 sequences.All other systems complete all nine sequences.Source: 1) TUM RGB-D · p. 5

2) 7-Scenes:

  • With registered colour, Pow3R-SLAM has the lowest 7-Scenes mean ATE (0.044 m vs. 0.048 m for MASt3R-SLAM and 0.055 m for ORB-SLAM3), beating MASt3R-SLAM on 6/7 sequences.Stairs is the exception (0.044 vs. 0.016 m); registration lowered Pow3R-SLAM’s ATE from 0.065 to 0.044 m.Source: 2) 7-Scenes · pp. 5–6

3) Replica:

Across the evaluated scenes, Pow3R-SLAM improves reconstruction metrics over MASt3R-SLAM and lowers unscaled error, while Replica and ETH3D-SLAM results have distinct scope conditions. Its regular and hybrid modes also run faster than MASt3R-SLAM.

5) Per-frame and Unscaled Scoring:

3) Replica:

  • On Replica, Pow3R-SLAM nearly halves mean ATE vs. MASt3R-SLAM (0.009 vs. 0.015 m), while ORB-SLAM3 is most accurate (0.005 m).Replica’s rendered depth is complete and noise-free, strengthening conditioning and scale.Source: Table II · p. 6, 3) Replica · p. 6

4) ETH3D-SLAM:

  • On ETH3D-SLAM, Pow3R-SLAM wins 12/14 shared completed sequences and has lower median ATE (0.0135 vs. 0.0280 m), but does not complete two sequences.Hybrid mode completes the same 14 scenes and wins 12/14, with mean ATE 0.034 vs. 0.039 m for MASt3R-SLAM on those scenes.Source: Table II · p. 6, 4) ETH3D-SLAM · p. 6

C. Runtime

  • Regular Pow3R-SLAM takes 903 s vs. MASt3R-SLAM’s 1443 s (1.6× faster); hybrid takes 698 s (2.1× faster), averaging 25.3 FPS.Runtime is summed over the 24-scene panel.Source: Table II · p. 6, C. Runtime · pp. 6–7

B. Geometry Evaluation

  • Across 15 scenes, Pow3R-SLAM’s mean Chamfer is 3.1 vs. 4.3 cm (30% lower) and RMSE Chamfer is 4.1 vs. 6.9 cm (40% lower) than MASt3R-SLAM’s.Accuracy and completion improve on both 7-Scenes and Replica.Source: B. Geometry Evaluation · p. 6, Table III · p. 7

D. Ablations

Across benchmarks, Pow3R-SLAM improves runtime, trajectory accuracy, and map density over MASt3R-SLAM. Ablations identify depth conditioning as the main accuracy contributor, while self-similar scenes remain challenging.

D. Ablations

  • Depth conditioning lowers mean ATE from 0.083 m to 0.032 m; scale further lowers Sim(3) ATE to 0.026 m and unscaled SE(3) error to 0.034 m.Depth conditioning, not intrinsics, has the larger role; map-only keyframes leave trajectories identical while improving Chamfer distance from 3.27 to 3.05 cm with 4.4× the points.Source: D. Ablations · p. 7, Table IV · p. 8
  • With only 3% of depth pixels at the conditioning site, mean ATE is 0.026 m, while dense depth noise raises it to 0.609 m at σ = 0.4.The prior is robust to sparse depth, but not to dense, wrong depth; depth completion helped 7-Scenes, was within noise on TUM, and regressed Replica while adding compute cost.Source: D. Ablations · p. 7

V. LIMITATIONS

  • Pow3R-SLAM remains worse than MASt3R-SLAM on some stairs and repetitive or planar scenes, where incorrect but mutually consistent matches may pull the backend to a wrong solution.The authors hypothesize that repeating, self-similar structures cause these failures.Source: §V · p. 8

VI. CONCLUSIONS

  • 1.6× faster than MASt3R-SLAM, with 15% lower mean keyframe trajectory error, 3.1× lower per-frame trajectory error without scale alignment, and 30% lower Chamfer distance on 7-Scenes and Replica.The hybrid ICP variant reaches 25.3 FPS and a 2.1× speedup over MASt3R-SLAM.Source: §VI · p. 8
  • Depth conditioning is the main source of accuracy gains and scaling is essential for metric output; future work targets confidence gating, realistic sensor noise, and multi-view priors.The authors conclude that depth-conditioned pointmaps provide a practical basis for real-time SLAM with improved accuracy, map completeness, and runtime.Source: §VI · p. 8

Related papers

Loading 2609.38054v1…