PaperlayerView exact revision on arXiv

Source-linked AI summary

All-St3R SLAM: Real-time RGB Dynamic SLAM with Pixel-wise Motion Masking

Yewon Kim, Paul Hongsuck Seo, Sang Chul Ahn

arXiv:2610.16143v1cs.CVcs.RO

TL;DR

Dynamic SLAM approaches face over-removal and high computational costs. This framework compares DUSt3R ego-motion with All-Tracker point tracking to mask independently moving pixels, and reports faster dense dynamic SLAM with competitive trajectory accuracy. Its reported limitations include residual map corruption in some slow-object scenarios and tracking failure with excessive subsampling.

  • Problem

    Existing removal-based approaches can over-remove, generalize poorly to unseen categories, and incur computational overhead that restricts real-time deployment.

    Source: §I · p. 1

  • Method

    The system combines DUSt3R ego-motion predictions with All-Tracker point tracking and masks pixels whose motions disagree, without object-level segmentation or detection.

    Source: §I · p. 1, C. Dynamic Masking · p. 4, §VI · p. 8

  • Results

    The system significantly outperforms dense dynamic SLAM systems in speed while maintaining competitive trajectory accuracy; on TUM static sequences, average ATE is 0.8 cm higher than MASt3R-SLAM (3.0 vs 3.8 cm).

    Source: B. Speed with High Tracking Accuracy · p. 5, §VI · p. 8

  • Takeaways & Limitations

    Pixel-wise masking preserves static regions within moving objects and generalizes to unseen object categories, reducing over-removal compared with object-level masking.

    Source: §I · p. 1, §VI · p. 8

    Slow objects can leave unmasked map residuals after a sudden camera-motion keyframe change, while larger subsampling can reduce static overlap enough to cause tracking failure.

    Source: §V · p. 8

Abstract

from arXiv · show

SLAM (Simultaneous Localization and Mapping) is a core technology in autonomous systems that enables them to estimate their own position and build a map of unknown environments simultaneously. SLAM systems in dynamic environments are of great practical importance. However, existing dynamic SLAM approaches still have several limitations like over-removal and high computational costs. To address these limitations, we propose a real-time RGB Dynamic SLAM framework that precisely removes moving objects at the pixel level without relying on predefined labels or segmentation models. Our system combines pixel motion predicted by ego-motion using DUSt3R with point tracking from All-Tracker, removing only actually moving pixels by analyzing discrepancies between the two motion sources. Unlike object-level masking methods, our approach eliminates only currently moving regions, allowing temporarily static parts to be utilized for tracking. Operating at approximately 10 FPS, the system achieves real-time performance on standard 30 FPS video streams by subsampling every 3 frames. Relying solely on RGB inputs, this significantly improves upon state-of-the-art dense dynamic SLAM approaches (0.5-3 FPS), presenting a highly practical solution.

I. INTRODUCTION

Dynamic SLAM must handle moving objects, but existing approaches trade dense coverage and generalization against over-removal and computational cost. This work compares ego-motion predictions with point tracking to mask moving pixels without segmentation or detection models.

I. INTRODUCTION

  • Existing removal methods trade lightweight but sparse geometry against object-level or hybrid methods with heavy priors, over-removal, limited generalization, and computational overhead.The cited examples include discarding parked cars, restricting generalization to unseen categories, and limiting real-time deployment.Source: §I · p. 1
  • The system identifies dynamic regions per pixel by comparing DUSt3R ego-motion predictions with All-Tracker point tracking, without relying on object-level segmentation or detection.Large motion discrepancies identify dynamic movement; the approach uses RGB inputs.Source: §I · pp. 1–2, Figure 1 · p. 1
  • Label-free geometric discrepancy masking preserves static pixels within partially moving objects and generalizes to unseen objects, reducing over-removal.Unlike object-level masking, it removes only currently moving regions.Source: §I · p. 1
  • Progressively accumulating motion inconsistencies across frames preserves more tracking correspondences and supports robust tracking in dynamic scenes.This contrasts with single-step removal, which risks discarding useful correspondences.Source: §I · pp. 1–2
  • Approximately 10 FPS is achieved with pairwise two-frame All-Tracker inference and lightweight geometric filtering; subsampling every 3 frames supports standard 30 FPS streams.The system uses monocular RGB inputs and avoids sequential tracking overhead without sacrificing tracking quality.Source: §I · p. 2

II. RELATED WORKS

Prior dynamic SLAM work includes object tracking and several removal strategies, each with trade-offs in computation, density, and generalization. The proposed framework builds on MASt3R-SLAM, using DUSt3R geometry and All-Tracker correspondences for dynamic RGB SLAM.

II. RELATED WORKS

  • Dynamic SLAM approaches either jointly track cameras and moving objects or remove dynamic outliers; this work focuses on geometric, object-level, and hybrid removal methods.Joint tracking can provide scene-level understanding but carries substantial computational cost and struggles with nonrigid or unstructured motion.Source: §II · p. 2
  • Geometric removal uses view-motion inconsistencies and is lightweight and training-free, but early feature-based methods are mostly sparse.Examples include epipolar geometry, point-tracking residuals, and depth disagreement.Source: §II · p. 2
  • Object-level masking uses segmentation, detection, or semantic embeddings, but may discard stationary instances, incur high computation, and fail to generalize beyond training categories.DynaSLAM typically operates at 1–2 FPS.Source: §II · p. 2
  • Hybrid methods combine object-level proposals with geometric verification, but cited systems retain substantial computational bottlenecks.DOG-SLAM verifies detected candidates geometrically, while WildGS-SLAM uses semantic embeddings and photometric-geometric consistency.Source: §II · p. 2

III. METHODOLOGY

  • The proposed real-time monocular dense SLAM framework builds on MASt3R-SLAM, replacing its geometric backbone with DUSt3R and adding pairwise All-Tracker correspondences for geometry-driven masking.The combination supplies dense image-level tracking alongside geometry without segmentation or detection models.Source: §III · p. 2

A. System Overview

Each incoming RGB frame is paired with a reference keyframe and processed through dense geometry and tracking, confidence refinement, pixel-wise dynamic masking, and pose tracking with map updates.

A. System Overview

  • The pipeline pairs each incoming Frame with a historically significant Keyframe and its canonical keyframe point cloud, then processes the pair through four sequential stages.Each keyframe maintains an independent point cloud; a new keyframe also triggers loop-closure detection and global pose-graph optimization.Source: A. System Overview · pp. 2–3
  • DUSt3R produces aligned dense point clouds, geometric matches, and confidence scores, while All-Tracker estimates dense image-level point tracking with confidence and visibility scores.Both methods process the current-frame/keyframe image pair.Source: A. System Overview · p. 2
  • Confidence reweighting uses 2D–3D consistency, high-frequency texture, and depth-boundary filtering to suppress noisy geometry and emphasize discriminative anchors.The refined correspondences feed subsequent tracking stages.Source: A. System Overview · p. 2
  • Pixel-wise masking compares ego-motion predictions with All-Tracker tracking and progressively filters independent motions while retaining valid static points within moving entities.The masking accumulates frame by frame.Source: A. System Overview · p. 2
  • Weighted optimization estimates the frame’s Sim(3) pose from dynamic-free correspondences; a frame becomes a new keyframe when valid matches fall below a threshold.New keyframes support loop closure and global pose-graph optimization.Source: A. System Overview · pp. 2–3

B. Confidence Reweighting

Confidence reweighting fuses geometric matches with point tracking and adjusts correspondence weights before pose optimization. It also downweights unreliable depth-boundary pixels and boosts textured regions.

B. Confidence Reweighting

  • DUSt3R geometric matches and All-Tracker point tracks are checked for pixel consistency, and verified matches receive greater confidence weight for pose optimization.The consistency threshold is 2 pixels, and verified matches receive a confidence multiplier of 1.2.Source: B. Confidence Reweighting · p. 3
  • A normalized depth-map Laplacian identifies flying pixels near depth discontinuities, whose confidence is reduced to limit noisy boundary geometry in pose estimation.This filtering targets artifacts that degrade correspondence reliability.Source: B. Confidence Reweighting · p. 3
  • An RGB-image Laplacian boosts confidence for pixels with strong edge responses, prioritizing high-frequency texture as traceable visual features.The operator is applied directly to both keyframe and current-frame images.Source: B. Confidence Reweighting · p. 3

C. Dynamic Masking

Dynamic pixels are identified by comparing camera-induced motion with observed point tracking, then masked progressively across frames. The method also uses frame subsampling to make slow motion easier to detect.

C. Dynamic Masking

  • Under a static-world assumption, DUSt3R pointmaps and PnP-RANSAC estimate relative camera pose for predicting pixel motion.The pose estimate is described as robust to circular dependence between ego-motion estimation and dynamic detection.Source: C. Dynamic Masking · pp. 3–4
  • All-Tracker supplies observed pixel displacement, while ego-motion predicts the motion expected for static scene points.Independent object motion can therefore create a discrepancy between the predicted and tracked flows.Source: C. Dynamic Masking · p. 4
  • Pixels whose flow discrepancy exceeds seven times its median absolute deviation are marked dynamic, preserving static parts such as a moving person’s torso.Large discrepancies indicate independent motion; static points should have matching predicted and tracked flows.Source: C. Dynamic Masking · p. 4
  • Dynamic masks are unioned frame by frame until a new keyframe is selected, retaining regions previously identified as moving.The accumulation prevents transient static observations of moving objects from corrupting the map.Source: C. Dynamic Masking · p. 4
  • A longer temporal baseline from frame subsampling magnifies slow-object pixel displacements, helping distinguish them from static background.The passage says this improves real-time performance as well as dynamic-region removal.Source: C. Dynamic Masking · p. 4

D. Lightweight Two-Frame All-Tracker

The tracker is reconfigured for lightweight pairwise inference to reduce latency and memory use while retaining fine-grained displacement tracking.

D. Lightweight Two-Frame All-Tracker

IV. EXPERIMENTS

In the crowd sequence, pixel-wise masking maintained tracking without relocalization, unlike SAM2 object-level masking, which discarded static regions. Experiments used three-frame subsampling on a single RTX 4090 GPU.

IV. EXPERIMENTS

  • Experiments used a single NVIDIA RTX 4090 GPU, images resized to a maximum dimension of 512 pixels, and three-frame subsampling.Source: §IV · p. 4

A. Robust tracking stability

  • The tracking-stability evaluation focuses on continuous online pose estimation in dynamic scenes, rather than long-term trajectory accuracy.Source: A. Robust tracking stability · pp. 4–5
  • Table I reports runtime in FPS and ATE RMSE in centimeters.Source: Table I · p. 5
  • Three relocalizations occurred within 20 keyframes with SAM2 object-level masking, while pixel-wise masking maintained tracking without relocalization.The pixel-wise method preserved static components as geometric anchors; the SAM2 mask discarded valid regions such as a person’s torso.Source: A. Robust tracking stability · p. 5, Figure 4 · p. 5

B. Speed with High Tracking Accuracy

The system runs at roughly 10 FPS with tracking accuracy comparable to WildGS-SLAM, while maintaining competitive trajectory accuracy in dynamic and static settings. Qualitative results show pixel-wise masking preserves static regions during occlusion and produces clean maps.

B. Speed with High Tracking Accuracy

  • ∼11.24 FPS on Wild-SLAM and ∼10.13 FPS on Bonn, versus WildGS-SLAM's roughly 0.5 FPS (Full) to 2 FPS (Fast), with comparable tracking accuracy.The reported peak GPU memory usages are 7.65 GB on Wild-SLAM and 9.46 GB on Bonn.Source: B. Speed with High Tracking Accuracy · p. 5
  • 2.46 cm Average ATE on Bonn Dynamic sequences gives the method second-best overall performance, close to WildGS-SLAM's 2.31 cm.Source: B. Speed with High Tracking Accuracy · p. 5

C. Dynamic Masking Analysis

B. Speed with High Tracking Accuracy

  • 2.27 cm average ATE on TUM RGB-D Dynamic at ∼10.18 FPS and 9.74 GB peak GPU memory; MonoGS and MonST3R-SW report 26.4 cm and 20.8 cm, respectively.The paper describes those methods as suffering catastrophic failures or large drift in these highly dynamic scenarios.Source: B. Speed with High Tracking Accuracy · p. 5
  • 3.8 cm average ATE on TUM static sequences is 0.8 cm higher than MASt3R-SLAM's 3.0 cm, while reconstruction quality is visually comparable with occasional scene improvements.The authors conclude that dynamic handling does not significantly degrade foundational tracking stability.Source: B. Speed with High Tracking Accuracy · p. 5

C. Dynamic Masking Analysis

  • WildGS-SLAM[8] uncertainty maps derived from finetuned DINOv2 features extracted at 1/14 of original resolution often appear blurry and coarse, while the proposed dynamic mask is sharp at full resolution.Source: C. Dynamic Masking Analysis · p. 5

D. Generalizable Dynamic Mask

Motion-cue masking generalizes to the ANYmal quadruped, and qualitative results show sharp masks and clean reconstructions. In crowded dynamic sequences, the full pipeline is more robust than the tested component-removal variants, while static-scene reconstruction remains visually comparable to MASt3R-SLAM.

D. Generalizable Dynamic Mask

E. Ablation Study

  • The Bonn dataset's Crowd and Crowd2 sequences show increased trajectory error for both ablations, while the full method remains robust.The comparison is between All-St3R SLAM, w/o Conf Reweighting, and w/o All-Tracker.Source: E. Ablation Study · p. 6
  • On the static freiburg1 room scene, DUSt3R with All-Tracker shows superior geometric alignment to MASt3R despite lacking its heavy matching head.The passage describes this as a qualitative reconstruction comparison.Source: Figure 8 · p. 7
  • The Bonn mask comparison shows a sharp, high-resolution pixel-wise mask with better boundary adherence than WildGS-SLAM's blurry uncertainty map.Source: Figure 9 · p. 7
  • All-Tracker uses image-level appearance cues for dense correspondences, avoiding geometric-proximity false matches that can arise with DUSt3R's Mgeo in crowded or occluded scenes.Mgeo can link distinct moving entities or dynamic points to the static background when their 3D points are spatially close.Source: E. Ablation Study · p. 7
  • The Bonn reconstruction removes the moving person to produce clean geometry, while live tracking aligns the current frame with the canonical map and masks the person.Source: Figure 11 · p. 7
  • The full pipeline can be comparable or slightly lower in performance when no dynamic motion is present, as confidence reweighting may focus pose alignment on specific regions and masking may remove useful static points.Source: Figure 12 · p. 7

V. DISCUSSION & LIMITATIONS

The discussion identifies limits involving metric scale, slow-moving objects and keyframe timing, larger frame gaps, and DUSt3R’s occasional reconstruction of moving objects as static background.

V. DISCUSSION & LIMITATIONS

  • DUSt3R’s lack of absolute metric scale means metric-scale applications need scale alignment or an IMU metric prior to anchor the initial scale.Source: §V · p. 7
  • Slow-moving objects may leave map-corrupting residuals if a camera-motion change creates a keyframe before masking finishes; nearby loop closure is the stated exception.The residuals otherwise remain in the map.Source: §V · pp. 7–8
  • Increasing the subsampling factor can reduce overlapping static background through larger baseline changes, ultimately causing camera-tracking failure.Source: §V · p. 8
  • When DUSt3R reconstructs a moving object as a flat texture on the static background, insufficient geometric separation can prevent its motion from being masked.Source: §V · p. 8

VI. CONCLUSIONS

The conclusion presents a real-time monocular dense SLAM framework combining DUSt3R geometry and lightweight All-Tracker, with pixel-wise motion discrepancy masking. It reports faster operation than existing dense dynamic SLAM systems while maintaining competitive trajectory accuracy.

VI. CONCLUSIONS

  • The framework combines DUSt3R’s dense geometry with a lightweight All-Tracker version for monocular dense SLAM without computationally expensive priors or depth sensors.Source: §VI · p. 8
  • Pixel-wise motion discrepancy preserves static regions within moving objects and generalizes to unseen object categories.Source: §VI · p. 8
  • The system significantly outperforms existing dense dynamic SLAM systems in speed while maintaining competitive trajectory accuracy.Source: §VI · p. 8
Loading 2610.16143v1…