Source-linked AI summary
DROID-SLAM in the Wild
Moyang Li, Zihan Zhu, Marc Pollefeys, Daniel Barath
TL;DR
Dynamic objects and clutter challenge SLAM systems that assume static scenes, depend on predefined motion priors, or require reliable geometric maps. DROID-W incorporates multi-view feature-based uncertainty optimization into differentiable bundle adjustment for camera tracking and scene reconstruction. It reports robust performance across challenging real-world scenarios while maintaining approximately 10 FPS, with initialization uncertainty estimation limited by unreliable early pose estimates.
Problem
Dynamic and non-rigid objects disrupt camera tracking and 3D reconstruction, while existing dynamic approaches remain limited by predefined motion priors or unreliable geometric mapping.
Method
DROID-W iteratively optimizes per-pixel dynamic uncertainty, camera poses, and scene geometry within differentiable bundle adjustment using multi-view feature similarity.
Results
DROID-W achieves state-of-the-art camera tracking and scene geometry reconstruction across challenging real-world scenarios while running at approximately 10 FPS.
Takeaways & Limitations
Uncertainty optimization enables robust camera tracking and accurate geometric reconstruction in challenging real-world dynamic scenes.
Takeaways & Limitations
Uncertainty optimization can be inaccurate during SLAM initialization because frame-to-frame alignment is unreliable when pose estimates are still uncertain.
Abstract
from arXiv · showhide
We present a robust, real-time RGB SLAM system that handles dynamic environments by leveraging differentiable Uncertainty-aware Bundle Adjustment. Traditional SLAM methods typically assume static scenes, leading to tracking failures in the presence of motion. Recent dynamic SLAM approaches attempt to address this challenge using predefined dynamic priors or uncertainty-aware mapping, but they remain limited when confronted with unknown dynamic objects or highly cluttered scenes where geometric mapping becomes unreliable. In contrast, our method estimates per-pixel uncertainty by exploiting multi-view visual feature inconsistency, enabling robust tracking and reconstruction even in real-world environments. The proposed system achieves state-of-the-art camera poses and scene geometry in cluttered dynamic scenarios while running in real time at around 10 FPS. Code and datasets are available at https://github.com/MoyangLi00/DROID-W.git.
1. Introduction
Reliable SLAM remains difficult in real-world environments because dynamic and non-rigid objects disrupt camera tracking and 3D reconstruction. DROID-W addresses these limitations with uncertainty optimization that avoids predefined motion priors and dependence on high-quality geometric mapping.
- Dynamic and non-rigid objects compromise pose estimation and 3D reconstruction, limiting SLAM robustness in practice.
- Many existing SLAM methods assume static environments, causing errors in camera tracking and scene reconstruction.
- Methods that detect or segment moving objects rely heavily on predefined dynamic-object knowledge, limiting robustness in diverse real-world scenes.
- Prior uncertainty-aware methods estimate pixel-wise uncertainty from DINO features but depend on perfectly static neural implicit or Gaussian Splatting maps.
- DROID-W updates dynamic uncertainty, camera poses, and scene geometry within differentiable bundle adjustment using multi-view feature similarity.
- DROID-W achieves state-of-the-art camera tracking and scene reconstruction in challenging real-world environments while running at approximately 10 FPS.
2. Related Works
Prior dynamic SLAM systems use residuals, object priors, segmentation, or scene representations to handle motion, but these strategies can fail for undefined objects or unreliable mapping. DROID-W instead uses multi-view visual feature similarity within an iterative SLAM pipeline to estimate uncertainty without coupling it to geometric mapping.
- Traditional Visual SLAM: Traditional visual SLAM methods assume static environments, allowing feature mismatches from object motion to degrade tracking and mapping.
- Dynamic filtering: Object detection and segmentation methods explicitly filter dynamic regions, but their effectiveness depends on recognizing moving objects and relevant classes.
- NeRF- and GS-based SLAM: Dynamic NeRF- and Gaussian Splatting-based systems often use semantic masks, yet undefined or unseen object classes remain difficult to handle.
- NeRF- and GS-based SLAM: Uncertainty optimization tied to scene representation degrades when geometric mapping struggles in complex environments.
- DROID-W: DROID-W estimates dynamic uncertainty from visual feature similarity between frames rather than relying on high-quality geometric mapping or predefined motion priors.
- Feed-forward Approaches: Feed-forward reconstruction methods can estimate convincing geometry but often struggle with accurate camera trajectories and metrically consistent structure over longer sequences.
- DROID-W: DROID-W alternates pose-depth refinement with uncertainty optimization, weighting reprojection residuals by per-pixel uncertainty and using monocular depth regularization.
3. Proposed Method
DROID-W adapts DROID-SLAM for dynamic, cluttered scenes by jointly refining poses, scene geometry, and per-pixel uncertainty through differentiable uncertainty-aware bundle adjustment. It estimates uncertainty from multi-view feature inconsistency, regularizes it locally, and integrates the resulting optimization throughout incremental and global SLAM.
- Uncertainty-aware Bundle Adjustment: DROID-W introduces differentiable Uncertainty-aware Bundle Adjustment to jointly optimize camera poses, depth, and per-pixel dynamic uncertainty.The system adapts DROID-SLAM for cluttered real-world RGB sequences and uses uncertainty to handle dynamic objects.
- DROID-SLAM Preliminaries: DROID-SLAM maintains camera poses and inverse depths, constructs a co-visibility frame graph, and iteratively refines them by minimizing dense correspondence residuals.The bundle-adjustment objective weights residuals using a confidence map and applies Gauss-Newton optimization to pose and disparity updates.
- Uncertainty-aware Bundle Adjustment: Per-pixel dynamic uncertainty downweights inconsistent correspondences and reduces the influence of high residuals caused by dynamic objects during optimization.The method incorporates uncertainty into a Mahalanobis distance and alternates pose-depth refinement with uncertainty optimization to avoid jointly optimizing all variables with Gauss-Newton.
- Dynamic Uncertainty Update: DROID-W estimates uncertainty from bidirectional multi-view DINOv2 feature similarity rather than reprojection residuals, which can become unreliable under large dynamic motion.Corresponding features are obtained through rigid-motion correspondences and bilinear interpolation; inconsistent regions are assigned higher uncertainty.
- Uncertainty Regularization: A local affine mapping with Softplus regularizes uncertainty spatially, while a logarithmic prior prevents the trivial solution of unbounded uncertainty.The uncertainty is parameterized as u = Softplus(θ · F), and a bias term prevents the prior loss from becoming negative.
- Overall SLAM System: The full system uses Metric3D monodepth for depth regularization, processes keyframes incrementally with sliding-window bundle adjustment, and freezes uncertainty during global bundle adjustment.Initialization accumulates 12 keyframes with sufficient motion before frontend tracking and subsequent global refinement.
4. Experiments
Experiments evaluate DROID-W across indoor, outdoor, highly dynamic, and cluttered benchmarks using tracking, reconstruction, runtime, qualitative, and ablation analyses. The method remains accurate and robust across challenging settings while operating in real time.
- Datasets and metrics: The evaluation covers Bonn RGB-D Dynamic, TUM RGB-D, DyCheck, DROID-W, and six challenging YouTube videos, using ATE for camera tracking.DROID-W contains seven outdoor sequences, while YouTube videos add diverse motion and cluttered scenes.
- Camera tracking: Our approach consistently achieves the best or second-best tracking results on TUM RGB-D and outperforms all baselines on average.On Bonn, it achieves the best camera tracking accuracy across all baselines; on DyCheck, it remains stable and accurate where prior methods often fail.
- Camera tracking: Our method significantly outperforms DROID-SLAM on high-dynamic sequences by handling motion-induced inconsistencies while remaining competitive on low-dynamic scenes.WildGS-SLAM suffers performance drops and near-complete failures when mapping and uncertainty estimation become unreliable.
- Runtime: 10 FPS is the approximate real-time speed of the system, which is 40× faster than WildGS-SLAM but slightly slower than DROID-SLAM.The additional cost comes from monocular depth estimation and DINOv2 feature extraction.
- Qualitative comparisons: Our method produces spatially coherent uncertainty maps that delineate dynamic regions and maintain stable confidence in static areas across challenging scenarios.WildGS-SLAM produces erroneous uncertainty near moving objects and severely incorrect predictions on difficult sequences.
- 3D reconstruction: Our method yields geometrically accurate and temporally consistent point clouds in challenging outdoor videos, avoiding the scale drift, erroneous geometry, and noisy distractors seen with DROID-SLAM.WildGS-SLAM struggles to reconstruct Gaussian maps under these conditions.
- Ablation study: The full system consistently outperforms ablated variants, while removing affine mapping or weight decay degrades uncertainty stability and performance.Direct uncertainty updates introduce temporal and spatial inconsistencies; removing affine-mapping regularization causes instability on some scenes.
5. Conclusion
The paper concludes that uncertainty optimization within differentiable bundle adjustment supports robust tracking and accurate reconstruction in challenging real-world dynamic scenes. It also identifies initialization as a limitation because unreliable early poses can produce inaccurate uncertainty estimates.
- Conclusion: DROID-W optimizes dynamic uncertainty within differentiable bundle adjustment using multi-view feature similarity for robust tracking and accurate geometric reconstruction.The conclusion reports effectiveness across challenging real-world scenarios where prior methods often struggle.
- Limitations: The method’s uncertainty optimization relies on frame-to-frame alignment, which can make uncertainty estimation inaccurate during SLAM initialization when pose estimates are unreliable.The paper suggests incorporating reconstruction priors to improve initialization robustness.
7. Dataset
The paper introduces DROID-W to evaluate dynamic SLAM under complex, unconstrained outdoor conditions. Its sequences combine long trajectories, high scene dynamics, clutter, motion blur, view-dependent effects, and low dynamic range.
- Dataset: DROID-W is an outdoor dataset of seven Downtown sequences captured with a Livox Mid-360 LiDAR rigidly mounted to an RGB camera.It provides 1200×1600 RGB frames, camera poses, synchronized IMU and LiDAR measurements, and RGB recording at 20 FPS.
- Dataset characteristics: The dataset includes long camera trajectories, many simultaneously moving objects, and challenging conditions such as motion blur, strong view-dependent effects, clutter, and low dynamic range.These characteristics are intended to represent complex real-world scenarios beyond limited indoor dynamic-SLAM benchmarks.
8. Uncertainty Optimization and Jacobians
This section derives the uncertainty energy and its Jacobians for differentiable optimization, including frame roles across connected edges.
- Equation (11) defines the uncertainty energy function used in the optimization.
- The subsequent derivation obtains Jacobians for differentiating the uncertainty objective.
- Bilinear interpolation supplies feature and uncertainty values at corresponding locations through a structured interpolation-weight matrix.
- The final Jacobians include the derivative of the warped correspondence with respect to θ.
- Frame l is the reference frame for edges (l, m) and the target frame for edges (k, l).
9. Additional Experiments
Additional experiments evaluate uncertainty estimation, reconstruction quality, ablations, and runtime across challenging real-world sequences. The results show robust uncertainty filtering, coherent static geometry, and approximately 10 FPS operation.
- 9.1. Uncertainty Estimation: Frame-to-frame feature alignment makes uncertainty estimation more robust than WildGS-SLAM in visually complex, in-the-wild environments.
- 9.1. Uncertainty Estimation: The method handles reflections, shadows, small dynamic objects, severe motion blur, and low dynamic range during uncertainty estimation.
- 9.1. Uncertainty Estimation: Uncertainty optimization delineates uncertain regions while maintaining high confidence in static areas.
- 9.1. Uncertainty Estimation: Multi-view feature similarity lets the system use stable regions for tracking and integrate inconsistency across a local frame-graph window.
- 9.1. Uncertainty Estimation: The system assigns highest uncertainty to the keyframe with the largest mirror-induced appearance change.
- 9.2. Point Cloud Reconstruction: DROID-SLAM reconstructs one corridor as two structures, whereas the proposed method produces coherent geometry and accurate poses under dynamic distractors.
- 9.2. Point Cloud Reconstruction: Static reconstructions remain geometrically consistent with static regions in dynamic point clouds, indicating effective suppression of dynamic or uncertain regions.
- 9.3. Ablation Studies: The prior regularization term prevents the trivial solution u →∞; without it, uncertainties become uniformly large.