Source-linked AI summary
DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras
Zachary Teed, Jia Deng
TL;DR
Visual SLAM must jointly build an environment map and localize the camera across monocular, stereo, and RGB-D imagery. DROID-SLAM addresses this with recurrent pose-and-depth updates through differentiable Dense Bundle Adjustment, achieving large benchmark gains, fewer catastrophic failures, and modality generalization from monocular training.
Problem
Visual SLAM requires accurate mapping and localization across monocular, stereo, and RGB-D camera recordings, while prior learned systems fall short of classical accuracy on common benchmarks.
Method
DROID-SLAM uses an end-to-end differentiable architecture with recurrent updates of camera poses and pixelwise depth through a Dense Bundle Adjustment layer.
Results
DROID-SLAM outperforms prior classical and learning-based SLAM systems by large margins on challenging benchmarks, with reported accuracy gains and substantially fewer catastrophic failures.
Takeaways & Limitations
A single model trained once on monocular synthetic data can use stereo and RGB-D inputs directly without retraining across four datasets and three modalities.
Takeaways & Limitations
Memory and resource requirements are the system’s biggest current limitation, with some long videos requiring a GPU with 24GB memory.
Abstract
from arXiv · showhide
We introduce DROID-SLAM, a new deep learning based SLAM system. DROID-SLAM consists of recurrent iterative updates of camera pose and pixelwise depth through a Dense Bundle Adjustment layer. DROID-SLAM is accurate, achieving large improvements over prior work, and robust, suffering from substantially fewer catastrophic failures. Despite training on monocular video, it can leverage stereo or RGB-D video to achieve improved performance at test time. The URL to our open source code is https://github.com/princeton-vl/DROID-SLAM.
1 Introduction
DROID-SLAM combines deep learning with recurrent, differentiable geometric optimization to address visual SLAM across camera modalities. It reports large accuracy gains, fewer catastrophic failures, and direct generalization from monocular training to stereo and RGB-D inputs.
- Results: 35% better AUC than second place is reported on ETH-3D, while the system tracks 30 of 32 RGB-D datasets versus 19 of 32 for the next best system.The AUC combines error and catastrophic-failure rate.
- Generalization: A single model trained once on synthetic monocular TartanAir data achieves results across four datasets and three modalities without retraining.The system can directly use stereo or RGB-D input at test time.
- Contribution: DROID-SLAM introduces an end-to-end differentiable architecture that combines classical optimization with deep networks for visual SLAM.Its design is presented as a new deep learning-based SLAM system.
- Method: The system recurrently updates camera poses and depth for an arbitrary number of frames, enabling joint global refinement for long trajectories and loop closures.This differs from two-frame optical-flow updates and is intended to minimize drift.
- Method: Dense Bundle Adjustment computes differentiable Gauss-Newton updates to camera poses and per-pixel depth using geometric compatibility with optical flow.The layer supports geometric constraints and allows monocular training to use stereo or RGB-D inputs without retraining.
- Evaluation: The paper evaluates DROID-SLAM across four datasets and three sensor modalities, including ablations of design decisions and hyperparameters.
2 Related Work
Related SLAM systems span indirect and direct formulations, while recent deep end-to-end methods often trade robustness for accuracy or lack large-scale capabilities. DROID-SLAM instead combines full-image processing with reprojection-error optimization and pixelwise depth variables.
- Modern SLAM jointly optimizes camera pose and a 3D map, with full Bundle Adjustment central to accuracy and adaptable to different sensors.
- Indirect methods match detected features and optimize reprojection error, whereas direct methods optimize photometric error over the image formation process.
- DROID-SLAM avoids feature detection and matching, uses the full image, and minimizes reprojection error rather than photometric error.The paper presents this as combining broader image information with an easier optimization objective.
- End-to-end learned SLAM methods often target small reconstructions and lack capabilities such as loop closure and global bundle adjustment.These limitations inhibit large-scale reconstruction according to the cited discussion.
- DeepFactors jointly optimizes pose and depth and supports short- and long-range loop closure, but optimizes a learned depth basis during inference.DROID-SLAM instead optimizes pixelwise depth, avoiding a representation tied to the training dataset.
3 Approach
DROID-SLAM jointly and recurrently refines camera poses and pixelwise inverse depths over a dynamic frame graph. A learned update operator uses visual correspondence and flow information, while Dense Bundle Adjustment converts predicted flow revisions into geometric pose and depth updates.
- Representation: The system represents each video with camera poses and pixelwise inverse depths that are iteratively updated during inference.The first two poses are fixed to remove gauge freedom, while depths remain free variables.
- Frame Graph: A dynamic frame graph connects overlapping views and adds long-range edges when the camera revisits mapped regions, enabling loop closure.Visibility is recomputed after pose or depth updates.
- Feature Extraction and Correlation: For each graph edge, dense feature correlations and flow-derived inputs provide visual similarity and motion information to the update network.Correlation volumes are organized into a four-level pyramid, and lookup uses bilinear interpolation around current correspondence coordinates.
- Update Operator: A convolutional GRU repeatedly predicts flow revisions, confidence weights, damping factors, and an inverse-depth upsampling mask from correlation, flow, and context features.Global context helps the network recognize and reject erroneous correspondences, including those caused by large moving objects.
- Iterative Refinement: Iterative applications of the update operator produce pose and depth sequences expected to converge toward a fixed-point reconstruction.Correspondences are recomputed from current pose and depth estimates at the start of each iteration.
- Dense Bundle Adjustment: The DBA layer maps predicted flow revisions across the frame graph into pose and pixelwise depth updates by minimizing weighted reprojection error.Gauss-Newton optimization with a Schur complement solves the updates efficiently, and the differentiable layer supports backpropagation during training.
4 Experiments
Experiments evaluate DROID-SLAM across datasets, sensor modalities, and competing classical and learned systems. The results show strong accuracy, robustness, generalization, and practical runtime, alongside substantial memory demands.
- Evaluation: Experiments compare DROID-SLAM with deep-learning and classical SLAM methods across diverse datasets and sensor modalities, emphasizing cross-dataset generalization.Accuracy is primarily evaluated using Absolute Trajectory Error (ATE).
- TartanAir: 62% lower error on the TartanAir monocular benchmark and 60% lower error on the stereo benchmark than the best prior competition results.The competing top submissions run 40x slower than real-time, while DROID-SLAM runs 16x faster than them.
- EuRoC: 2.2cm average ATE on EuRoC monocular sequences, with 82% lower error than zero-failure methods and 43% lower error than ORB-SLAM3 on shared successful sequences.The comparison with ORB-SLAM3 uses only the 10 of 11 sequences where ORB-SLAM3 succeeds.
- Robustness and accuracy: DROID-SLAM successfully tracks all 9 evaluated sequences while achieving 83% lower ATE than DeepFactors and 90% lower ATE than DeepV2D.The compared methods also succeed on all videos in this evaluation.
- ETH-3D: 30/32 ETH-3D RGB-D datasets are successfully tracked, compared with 19/32 for the next-best system, while DROID-SLAM ranks first on both train and test splits without finetuning.Datasets without image data are excluded from prediction submission.
- Runtime and resources: The system runs in real time at 20fps on EuRoC and 30fps on TUM-RGBD using two RTX 3090 GPUs and downsampled, skipped frames.The backend requires substantially more memory than the frontend; several datasets require a 24GB GPU.
5 Conclusion
The paper concludes that DROID-SLAM is an end-to-end neural architecture for visual SLAM. It is presented as accurate, robust, and versatile across monocular, stereo, and RGB-D video, outperforming prior work by large margins.
- DROID-SLAM is an end-to-end neural architecture for visual SLAM that supports monocular, stereo, and RGB-D video.
- The system is accurate, robust, and versatile, outperforming prior work by large margins on challenging benchmarks.
A Additional Results
Additional EuRoC results evaluate stereo SLAM using the network trained only on synthetic monocular video. Stereo input reduces trajectory error substantially relative to ORB-SLAM3.
- Stereo EuRoC: Stereo SLAM results are reported on the EuRoC datasets using ATE in meters.
- Stereo EuRoC: 71% lower average ATE than ORB-SLAM3 is achieved on EuRoC with stereo input.The network was trained on synthetic monocular video, and stereo recovers the camera trajectory up to scale.
B Ablations
Ablations examine sensor inputs, optimization scope, keyframe count, global context, and training with Dense Bundle Adjustment. They show benefits from stereo and global optimization and instability when DBA is omitted during training.
- Keyframes: Five keyframes are used in the experiments when evaluating tracking accuracy as a function of keyframe count.
- Visualizations: Validation visualizations include keyframe depth estimates, optical flow, and associated confidence weights.
- Input and optimization: Stereo video and global optimization both improve system performance, even though the network is trained only on monocular video.
- Bundle adjustment training: Training with the DBA layer is important because omitting DBA during training makes the SLAM system unstable and prone to failure.
- Network architecture: Global context in the GRU is evaluated through spatial pooling as a component of the network architecture.
C Camera Model and Jacobians
The camera model represents 3D points homogeneously and re-projects image points using inverse depth and camera intrinsics. Jacobians for pose and depth optimization are computed with local pose parameterizations, chain-rule differentiation, and the adjoint operator.
- 3D points are represented in homogeneous coordinates, and image points with inverse depth are re-projected between frames using a warping function.
- The inverse projection uses camera intrinsic parameters c = (fx, fy, cx, cy).
- Optimization requires Jacobians with respect to the camera transforms Gi and Gj and inverse depth d, treating d as a vector in R1.
- Local pose parameterizations eξiGi and eξjGj support Jacobian computation for the 3D point transformation, with the adjoint operator moving the ξi term to the front.
- The full Jacobians are obtained using the chain rule, with projection and inverse-projection Jacobians used in the derivation.
D Network Architecture
The feature and context encoders operate at one-eighth input resolution with six residual blocks, while differing in normalization and output dimensionality. The update operator injects context, correlation, and flow features into a GRU and predicts revisions and confidence weights from its hidden state.
- Both encoders use six basic residual blocks to extract features at 1/8 the input image resolution.
- The feature encoder uses instance normalization and outputs D=128 features, whereas the context encoder uses no normalization and outputs D=256 features.
- At each update iteration, context, correlation, and flow features are injected into the GRU.
- The update operator predicts revision r and confidence weights w from the updated hidden state.