Source-linked AI summary
CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction
Keisuke Tateno, Federico Tombari, Iro Laina, Nassir Navab
TL;DR
Monocular SLAM is limited by absolute-scale ambiguity, failures in difficult image regions and pure rotations, while CNN depth maps have blurred borders. CNN-SLAM fuses learned depth with direct monocular SLAM, adds semantic label fusion, and achieves robust, accurate reconstruction across two benchmarks.
Problem
Monocular reconstruction is limited by absolute-scale ambiguity, tracking failures under pure rotation, sparse or weak depth in texture-less regions, and blurred borders in CNN-predicted depth.
Method
CNN-SLAM fuses dense CNN-predicted depth with direct monocular SLAM, refines borders through small-baseline stereo, and incrementally fuses semantic labels into the global 3D model.
Results
The approach achieves robust and accurate pose estimation and reconstruction on two public SLAM benchmarks, while also producing joint 3D and semantic reconstruction from a single view.
Takeaways & Limitations
Integrating learned depth with monocular SLAM addresses absolute scale, texture-less regions, and pure rotational motion while retaining direct-SLAM robustness during camera translation.
Takeaways & Limitations
Evaluation uses CNN models trained on NYU Depth v2 indoor sequences, so the reported generalization scope is tied to that training setup.
Abstract
from arXiv · showhide
Given the recent advances in depth prediction from Convolutional Neural Networks (CNNs), this paper investigates how predicted depth maps from a deep neural network can be deployed for accurate and dense monocular reconstruction. We propose a method where CNN-predicted dense depth maps are naturally fused together with depth measurements obtained from direct monocular SLAM. Our fusion scheme privileges depth prediction in image locations where monocular SLAM approaches tend to fail, e.g. along low-textured regions, and vice-versa. We demonstrate the use of depth prediction for estimating the absolute scale of the reconstruction, hence overcoming one of the major limitations of monocular SLAM. Finally, we propose a framework to efficiently fuse semantic labels, obtained from a single frame, with dense SLAM, yielding semantically coherent scene reconstruction from a single view. Evaluation results on two benchmark datasets show the robustness and accuracy of our approach.
1. Introduction
CNN-SLAM combines learned single-image depth with direct monocular SLAM to address scale ambiguity, tracking failures, blurred depth borders, and sparse reconstruction. It also extends the framework to semantic reconstruction and evaluates generalization across benchmark environments.
- Depth-camera methods support dense reconstruction but have limited working ranges and can fail or perform poorly under sunlight.
- Monocular SLAM reconstructs scenes from nearby frames but has inherently ambiguous absolute scale and can fail during pure rotational motion.
- CNN depth prediction provides single-image absolute scale and dense estimates in low-texture regions, but blurred borders can remove shape details.
- The proposed fusion refines CNN depth with small-baseline stereo while using the prediction to provide scale information and improve tracking robustness.
- The framework also fuses pixel-wise semantic labels with dense SLAM to produce semantically coherent 3D reconstruction from a single view.
- Experiments on two public SLAM benchmarks compare pose estimation and reconstruction against monocular SLAM and depth-estimation methods, including training on a different environment and RGB sensor.
2. Related work
Related work is organized by sensing modality and methodology, covering depth-camera and monocular SLAM alongside feature-based and direct approaches. It also reviews single-view depth prediction, from hand-crafted methods to deep convolutional architectures.
- SLAM: SLAM methods are categorized by input as depth-camera-based or monocular-camera-based, and by methodology as feature-based or direct.
- SLAM: ORB-SLAM represents feature-based monocular SLAM, using sparse ORB features, local bundle adjustment, and pose-graph optimization for pose estimation and sparse reconstruction.
- SLAM: DTAM performs real-time dense reconstruction with short-baseline multiple-view stereo and regularization, while LSD-SLAM uses semi-dense depth on image-gradient regions for CPU efficiency.
- SLAM: MLM extends LSD-SLAM with a dense approach intended to increase map density and improve reconstruction accuracy.
- Depth prediction from single view: Single-view depth prediction evolved from hand-crafted features and graphical models with strong geometric assumptions toward deep convolutional architectures.
3. Proposed Monocular Semantic SLAM
The framework fuses CNN-predicted dense depth with direct monocular SLAM, refining and uncertainty-weighting key-frame maps while also attaching semantic labels to the global reconstruction.
- CNN-based Depth Prediction and Semantic Segmentation: CNN-predicted dense depth maps are generated for key-frames and paired with depth uncertainty maps initialized from prediction confidence.The system uses a ResNet-based fully convolutional network for depth prediction and retrains it for pixel-wise semantic labels.
- Frame-wise Depth Refinement: Depth refinement uses small-baseline stereo matching and uncertainty propagation, concentrating refinement near image gradients where matching is more accurate.Dense key-frame maps allow refinement across all pixels, while uncertainty weighting favors measurements according to their confidence.
- Camera Pose Estimation: Camera pose is estimated relative to the nearest key-frame by minimizing photometric residuals with weighted Gauss–Newton optimization.Residual computation is restricted to high color-gradient pixels for efficiency, using the key-frame vertex map and camera projection.
- Key-frame Creation and Pose Graph Optimization: CNN depth is adjusted for differences between current and training-camera focal lengths before fusion with direct monocular depth measurements.The adjustment addresses inaccurate absolute reconstruction scale caused by mismatched camera intrinsics.
- Key-frame Creation and Pose Graph Optimization: On one ICL-NUIM sequence, depth adjustment improved both depth accuracy and pose trajectory accuracy over direct CNN prediction.Figure 3 compares direct prediction, adjusted depth, and adjusted-plus-refined depth using pose trajectories and depth pixels within 10% of ground truth.
- Frame-wise Depth Refinement: After adjustment and refinement, ICL-NUIM results showed significant improvement in both depth estimation and pose trajectory compared with earlier stages.The final comparison corresponds to the adjusted and refined depth map in Figure 3.
4. Evaluation
The evaluation compares the proposed method with monocular SLAM, dense depth estimation, and CNN-based depth fusion on two public benchmarks, including tests of generalization, rotational motion, and semantic reconstruction. Results report stronger pose and reconstruction accuracy, robustness under pure rotation, and joint 3D-semantic reconstruction.
- 4. Evaluation: The evaluation uses quantitative comparisons on ICL-NUIM and TUM, plus qualitative tests of rotational robustness and semantic label fusion.The CNN was trained on NYU Depth v2 indoor sequences to test generalization to unseen benchmark environments.
- 4.1. Comparison against SLAM state of the art: The method is compared with LSD-SLAM, ORB-SLAM, REMODE, and CNN-predicted depths used without refinement.Pose accuracy is measured with Absolute Trajectory Error, while reconstruction is assessed by correctly estimated depth percentage.
- 4.1. Comparison against SLAM state of the art: The proposed approach reports higher pose trajectory accuracy and denser, more accurate key-frame reconstructions than monocular baselines and unrefined CNN-depth fusion.Its average pose accuracy exceeds LSD-SLAM even after bootstrapping, indicating benefits beyond scale correction.
- 4.1. Comparison against SLAM state of the art: CNN-guided refinement reduces prediction blur and supplies dense estimates in low-textured regions where REMODE tends to fail.Figure 4 compares ground truth, the refined key-frame, raw CNN prediction, LSD-SLAM with bootstrapping, and REMODE on ICL-NUIM office2.
- 4.2. Accuracy under pure rotational motion: Under mostly pure rotational motion, the proposed method reconstructs scene structure, whereas LSD-SLAM is noisy and ORB-SLAM completely fails.The comparison uses the TUM fr1/rpy sequence, where consecutive frames generally lack the stereo baseline needed for monocular depth estimation.
- 4.3. Joint 3D and semantic reconstruction: The method also produces joint 3D and semantic reconstructions on an office sequence and NYU Depth V2 kitchen sequences.Figure 6 displays color reconstructions alongside semantic labels; the paper describes this as the first demonstration of joint 3D and semantic reconstruction with a monocular camera.
5. Conclusion
The conclusion presents CNN-depth and SLAM integration as a way to address absolute-scale ambiguity, textureless regions, and pure rotational motion. It retains direct monocular SLAM strengths while supporting semantic fusion and identifies closing the depth-prediction loop as future work.
- 5. Conclusion: Integrating deep-neural-network depth prediction with SLAM addresses absolute scale, textureless regions, and pure rotational motion.The conclusion frames these as inherent limitations of traditional monocular reconstruction.
- 5. Conclusion: Refining CNN-predicted depths with small-baseline stereo matching preserves direct monocular SLAM robustness and accuracy during camera translation and high image gradients.
- 5. Conclusion: The framework jointly reconstructs scenes and fuses semantic segmentation labels with the global 3D model from a monocular camera.
- 5. Conclusion: Future work includes closing the loop by improving depth prediction with geometrically refined depth maps.