PaperlayerView exact revision on arXiv

Source-linked AI summary

DXSLAM: A Robust and Efficient Visual SLAM System with Deep Features

Dongjiang Li, Xuesong Shi, Qiwei Long, Shenghui Liu, Wei Yang, Fangshi Wang, Qi Wei, Fei Qiao

arXiv:2008.05416v1cs.CVcs.RO

TL;DR

Visual SLAM systems often rely on hand-crafted features that are vulnerable to complex environmental and viewpoint changes. DXSLAM integrates a deep CNN’s local and global features into a modern SLAM pipeline, adding feature-based vocabulary methods for loop closure and CPU optimizations for deployment. The resulting modules and full system outperform the ORB-SLAM2 baseline in robustness-related evaluations, while the optimized system runs in real time without a GPU.

  • Problem

    Hand-crafted visual features can fail to provide consistent detection and association in complex environments and under scene or viewpoint changes.

  • Method

    DXSLAM uses a deep CNN to extract local and global features for pose tracking, mapping, loop closure, and re-localization within a modern SLAM pipeline.

  • Results

    The proposed modules and full system outperform ORB-SLAM2 in robustness-related evaluations, with lower trajectory errors and higher correct rates across evaluated data.

  • Takeaways & Limitations

    Deep CNN features can be incorporated into a real-time SLAM system that operates on GPU-free mobile computing platforms.

Abstract

from arXiv · show

A robust and efficient Simultaneous Localization and Mapping (SLAM) system is essential for robot autonomy. For visual SLAM algorithms, though the theoretical framework has been well established for most aspects, feature extraction and association is still empirically designed in most cases, and can be vulnerable in complex environments. This paper shows that feature extraction with deep convolutional neural networks (CNNs) can be seamlessly incorporated into a modern SLAM framework. The proposed SLAM system utilizes a state-of-the-art CNN to detect keypoints in each image frame, and to give not only keypoint descriptors, but also a global descriptor of the whole image. These local and global features are then used by different SLAM modules, resulting in much more robustness against environmental changes and viewpoint changes compared with using hand-crafted features. We also train a visual vocabulary of local features with a Bag of Words (BoW) method. Based on the local features, global features, and the vocabulary, a highly reliable loop closure detection method is built. Experimental results show that all the proposed modules significantly outperforms the baseline, and the full system achieves much lower trajectory errors and much higher correct rates on all evaluated data. Furthermore, by optimizing the CNN with Intel OpenVINO toolkit and utilizing the Fast BoW library, the system benefits greatly from the SIMD (single-instruction-multiple-data) techniques in modern CPUs. The full system can run in real-time without any GPU or other accelerators. The code is public at https://github.com/ivipsourcecode/dxslam.

I. INTRODUCTION

DXSLAM addresses the vulnerability of hand-crafted visual features in changing environments by integrating deep CNN features throughout a modern SLAM pipeline. It combines local and global features, visual vocabulary, and CPU optimization to improve robustness and enable real-time operation without GPUs.

  • Motivation: Hand-crafted features can produce inconsistent detection and association in complex environments, including failures to recognize revisited scenes after scene or viewpoint changes.The cited examples include SIFT, Shi-Tomasi, ORB, and ORB-SLAM2.
  • Approach: DXSLAM uses a deep CNN to extract local and global features from each image frame for pose tracking, local mapping, loop closure detection, and re-localization.The deep features are incorporated into a modern SLAM pipeline largely based on ORB-SLAM2.
  • Contributions: The full system uses deep CNN features for loop closure, global optimization, and re-localization, providing greater robustness to environmental and viewpoint changes than hand-crafted features.The contribution statement describes this as a full SLAM system with loop closure and global optimization.
  • Contributions: DXSLAM combines global-feature retrieval with group matching for re-localization, achieving a higher success rate and lower computation cost than conventional BoW-based methods.The re-localization method is designed around global feature-based image retrieval and group matching.
  • Contributions: A loop closure method combines global and local features with a newly trained visual vocabulary that aggregates local features.The vocabulary supports feature-based loop closure detection.
  • Efficiency: Intel OpenVINO and Fast BoW optimize feature extraction and loop closure for modern CPUs, allowing real-time operation without GPUs.The system uses SIMD techniques available in modern CPUs.

II. RELATED WORKS

Related work covers conventional keypoint features, learned local descriptors and detectors, CNN-based pose estimation, and deep-feature methods incorporated into SLAM. DXSLAM builds on these developments by selecting HF-Net for joint local and global feature extraction.

  • Hand-crafted and learned features: Visual SLAM pipelines commonly detect keypoints and match descriptors, with Shi-Tomasi and ORB widely used for their balance of effectiveness and efficiency.These features appear in systems including MonoSLAM, ORB-SLAM2, and VINS-Mono.
  • Hand-crafted and learned features: Deep CNN features from ImageNet models and task-specific networks can outperform hand-crafted local features, while learned systems such as SuperPoint jointly detect and describe keypoints.SuperPoint uses self-supervised learning and a fully convolutional network.
  • Deep-feature SLAM: DXSLAM empirically selects HF-Net for SuperPoint-like keypoint detection and description, while noting that other CNN-based approaches could also fit the proposed system.GCNv2 is identified as a closely related system incorporating deep local features into an ORB-SLAM2 pipeline.
  • Deep-feature SLAM: CNNs for end-to-end pose estimation make feature extraction implicit, but their accuracy is reported as not yet competitive.The paper presents these approaches as an avenue for further investigation.

B. Loop Closure Detection

Loop closure detection must recognize previously visited places while prioritizing precision because false loop closures can damage the map. Related approaches use visual vocabularies, BoW aggregation, and CNN-derived features for efficient retrieval and verification.

  • Loop closure requirements: Loop closure detection recognizes previously visited places from current camera measurements and helps correct accumulated errors while building a consistent map.The paper emphasizes precision because false loop closures may damage the map.
  • BoW-based methods: Tree-structured visual vocabularies aggregate local features into image vectors to improve real-time image retrieval efficiency.FAB-MAP is cited as an application using SURF and BoW for place recognition under viewpoint changes.
  • BoW-based methods: FBoW accelerates vocabulary loading and matching by applying SIMD techniques on x86 CPUs.It is described as an optimized implementation of BoW.
  • Deep-feature methods: Recent methods combine CNN representations with BoW-based loop closure, including bag-of-SuperPoints with graph verification and CNN-based visual vocabularies for challenging environments.The cited bag-of-SuperPoints method can lose robustness under low illumination in the reported experiments.
  • Re-localization: Re-localization resembles loop closure but typically uses a larger candidate database and emphasizes high recall rather than loop closure’s high precision.Candidate retrieval can use BoW-aggregated local features or learned global CNN features.

III. SLAM SYSTEM WITH DEEP FEATURES

DXSLAM keeps the ORB-SLAM2-like pipeline but replaces its feature extraction with HF-Net, which supplies local and global features for localization, mapping, re-localization, and loop closure. CPU-oriented optimization and feature-selection experiments support an efficient implementation based on HF-Net.

  • System modules: Global features support an efficient re-localization module for initialization or tracking failures, while loop closure uses global features and local matching with a pre-trained BoW vocabulary.The system uses these complementary representations to mitigate accumulated localization errors and improve global consistency.
  • Efficiency: FBoW provides a binary visual vocabulary that reduces initialization time and improves matching efficiency compared with conventional BoW methods.The implementation adopts FBoW specifically to improve system efficiency.
  • Feature extraction: HF-Net supplies keypoints, local descriptors, and global image descriptors through one CNN model for the localization and mapping pipeline.Its shared encoder feeds decoders for keypoint scores, dense local descriptors, and a NetVLAD-based global descriptor.
  • Feature evaluation: HF-Net features are experimentally superior to features from alternative deep CNN extractors for the SLAM system.The selection is motivated by both functionality and comparative experimental results.
  • CPU implementation: OpenVINO re-implements HF-Net inference using SIMD operations on x86 CPUs, with unsupported local-descriptor upsampling moved into post-processing.The optimization begins from the TensorFlow implementation and uses Intel’s Model Optimizer and Inference Engine.

C. Vocabulary Training

DXSLAM trains a visual vocabulary from selected local descriptors and uses FBoW to make vocabulary loading and feature matching efficient.

  • Vocabulary construction: The system selects the top 300 descriptors by keypoint detection scores before clustering them into visual words and parent nodes.The vocabulary is trained on OpenLORIS-Scene images.
  • Vocabulary construction: The trained vocabulary quantizes features into visual words and represents images with corresponding word-histogram vectors for matching.
  • Efficiency: FBoW stores the trained vocabulary in binary form, reducing DXSLAM initialization to about 40 milliseconds versus about 6 seconds for ORB-SLAM2.

D. Re-localization with Global Features

DXSLAM addresses re-localization failures by combining global-descriptor retrieval with group matching, mitigating both candidate-retrieval failures and insufficient local matches.

  • Failure modes: ORB-SLAM2 re-localization can fail when BoW retrieves no candidate frames or when too few local matches support pose estimation.
  • Global retrieval: Global descriptors provide coarse candidate retrieval that is more robust to environmental and viewpoint changes than BoW retrieval.
  • Group matching: Group matching compares the current frame with keypoints from all retrieved candidates, increasing the available matches for pose estimation.
  • Candidate verification: DXSLAM selects top K candidates using local-feature visual-vector similarity, then uses global-descriptor distance and a threshold to report a loop candidate.
  • Candidate verification: Because BoW discards spatial relations, DXSLAM uses global descriptors as a complementary criterion to reduce false matches.

IV. EVALUATION

The evaluation examines individual modules and the full system across SLAM and loop-closure datasets, using OpenLORIS-Scene for robustness and TUM RGB-D for localization accuracy.

  • Evaluation design: The evaluation compares feature extractors, tests re-localization and loop detection under controlled challenges, and measures full-system lifelong SLAM performance.
  • Evaluation design: The experiments use separate SLAM-evaluation and LCD-evaluation datasets.
  • SLAM datasets: OpenLORIS-Scene supplies challenging robotic data with blur, featureless images, dim lighting, and environmental changes.
  • SLAM datasets: The study mainly uses OpenLORIS-Scene for robustness and TUM RGB-D for localization accuracy.
  • LCD datasets: City Center and New College evaluate LCD under dynamic objects, sunlight, wind, viewpoint changes, and repeated visual elements.

B. Feature Evaluation

Feature evaluation compares CNN-based and hand-crafted front ends within a common ORB-SLAM2 framework, while re-localization testing examines challenging OpenLORIS-Scene conditions.

  • Feature comparison: The feature comparison substitutes HF-Net, SuperPoint, and D2-Net into ORB-SLAM2 without retraining, finetuning, re-localization, or loop closure.
  • Feature comparison: Fig. 2 reports per-sequence successful initialization and tracking, average correct rate, and average ATE RMSE for each feature configuration.
  • Feature comparison: CNN-based visual SLAM is more robust than ORB-SLAM2 and DS-SLAM on the evaluated OpenLORIS scenes.
  • Feature comparison: HF-Net avoids the low-light feature shortage that causes SuperPoint tracking failures in the office scene.
  • Re-localization evaluation: The office re-localization test reports scores under controlled challenges, and the proposed method outperforms ORB-SLAM2 in all cases.
  • Re-localization evaluation: The method fails to re-localize under a significant viewpoint reversal but substantially improves robustness to illumination changes.

D. Loop Closure Detection Evaluation

The proposed loop-closure evaluation combines vocabulary-based candidate retrieval with global-descriptor information to remove wrong matches, improving precision-recall over ORB-SLAM2. Full-system tests further report greater robustness and accuracy in changing and dynamic environments.

  • The LCD pipeline retrieves loop candidates with FBoW and uses global-descriptor spatial information to remove wrong matches.The full method is HF-FBoW-GLB; HF-FBoW selects only the top candidate by similarity score.
  • Group matching uses retrieved keyframes and their best-quality keypoints, increasing both the quantity and quality of matches over frame-to-frame matching.The query frame is matched against a group of five retrieved candidate images.
  • The combined local- and global-feature method improves precision and recall over ORB-SLAM2 on the evaluated City Center and New College camera data.The comparison includes HF-FBoW-GLB, HF-FBoW, and ORB-BoW.
  • DXSLAM produces more correct pose estimations and relocalizations than DXSLAM-no-incre on the evaluated office, cafe, and market scenes.The comparison evaluates the incremental visual-vocabulary method against its non-incremental version.
  • On TUM RGB-D, DXSLAM shows comparable localization results to DS-SLAM while resisting dynamic environments without explicitly addressing dynamic features.

F. Runtime Performance

Runtime evaluation compares precision-recall behavior, lifelong-SLAM robustness and accuracy, and feature-extraction timing. OpenVINO optimization reduces HF-Net CPU processing time by 68% to 46.2 ms per image, or 21.6 FPS.

  • OpenVINO optimization reduces HF-Net CPU feature-extraction time by 68% to 46.2 ms per image, achieving 21.6 FPS.The measurement uses 640x480 images from the OpenLORIS-Scene office1-1 sequence on an Intel NUC.
  • Figure 4 compares LCD precision-recall curves for HF-FBoW-GLB, HF-FBoW without global features, and ORB-SLAM2.
  • Figure 5 reports lifelong-SLAM correct rate and average ATE RMSE for OpenLORIS-Scene evaluations.Blue dots and lines indicate successful initialization, relocalization, and tracking; larger correct rates and smaller ATE RMSE values are preferable.
  • Table III reports average feature-extraction time in milliseconds per image for the evaluated models.

V. CONCLUSIONS

DXSLAM incorporates deep CNN-based features into modern SLAM and uses CPU SIMD techniques to support real-time operation on GPU-free mobile platforms. The system is presented as a first step toward lifelong SLAM, with future work focused on better features and more advanced feature-based systems.

  • Conclusions: DXSLAM incorporates deep CNN-based features into a modern SLAM system and identifies HF-Net for its robustness, efficiency, and local and global feature extraction.The paper frames this as improving SLAM system performance.
  • Conclusions: SIMD techniques enable DXSLAM to run in real time on GPU-free mobile computing platforms, enabling easier employment in real robots.
  • Conclusions: DXSLAM is a first step toward lifelong SLAM, with future work targeting better CNN features and integration into more advanced feature-based SLAM systems.The paper gives improving CNN architecture and training strategy, and incorporating deep features into systems such as ORB-SLAM3, as two future directions.
Loading 2008.05416v1…