PaperlayerView exact revision on arXiv

Source-linked AI summary

LIFT-SLAM: a deep-learning feature-based monocular visual SLAM method

Hudson M. S. Bruno, Esther L. Colombini

arXiv:2104.00099v2cs.CVcs.AIcs.RO

TL;DR

Traditional VSLAM can struggle in challenging environments, while prior learned approaches have not yet outperformed it and hybrid robustness remains under-evaluated. LIFT-SLAM combines learned LIFT features with an ORB-SLAM-based monocular pipeline and an adaptive matching-threshold approach. The authors report improved robustness without losing accuracy and performance across different environments and image distortions.

  • Problem

    Traditional VSLAM can fail under challenging motion, environments, or camera conditions, while learned methods have not yet outperformed traditional methods and hybrid robustness remains under-evaluated.

    Source: §1 · p. 2

  • Method

    LIFT-SLAM uses LIFT feature extraction in a traditional ORB-SLAM-based monocular VSLAM pipeline and adapts matching thresholds during execution.

    Source: §1 · pp. 2–3, §5 · p. 25

  • Results

    The authors report that the hybrid algorithm is more robust than traditional VSLAM without losing accuracy, and operates in indoor and outdoor environments while improving results under gamma power transformation and quantile-based truncation.

    Source: §5 · p. 25

  • Takeaways & Limitations

    The authors conclude that selecting learned features could improve algorithm performance.

    Source: §5 · p. 25

    In large environments, good performance depends on loop-closure detection; when a loop is missed, drift error increases indefinitely because pose-graph optimization does not correct it.

    Source: §4.4 · p. 23

Abstract

from arXiv · show

The Simultaneous Localization and Mapping (SLAM) problem addresses the possibility of a robot to localize itself in an unknown environment and simultaneously build a consistent map of this environment. Recently, cameras have been successfully used to get the environment's features to perform SLAM, which is referred to as visual SLAM (VSLAM). However, classical VSLAM algorithms can be easily induced to fail when either the motion of the robot or the environment is too challenging. Although new approaches based on Deep Neural Networks (DNNs) have achieved promising results in VSLAM, they still are unable to outperform traditional methods. To leverage the robustness of deep learning to enhance traditional VSLAM systems, we propose to combine the potential of deep learning-based feature descriptors with the traditional geometry-based VSLAM, building a new VSLAM system called LIFT-SLAM. Experiments conducted on KITTI and Euroc datasets show that deep learning can be used to improve the performance of traditional VSLAM systems, as the proposed approach was able to achieve results comparable to the state-of-the-art while being robust to sensorial noise. We enhance the proposed VSLAM pipeline by avoiding parameter tuning for specific datasets with an adaptive approach while evaluating how transfer learning can affect the quality of the features extracted.

1. Introduction

The introduction frames monocular VSLAM as useful but vulnerable to challenging scenes and camera conditions, while noting gaps in evaluation of learned-feature hybrids. It proposes LIFT feature extraction within an ORB-SLAM pipeline and experiments spanning transfer learning, adaptive thresholds, and multiple datasets.

1. Introduction

  • Traditional VSLAM can fail in poor illumination, featureless areas, high-speed motion, or camera distortion, and monocular systems also face scale uncertainty.The conventional pipeline includes initialization, feature detection and matching, outlier rejection, motion estimation, optimization, and relocalization.Source: §1 · p. 2
  • Hybrid learned-feature methods can leverage deep learning to enhance traditional VSLAM, but their robustness in challenging conditions and across varied scenarios remains under-evaluated.The introduction also notes that end-to-end DNN approaches have not yet outperformed traditional methods [1].Source: §1 · p. 2
  • LIFT-SLAM applies Learned Invariant Feature Transform (LIFT) feature extraction within a traditional monocular VSLAM pipeline based on ORB-SLAM.The proposal uses learned features with a conventional geometry-based system.Source: §1 · pp. 2–3
  • The work evaluates how transfer learning and fine-tuning affect the resulting hybrid system’s feature quality.The introduction lists this as one of the paper’s contributions.Source: §1 · p. 3
  • The experiments use KITTI and Euroc and assess robustness under camera distortions; an adaptive extension aims to avoid dataset-dependent parameter tuning.The adaptive approach is described as enhancing performance without fine-tuning parameters that usually depend on the dataset.Source: §1 · p. 3

2. Related Work

Related work covers feature-based systems, end-to-end learned pose estimation, and hybrid methods that replace selected VSLAM modules. The cited learned-feature approaches are presented as promising, while their robustness across challenging scenarios and multiple datasets remains insufficiently evaluated.

Feature-based approaches

  • ORB-SLAM uses ORB features, Bag-of-Words place recognition, and local and global pose-graph optimization, but may fail in some situations and require per-dataset parameter tuning.It is described as a state-of-the-art feature-based monocular VSLAM algorithm.Source: §2 · pp. 3–4

End-to-end deep learning-based approaches

  • End-to-end approaches include DeepVO and UnDeepVO [1], while another approach adds Neural Graph Optimization; these methods are reported as not yet accurate enough to surpass traditional methods.DeepVO estimates pose using CNN features and stacked LSTM layers; UnDeepVO uses stereo pairs during training to recover scale and monocular images for testing.Source: §2 · p. 4

Hybrid approaches

  • Hybrid approaches replace selected traditional VSLAM modules; Neural Bundler estimates motion before constructing a conventional pose graph, while SuperGlue performs feature matching.SuperGlue is described as a learnable “middle-end” between the front-end and back-end.Source: §2 · pp. 4–5
  • Locally learned feature methods combine learned extraction or descriptors with traditional VSLAM back-ends, but the cited approaches lack robustness evaluation across challenging scenarios or multiple datasets.Examples include DF-SLAM [5] and SuperPointVO.Source: §2 · p. 5

3. Proposed Method

LIFT-SLAM combines LIFT feature extraction with an ORB-SLAM-based monocular VSLAM pipeline, and adds fine-tuning and adaptive matching thresholds. The method description covers LIFT’s training and processing, pipeline stages, and feature-matching comparisons.

3. Proposed Method

  • LIFT-SLAM uses LIFT to extract features in an ORB-SLAM-based monocular VSLAM system that builds sparse, keyframe-based graph maps for bundle adjustment.The proposed system optimizes estimated camera poses through bundle adjustment.Source: §3 · p. 5

3.1. LIFT

  • LIFT detects feature locations from score maps, estimates patch orientation, and computes descriptors; its four-branch training uses same-point views, a different-point view, and a featureless patch.The featureless fourth branch trains only the detector, while the other patches include different views of physical points.Source: §3.1 · pp. 5, 7
  • Across frame-skipping and gamma-exposure tests, LIFT retained correct matches while ORB produced wrong frame-skip matches and spatially clustered gamma matches (Figure 3).The comparison reports that LIFT’s matched keypoints were spread across the image under gamma transformations.Source: §3.1 · p. 9, Figure 3 · p. 10

3.2. LIFT-SLAM Pipeline

  • Tracking and mapping run sequentially, while loop closure runs in a separate thread; tracking uses LIFT features for matching and predicts poses with a constant-velocity model.Mapping adds keyframes and triangulated map points, then locally optimizes the covisibility graph; loop closure validates candidates geometrically before correction.Source: §3.2 · pp. 9, 11, 12

3.3. Versions of LIFT-SLAM

  • The authors create LIFT-SLAM variants, including transfer-learning fine-tuning of the orientation estimator to address differences between photo-tourism images and sequential VO imagery.Only the orientation estimator is updated after training the network.Source: §3.1 · p. 6, §3.3 · p. 12

3.3.1. Fine-tuned LIFT-SLAM

  • The fine-tuned models use SIFT keypoints from VisualSFM as ground truth and are trained separately on KITTI and Euroc image sets.The KITTI set uses sequences 00, 06, 09, and 10; the Euroc set uses MH 04, V1 03, and V2 03.Source: §3.3.1 · pp. 12–13

3.4. Adaptive LIFT-SLAM

  • Matching thresholds trade off rejecting incorrect associations against losing good matches and camera tracking; fixed best values vary across datasets.The lower threshold is used when stricter match quality is needed, such as relocalization or map-point triangulation.Source: §3.4 · p. 13
  • The adaptive method sets thresholds online from current-frame outliers and previous-frame map points, increasing thresholds when their difference shrinks.The described relationship is illustrated by the threshold-variation figure; the table caption gives different threshold pairs for Euroc MH 01 and KITTI 05.Source: §3.4 · pp. 13–14, Table 1 · p. 13, Figure 5 · p. 14

4. Experiments

Across KITTI and Euroc, LIFT-SLAM variants generally improved on ORB-SLAM, and Adaptive LIFT-SLAM fine-tuned with Euroc had the best overall results. Tests also found greater robustness than ORB-SLAM under the applied image distortions, while results against literature were competitive without KITTI fine-tuning.

4.1. Datasets

  • Experiments used KITTI sequences 00–10 with ground truth and Euroc’s indoor and outdoor sequences, chosen to vary camera motion and environments.KITTI evaluation used translation and rotation RPE and ATE, averaged over five executions; Euroc was evaluated by ATE.Source: §4.1 · p. 15, §4.2 · p. 15

4.2. Trajectory Evaluation

  • On KITTI, LIFT-SLAM generally outperformed ORB-SLAM, especially on shorter sequences, and proposed variants improved over original LIFT-SLAM on most sequences.Fine-tuning with Euroc also improved the algorithm’s performance on KITTI.Source: Table 2 · p. 16, §4.2 · p. 16
  • Adaptive LIFT-SLAM fine-tuned with Euroc sequences had the best overall results across the Euroc quantitative and qualitative comparisons.ORB-SLAM had the smallest average in three sequences, while the proposed LIFT-SLAM versions outperformed original LIFT-SLAM in all sequences.Source: §4.2 · pp. 16, 18, Table 3 · p. 18

4.3. Robustness tests

  • Salt-and-pepper noise prevented every tested algorithm from initializing a map, so results for that scenario were omitted.The distortion was intended to simulate camera sensor-cell malfunction.Source: §4.3 · p. 20
  • Under the applied distortions, LIFT-SLAM was more robust than ORB-SLAM, though it failed in some KITTI 10 and Euroc MH 02 cases.ORB-SLAM could not track in some distorted KITTI 03, 06, and 10 cases; LIFT-SLAM trajectories were less affected in KITTI 03 and 06, while ORB-SLAM often failed to track much of KITTI 07.Source: §4.3 · p. 20, Figure 9 · p. 22, Table 4 · p. 23

4.4. Comparison with literature

  • LIFT-SLAM was slower than other state-of-the-art algorithms; descriptor generation took 36 seconds for KITTI images and 35 seconds for Euroc images on a GTX 1050 Ti.The authors attribute the runtime to unoptimized LIFT code and unnecessary time-consuming operations.Source: §4.4 · pp. 23–24
  • LIFT-SLAM had the smallest errors on KITTI sequences 00, 03, 04, 07, 08, and 10, with competitive results despite no KITTI fine-tuning.DeepVO and NeuralBundler were trained on some KITTI sequences; qualitative trajectories were closer to ground truth for LIFT-SLAM in most sequences.Source: Table 5 · p. 24, §4.4 · pp. 24–25

5. Conclusion

The conclusion reports that learned feature extraction and transfer learning improved a traditional monocular VSLAM system across datasets. Adaptive matching thresholds removed fixed threshold values without requiring dataset-specific fine-tuning, and distortion experiments found improved results under the tested conditions.

5. Conclusion

  • LIFT feature extraction improved a traditional monocular VSLAM system, while transfer learning improved performance across different VSLAM problems and datasets.The approach also adaptively adjusted matching thresholds based on outlier counts, eliminating fixed values without dataset fine-tuning.Source: §5 · p. 25
  • The conclusion evaluates five LIFT-SLAM variants, combining adaptive matching thresholds with fine-tuning on KITTI or Euroc sequences.The variants include fine-tuned, adaptive, and adaptively fine-tuned systems.Source: §5 · p. 25
  • The robustness experiments found that the hybrid system outperformed a traditional VSLAM algorithm under gamma and quantile-truncation image distortions without losing accuracy.The authors propose attention-based feature selection as future work, based on the indication that selecting learned features could improve performance.Source: §5 · p. 25
Loading 2104.00099v2…