PaperlayerView exact revision on arXiv

Source-linked AI summary

AirSLAM: An Efficient and Illumination-Robust Point-Line Visual SLAM System

Kuan Xu, Yuefan Hao, Shenghai Yuan, Chen Wang, Lihua Xie

arXiv:2408.03520v4cs.RO

TL;DR

Severe illumination challenges both frame-to-frame tracking and relocalization against existing maps, while many approaches address only one challenge. AirSLAM combines learned point-and-line features with geometric optimization and map reuse; the authors report performance in illumination-challenging environments and rates of 73Hz on a PC and 40Hz on an embedded platform.

  • Problem

    Existing vSLAM systems struggle with severe lighting, and many methods focus on only short-term tracking or long-term relocalization rather than both challenges.

    Source: §I · p. 1

  • Method

    AirSLAM combines learned point-and-line feature detection and matching with geometric optimization, and uses a multi-stage pipeline to relocalize against a reused map.

    Source: §I · p. 1, B. System Architecture · p. 3

  • Results

    The authors report that AirSLAM outperforms other state-of-the-art vSLAM systems in accuracy, efficiency, and robustness under illumination-challenging conditions, running at 73Hz on a PC and 40Hz on an embedded platform.

    Source: §I · p. 2, §VIII · p. 17

  • Takeaways & Limitations

    The system is designed for deployment on real robots, with feature networks accelerated using C++ and NVIDIA TensorRT.

    Source: §I · p. 2, B. System Architecture · p. 3

    AirSLAM relies on enough line features and is best suited to man-made environments; in unstructured environments it degrades into a point-only system.

    Source: §VIII · p. 17

Abstract

from arXiv · show

In this paper, we present an efficient visual SLAM system designed to tackle both short-term and long-term illumination challenges. Our system adopts a hybrid approach that combines deep learning techniques for feature detection and matching with traditional backend optimization methods. Specifically, we propose a unified convolutional neural network (CNN) that simultaneously extracts keypoints and structural lines. These features are then associated, matched, triangulated, and optimized in a coupled manner. Additionally, we introduce a lightweight relocalization pipeline that reuses the built map, where keypoints, lines, and a structure graph are used to match the query frame with the map. To enhance the applicability of the proposed system to real-world robots, we deploy and accelerate the feature detection and matching networks using C++ and NVIDIA TensorRT. Extensive experiments conducted on various datasets demonstrate that our system outperforms other state-of-the-art visual SLAM systems in illumination-challenging environments. Efficiency evaluations show that our system can run at a rate of 73Hz on a PC and 40Hz on an embedded platform. Our implementation is open-sourced: https://github.com/sair-lab/AirSLAM.

I. INTRODUCTION

AirSLAM targets illumination challenges in both adjacent-frame pose estimation and query-to-map matching, alongside the efficiency demands of real-world deployment. It combines learned point-and-line features with geometric optimization and map reuse.

I. INTRODUCTION

  • Existing vSLAM systems struggle with severe lighting: adjacent-frame tracking and pose estimation can fail, while illumination changes can reduce query-to-map relocalization success.The paper identifies these as short-term and long-term illumination challenges, respectively.Source: §I · p. 1
  • Many existing approaches target only one illumination challenge, while learned systems may struggle to produce maps for long-term localization and neural pipelines can demand powerful GPUs.AirSLAM is presented as a unified system for the simultaneous challenges and resource-constrained deployment needs described by the authors.Source: §I · p. 1
  • AirSLAM combines learning-based feature detection and matching with traditional geometric pose and map optimization, using point and line features across tracking, mapping, and relocalization.Its unified model, PLNet, detects keypoints and structural lines; the system also proposes multi-stage relocalization using appearance and geometry.Source: §I · pp. 1–2
  • 73Hz on a PC and 40Hz on an embedded platform; AirSLAM reports accurate, robust mapping and relocalization under illumination-challenging conditions.The authors also describe deploying and accelerating the feature networks using C++ and NVIDIA TensorRT.Source: §I · p. 2
  • The paper extends AirVO, a visual-only odometry system, with a unified point-line detector, LightGlue matching, stereo and stereo-inertial support, loop closure, map optimization, and multi-stage relocalization.AirVO used SuperPoint, modified LSD, and SuperGlue, and did not reuse a map for drift-free relocalization.Source: §I · p. 2

1) Keypoint Detection:

  • Handcrafted keypoint features are usually efficient but not robust enough in challenging environments; learning-based alternatives have been applied to visual SLAM, though some are not yet efficient enough.The introduction cites ORB, FAST, and BRISK as handcrafted examples and R2D2, DISK, and D2Net among learning-based features.Source: 1) Keypoint Detection · p. 2

2) Line Detection:

The paper motivates AirSLAM as a response to illumination-robustness and deployment challenges, then organizes mapping, offline optimization, and map reuse into a hybrid system. Related work distinguishes short-term tracking from long-term relocalization challenges, while the system architecture separates online and offline work.

2) Line Detection:

  • Learning-based line detectors are more robust in challenging environments than handcrafted alternatives but are difficult to use in real-time SLAM because of their efficiency costs.StereoVO, using SuperPoint and SOLD2, is reported to run at about 7Hz on a good GPU.Source: 2) Line Detection · p. 2

B. Short-Term Illumination Challenge

  • Short-term illumination changes can disrupt tracking, while long-term changes can reduce relocalization success; existing approaches often address only one challenge or remain inefficient.Learning-based VO/SLAM methods trained on diverse lighting may require powerful GPUs and long training times, while efficient variants can be less accurate than traditional systems.Source: B. Short-Term Illumination Challenge · pp. 2–3

C. Long-Term Illumination Challenge

  • Visual place recognition retrieves similar images but usually cannot directly provide the accurate pose estimation required in robot applications.Hloc combines global retrieval with local-feature matching; another long-term localization approach relies on GPS, compass, and gravity sensors for coarse retrieval.Source: C. Long-Term Illumination Challenge · p. 3

A. Notations

  • The paper’s notation uses R for real numbers, Euclidean norms for vectors, and T, R, and t for coordinate-frame transformations, rotations, and translations.Superscripts and subscripts may be omitted when their meaning remains unambiguous in context.Source: A. Notations · p. 3

B. System Architecture

  • The system uses stereo VO/VIO to build an initial map, processes it through offline optimization, and reuses the optimized map for online relocalization.The online relocalization pipeline retrieves candidates with a keypoint vocabulary, filters them with a junction vocabulary and structure graph, then matches features to estimate pose.Source: B. System Architecture · pp. 3–4, Figure 1 · p. 3

IV. FEATURE DETECTION

PLNet addresses the cost of separate keypoint and line detectors with a unified network. Its shared backbone supports both feature types, and a two-round training procedure accommodates their different data and training requirements.

A. Motivation

  • Existing point-line systems typically detect keypoints and lines separately, while jointly training their detection networks can hinder real-time performance on resource-constrained platforms.Keypoint and line detection also typically require different real-image datasets and training procedures; the Wireframe dataset contains 5,462 discontinuous images.Source: A. Motivation · p. 4

B. Architecture Design

  • PLNet uses a shared backbone, a keypoint module, and a line module, building on the overlap between line junctions and keypoints and edge information in keypoint feature maps.The line module uses a reduced U-Net-like CNN to increase receptive field for line detection, then processes outputs to produce junctions and lines.Source: B. Architecture Design · pp. 4–5

C. Network Training

  • PLNet training first fits the backbone and keypoint module, then freezes them while training the line module on the Wireframe dataset.Line segments are encoded with an attraction region field; seven types of photometric augmentation are applied to improve robustness in illumination-challenging environments.Source: C. Network Training · p. 5

V. STEREO VISUAL ODOMETRY

The stereo VO/VIO pipeline couples learned keypoint and line features with geometric optimization. It matches lines through their associated keypoints, triangulates 3D features, and represents spatial lines to support mapping and optimization.

A. Overview

  • The stereo VO/VIO pipeline detects keypoints and lines with PLNet, matches keypoints with LightGlue, associates and matches lines, then triangulates features and optimizes points, lines, and keyframe poses.When an IMU is available, preintegrated measurements enter initial pose estimation and local bundle adjustment.Source: A. Overview · p. 5

B. Feature Matching

  • Line matching inherits keypoint matches by associating nearby keypoints with line segments, avoiding a separate line-matching network.For a line pair, the method uses the fraction of associated point matches and a minimum matching-count threshold to decide whether the lines correspond.Source: B. Feature Matching · pp. 5–6

C. 3D Feature Processing

  • Stereo-matched keypoints are triangulated from disparity, while unmatched keypoints use multi-frame observations; line features require separate treatment because they have more degrees of freedom and are easier to degenerate during triangulation.Projection uses a pinhole or fisheye camera model depending on the camera.Source: C. 3D Feature Processing · p. 6

1) 3D Line Representation:

  • Plücker coordinates represent 3D lines for triangulation, transformation, and projection, but their six dimensions exceed a line’s four degrees of freedom and can increase optimization cost and numerical instability.The system also uses an orthonormal representation for 3D lines; the conversion can be done with QR decomposition.Source: 1) 3D Line Representation · p. 6

2) Triangulation:

The system initializes 3D lines from image observations, transforms them for image reprojection, and selects keyframes to balance computational efficiency and tracking robustness.

2) Triangulation:

  • Triangulation initializes a 3D line either by intersecting planes back-projected from two image segments or, when that method fails, from two triangulated points on the line.The point-based fallback is efficient because those points are already triangulated.Source: 2) Triangulation · pp. 6–7

3) Re-projection:

  • Plücker coordinates transform 3D lines from the world frame to the camera frame before projection onto the image plane.The transformation uses the world-to-camera rotation and translation.Source: 3) Re-projection · p. 7

D. Keyframe Selection

  • Keyframe selection balances sparse keyframes for lower resource use against the greater tracking-failure risk of sparser keyframes.A frame is selected when specified parallax or tracked-feature-count conditions are met.Source: D. Keyframe Selection · p. 7

E. Local Graph Optimization

Local bundle adjustment optimizes a graph of keyframes, map points, and 3D lines using visual constraints and, when available, IMU constraints.

E. Local Graph Optimization

  • A local graph of neighboring keyframes, map points, and 3D lines is optimized after each new keyframe is inserted, using point, line, and optional IMU constraints.The graph contains pose-constraint edges.Source: E. Local Graph Optimization · p. 7

1) Point Re-projection Error:

  • Point reprojection error compares an observed map-point location with its camera-projected location.The camera projection is denoted by π.Source: 1) Point Re-projection Error · p. 7

2) Line Re-projection Error:

  • Line reprojection error uses the distances from the observed line's two endpoints to the reprojected 3D line.The endpoint-to-line distance is defined through equation (3).Source: 2) Line Re-projection Error · p. 7

3) IMU Residuals:

  • IMU measurements are pre-integrated between keyframes, with sensor biases modeled as constant over that interval and gravity and initial bias values estimated during initialization.The factor graph is optimized with g2o and Levenberg–Marquardt, rejecting point and line outliers with large residuals.Source: 3) IMU Residuals · pp. 7–8

F. Initial Map

The map is saved for later processing and reuse; offline optimization detects loops, merges duplicate features, and jointly refines the map.

F. Initial Map

1) Loop Closure Detection:

  • Loop closure detection retrieves candidates with DBoW2 and matches features with LightGlue, using a keypoint vocabulary trained on 35k images.The vocabulary has four layers with ten nodes per layer, totaling 10,000 words.Source: 1) Loop Closure Detection · p. 8

2) Map Merging:

  • Map merging identifies duplicate points through matched loop-pair keypoints and duplicate 3D lines when they share more than three associated map points.Duplicate features are reduced to one retained map feature.Source: 2) Map Merging · p. 8

3) Global Bundle Adjustment:

  • Global bundle adjustment jointly optimizes all keyframes and features with loop-closure residuals, first for 50 iterations without outlier rejection and then for 40 with rejection.For maps containing more than 80k map points, pose graph optimization precedes this step.Source: 3) Global Bundle Adjustment · p. 8
  • Offline global bundle adjustment runs after all loop closures are detected, reducing optimization iterations compared with systems that perform it after each new loop.For maps with more than 80k map points, pose graph optimization is performed before global bundle adjustment; it adjusts keyframe poses.Source: 3) Global Bundle Adjustment · p. 8

4) Scene-Dependent Vocabulary:

  • The scene-dependent junction vocabulary is trained on map keyframes and contains 1,000 words in a roughly 1 MB vocabulary.Its training database is generally smaller than that used for the keypoint vocabulary.Source: 4) Scene-Dependent Vocabulary · pp. 8–9

5) Optimized Map:

The optimized map is reused for relocalization across lighting conditions through a staged candidate-filtering pipeline that combines appearance and structural information.

5) Optimized Map:

  • The optimized map adds bag-of-words data, a global co-visibility graph, and a scene-dependent junction vocabulary, while duplicate-feature fusion keeps its size similar to the initial map.It is saved for subsequent map reuse.Source: 5) Optimized Map · p. 9

1) The First Step:

  • The first relocalization step detects keypoints, junctions, and lines in the query image and retains all retrieved candidates for the next step.Unlike loop-closure candidate selection, it does not filter candidates using the co-visibility graph.Source: 1) The First Step · p. 9

2) The Second Step:

  • The relocalization pipeline uses structural lines and a similarity score combining appearance and structure, supporting performance in illumination-challenging scenes while retaining efficient DBoW retrieval.The paper identifies structural information as invariant to lighting conditions.Source: 2) The Second Step · p. 9, B. Map Reuse · p. 9
  • The second step filters candidates using junction matches and line structure, then retains the top three by combined similarity; it takes 0.7 ms.The similarity combines keypoint, junction, and structure-graph similarity.Source: 2) The Second Step · p. 9

3) The Third Step:

The relocalization pipeline selects the query frame’s best-matching candidate and estimates its pose with PnP and RANSAC. The experiments assess PLNet’s line detection against wireframe-parsing baselines using structural accuracy and speed metrics.

3) The Third Step:

  • The query frame is matched to retained candidates, the one with the most inliers is selected, and PnP with RANSAC estimates pose; more than 20 inliers validates it.LightGlue performs candidate matching, after which query keypoints are associated with map points.Source: 3) The Third Step · pp. 9–10

VII. EXPERIMENTS

  • The evaluation covers line detection, mapping accuracy, three illumination scenarios, day/night map reuse, ablations, and system efficiency.Most experiments use an Intel i9-13900 CPU and NVIDIA GeForce RTX 4080 GPU; efficiency is also tested on Jetson Orin.Source: §VII · p. 10

1) Datasets and Baseliens:

  • The experiments evaluate line detection on Wireframe and YorkUrban, using 5,000 Wireframe training images, 462 test images, and 102 YorkUrban test images resized to 512 × 512.PLNet is compared with AFM, AFM++, L-CNN, LETR, F-Clip, ELSD, and HAWPv2.Source: 1) Datasets and Baseliens · p. 10

2) Evaluation Metrics:

  • Structural average precision at thresholds 5, 10, and 15 measures line-detection accuracy, while frames per second measures efficiency.The corresponding accuracy metrics are sAP5, sAP10, and sAP15.Source: 2) Evaluation Metrics · p. 10
  • PLNet detects structural lines, whereas SOLD2 detects more general high-gradient lines, including patterns on floors and walls.The figure marks detected line features in red and their endpoints in green.Source: Figure 7 · p. 10

3) Results and Analysis:

PLNet achieves comparable line-detection accuracy to state-of-the-art methods, with better generalization reported on YorkUrban and grayscale inputs. AirVIO leads most no-loop EuRoC sequences, while AirSLAM’s loop detection substantially reduces its average error.

3) Results and Analysis:

  • PLNet ranks second on Wireframe and first on YorkUrban; its gaps to the Wireframe leader are 0.5, 0.5, and 0.4 sAP points, while its YorkUrban margins are 0.4, 0.8, and 0.9.The comparisons are for sAP5, sAP10, and sAP15, respectively.Source: 3) Results and Analysis · p. 10
  • On grayscale inputs, PLNet significantly outperforms HAWPv2 on both datasets; the authors attribute its stronger generalization to a backbone trained on a large, diverse dataset.On the color-input Wireframe training dataset, HAWPv2 outperforms PLNet, but PLNet surpasses it on YorkUrban.Source: 3) Results and Analysis · pp. 10–11
  • PLNet remains among the fastest line detectors despite also detecting keypoints, processing each image only 0.86 ms slower than HAWPv2.The reported PLNet speed includes detecting both keypoints and lines, unlike the line-only baselines.Source: 3) Results and Analysis · p. 11

B. Mapping Accuracy

  • Without loop detection, AirVIO has the best results on 8 of 11 EuRoC sequences and 20% lower average translational error than Kimera-VIO; loop detection reduces AirSLAM’s average error by 74%.With loop detection, AirSLAM performs comparably to ORB-SLAM3 and DROID-SLAM.Source: Table II · p. 11, B. Mapping Accuracy · p. 12

C. Mapping Robustness

  • The robustness evaluation compares representative systems across onboard, dynamic, and low-light illumination scenarios.The selected baselines include ORB-SLAM3, DROID-SLAM, Basalt, Kimera, and OKIVS.Source: C. Mapping Robustness · p. 12

1) Onboard Illumination:

AirSLAM is evaluated under onboard, dynamic, and simulated low illumination. It performs strongly in the first two settings and comparably to Basalt and DROID-SLAM in low light, while the authors relate system differences to the lighting conditions and feature demands.

1) Onboard Illumination:

  • AirSLAM is most accurate on 7 of 9 OIVIO onboard-illumination sequences and has the smallest average error; onboard lighting has almost no impact on AirSLAM or ORB-SLAM3.The comparison also reports reduced accuracy for OKVIS, Basalt, and PL-SLAM, with Kimera experiencing tracking failures and large drift on three sequences.Source: 1) Onboard Illumination · p. 12, Table IV · p. 12

2) Dynamic Illumination:

  • AirSLAM achieves the best results on 7 of 10 UMA-VI dynamic-lighting sequences, with average RMSE about 0.441 m versus 1.134 m for OKVIS and 0.724 m for Basalt.OKVIS and Basalt complete all sequences, while PL-SLAM and ORB-SLAM3 fail on most.Source: 2) Dynamic Illumination · p. 13

3) Low Illumination:

  • The Dark EuRoC sequence is generated by applying 12 gamma-based brightness adjustments to EuRoC’s “V2 01 easy” sequence, with levels made progressively darker.L0 is the original sequence.Source: 3) Low Illumination · p. 13
  • Under progressively darker Dark EuRoC conditions, AirSLAM performs comparably to Basalt and DROID-SLAM; ORB-SLAM3 and OKVIS fail or incur large drift at L10 and L11.Optical-flow methods are more stable than feature-based methods in this low-light scenario.Source: 3) Low Illumination · p. 13

4) Result Analysis:

  • The authors report that descriptor-based systems are more robust in dynamic lighting, while direct methods are more stable in low light because corners are easier to detect than high-quality features and descriptors.AirSLAM achieves relatively high accuracy across the three illumination-challenging scenarios.Source: 4) Result Analysis · pp. 13–14

D. Map Reuse

The map-reuse evaluation uses daytime maps and nighttime query images, comparing AirSLAM with visual-place-recognition pipelines. AirSLAM offers higher average recall and speed, while performance remains constrained for pairs with little or no overlap.

1) Dataset:

  • Daytime “abandoned factory” sequences build the maps, while nighttime images within 3 m and 15° of mapping images serve as queries; selected pairs may lack valid overlap.The evaluation uses TartanAir stereo sequences and accurate ground truth.Source: 1) Dataset · p. 14

2) Baseline:

  • The comparison includes Hloc pipelines built from global feature retrieval, local feature extraction, and local feature matching; traditional vSLAM and ACE had recall below 1%.All methods use 400 local features and retrieve three candidates during coarse localization.Source: 2) Baseline · p. 14, 3) Results · p. 14

3) Results:

  • AirSLAM’s average relocalization recall is 4.2% higher than the second-best algorithm and it is about 2.4 times faster; it has the best results on 5 of 11 sequences.Success requires the estimated pose to be within 2 m and 15° of ground truth.Source: 3) Results · p. 15

4) Analysis:

  • AirSLAM maintains 70%–90% recall on most sequences, while VPR methods vary from near 100% on some sequences to below 40% on others.AirSLAM uses DBoW and low-level local features, yielding similar performance on image pairs with distinct landmarks and pairs without them.Source: 4) Analysis · p. 15
  • Neither AirSLAM nor VPR methods can process image pairs with little overlap because both rely on local feature matching; such pairs are abundant in P011.VPR methods also perform poorly on pairs with overlap but without large buildings or landmarks.Source: 4) Analysis · p. 15

1) Line Feature In Mapping:

  • Adding line features reduces mapping RMSE by 34.8%, 19.4%, 19.1%, and 17.1% on EuRoC, OIVIO, UMA-VI, and Dark-EuRoC, respectively.The authors attribute the improvement to lines being observed across multiple frames and tracked using sets of points.Source: 1) Line Feature In Mapping · p. 15, Table VI · p. 15

2) Structure Graph In Relocalization:

AirSLAM combines learned point-and-line feature processing with geometric optimization and reports relocalization, efficiency, and accuracy advantages. Its efficiency results span PC odometry, map building, and Jetson deployment, while its line-feature dependence makes it best suited to man-made environments.

2) Structure Graph In Relocalization:

  • With candidate counts of 3, 5, and 10, the structure-graph system improves average relocalization recall by 9.6%, 6.0%, and 3.8%, respectively, over w/o G.The evaluation uses the TartanAir Day/Night Localization dataset; junctions, line features, and structure graphs filter relocalization candidates.Source: 2) Structure Graph In Relocalization · p. 15

F. Efficiency Analysis

1) Odometry Efficiency:

  • 73 FPS on a PC, with around 3GB GPU memory versus about 8GB for DROID-SLAM, is reported for AirSLAM odometry.Loop detection and global bundle adjustment are disabled for the method comparison.Source: 1) Odometry Efficiency · p. 16, Figure 14 · p. 16

2) Mapping Efficiency:

  • AirSLAM is much more efficient than Hloc for map building, especially as input images increase.Mapping time measures initial map building and offline map optimization on TartanAir; Hloc builds monocular point-only maps without real scale, while AirSLAM builds stereo-and-IMU point-line maps with scale.Source: 2) Mapping Efficiency · p. 16, Figure 14 · p. 16

3) Embedded Platform:

VIII. CONCLUSION

  • AirSLAM outperforms other SOTA vSLAM systems in accuracy, efficiency, and robustness in illumination-challenging environments.Source: §VIII · p. 17
  • AirSLAM relies on enough line features and is best suited to man-made environments; in unstructured settings, it degrades to point-only SLAM.Source: §VIII · p. 17

Related papers

Loading 2408.03520v4…