Source-linked AI summary
Robust Visual SLAM with Point and Line Features
Xingxing Zuo, Xiaojia Xie, Yong Liu, Guoquan Huang
TL;DR
The paper addresses the challenge of robust visual SLAM when point features alone provide limited geometric information and line features require suitable parameterization and optimization. It proposes a point-and-line SLAM system with improved line matching, orthonormal line representation, and analytical Jacobians, and reports stronger performance across synthetic and real-world evaluations.
Problem
Point-only visual SLAM provides limited geometric information in some environments, while incorporating line features introduces data-association, parameterization, and Jacobian-computation challenges.
Method
The system combines point and line re-projection errors, improves line extraction and matching, and uses orthonormal line parameters with analytically derived Jacobians.
Results
The proposed system outperforms state-of-the-art methods across various synthetic and real-world scenarios, with improved performance over ORB-SLAM2 in selected KITTI sequences.
Takeaways & Limitations
Fusing point and line features produces more robust estimation, and the system operates in real time.
Abstract
from arXiv · showhide
In this paper, we develop a robust efficient visual SLAM system that utilizes heterogeneous point and line features. By leveraging ORB-SLAM [1], the proposed system consists of stereo matching, frame tracking, local mapping, loop detection, and bundle adjustment of both point and line features. In particular, as the main theoretical contributions of this paper, we, for the first time, employ the orthonormal representation as the minimal parameterization to model line features along with point features in visual SLAM and analytically derive the Jacobians of the re-projection errors with respect to the line parameters, which significantly improves the SLAM solution. The proposed SLAM has been extensively tested in both synthetic and real-world experiments whose results demonstrate that the proposed system outperforms the state-of-the-art methods in various scenarios.
I. INTRODUCTION
The paper develops a robust visual SLAM system that combines point and line features to provide stronger geometric constraints, addressing line-feature parameterization and Jacobian-computation challenges.
- Point and line features can provide more geometric constraints than either feature type alone, while line features are less sensitive to lighting variation and position ambiguity.
- Integrating line features into graph-based SLAM is challenging because spatial lines are often over-parameterized.
- The system uses a unified cost function that combines point and line re-projection errors.
- An improved line-feature extraction and matching method merges suitable line segments to make data association more robust and accurate.
- The back-end uses an orthonormal minimal representation for spatial lines and analytically computes Jacobians of line re-projection errors.
- The complete system includes stereo matching, frame tracking, local mapping, bundle adjustment for points and lines, and point-line loop detection.
III. DETECTION AND REPRESENTATION OF LINE FEATURES
The system improves line-feature extraction and matching, then represents 3D lines with a minimal orthonormal parameterization for SLAM optimization.
- Extraction and description: LSD can divide one physical line into multiple segments, causing failures in line-feature matching and tracking.
- Extraction and description: The improved detector merges segments with similar direction and sufficiently small endpoint and midpoint distances.The merged segments are represented with the 256-bit LBD descriptor.
- Line feature matching: Matched line features must satisfy angular, length-ratio, overlap, and LBD-descriptor distance constraints.
- Geometric representation: Plücker coordinates represent a 3D line with six values comprising its direction and plane-normal vectors.
- Geometric representation: Because 3D lines have four degrees of freedom, orthonormal representation provides a minimal four-parameter alternative to over-parameterized Plücker coordinates.
- Geometric representation: The system uses orthonormal representation in back-end optimization, while retaining Plücker coordinates for projection, endpoint trimming, and line initialization.
IV. GRAPH OPTIMIZATION WITH POINT AND LINE MEASUREMENTS
The system incorporates line measurements into a graph-based visual SLAM framework while handling point measurements using the standard ORB-SLAM approach.
- Line measurements are integrated into the graph-based visual SLAM system, while point measurements follow standard treatment such as ORB-SLAM.
A. Measurement Models of Point and Line Features
The measurement model transforms 3D lines into the camera frame, projects them into the image, and constructs point- and line-based graph optimization errors. The resulting cost uses inverse covariances and robust Huber functions for point and line observations.
- Line projection: The world-to-camera transformation uses rotation and translation components to convert each 3D line into camera-frame Plücker coordinates.The transformation is denoted Tcw ∈ SE(3), with Rcw ∈ SO(3) and tcw ∈ R3.
- Line projection: The camera-frame 3D line is projected onto the image plane using the camera’s known intrinsic parameters and a line projection matrix.
- Graph optimization: The graph model represents camera poses, 3D points, and 3D lines as vertices, with pose-point and pose-line edges built from front-end data associations.
- Graph optimization: Re-projection errors for point and line observations form a robust cost using inverse covariance matrices and Huber loss functions.The point and line terms are denoted by Σp^-1, Σl^-1 and ρp, ρl, respectively.
B. Jacobian of Line Re-projection Error
The paper derives analytical Jacobians for line re-projection errors under orthonormal line parameterization, including pose and line-parameter updates, enabling iterative graph optimization.
- The paper derives analytical Jacobians of line re-projection errors with respect to pose changes and the four-dimensional orthonormal line update.The line error is defined from the back-projected line, and the final derivatives are obtained through the chain rule.
- The Jacobian with respect to the back-projected line uses the line's image coordinates and the matched segment endpoints.The endpoints are represented by homogeneous image coordinates, while the projected 3D line is written using camera calibration and line-normal components.
- Pose perturbations are divided into translation and rotation components when deriving the Jacobian with respect to pose.The translation and rotation perturbations are handled separately before stacking their Jacobians.
- The resulting analytical Jacobians can be used with iterative solvers such as Gaussian-Newton for graph optimization.
A. System Implementation
The system is implemented as a graph-based visual SLAM architecture built on ORB-SLAM2, with separate threads for tracking, local mapping, and loop closing.
- The proposed architecture uses three parallel threads: Tracking, Local Mapping, and Loop Closing.A global bundle-adjustment thread starts after loop closing finishes.
- The graph-based system incorporates both point and line features throughout its visual SLAM architecture.
1) Tracking:
Tracking extracts and matches point and line features, estimates motion with point-line constraints, and handles partial line visibility during projection and matching.
- Tracking: Each stereo frame is processed in parallel to detect ORB point features and LSD-detected, LBD-described line features.Features are classified as stereo or monocular according to whether left-image features find matches in the right image.
- Tracking: Tracking first estimates pose from adjacent-frame correspondences, then refines it against the local map using motion-only bundle adjustment.The pose optimization uses the Levenberg-Marquardt algorithm.
- Tracking: A constant-velocity prior projects map points and lines into the current frame to create additional associations and constrain guided matching.
- Tracking: The line-visibility procedure transforms 3D lines, removes lines behind the camera, intersects partially visible lines with the viewing plane, and clips projections with Liang-Barsky.This accounts for the fact that a 3D line may be only partially observed in the image.
- Tracking: After matching, keyframe insertion follows ORB-SLAM2 policy with additional conditions based on line features.
- Tracking: Local mapping triangulates points and lines, removes outlier landmarks, deletes redundant keyframes, and optimizes the local map with bundle adjustment.The backend models 3D lines as infinite spatial lines, so their endpoints do not affect the final optimization result.
3) Loop Closing and Global BA:
Loop closing combines point- and line-feature vocabularies for candidate detection and pose correction, while experiments evaluate accuracy, drift, and runtime on synthetic and real data.
- Loop Closing and Global BA: Loop detection builds separate DBOW vocabularies for ORB and LBD features and combines their similarity scores using an empirical scene-dependent weight.Candidate correspondences are refined with temporal consistency before transformation estimation.
- Loop Closing and Global BA: The system estimates loop transformations with point correspondences and EPnP-RANSAC, falling back to a stereo-view line-based method when necessary.Pose-graph optimization then corrects accumulated drift.
- Synthetic data: In the synthetic scene, fused point-line odometry outperforms point-only and line-only methods, especially when few point features are available.The experiment uses 25 lines, variable point counts, one-pixel Gaussian observation noise, and RPE as the evaluation metric.
- Real data: On it3f without loop closure, PL-SLAM performs better than ORB-SLAM2, while SLSLAM reports the smallest vertical drift at -0.5455 meters.The authors associate degraded point performance with low texture, reflective surfaces, and ceiling-light effects, and note that unweighted point errors can affect the fused system.
- Real data: For 640×480 images, the proposed system requires 42.401ms for feature extraction and stereo matching and tracks at 15.1 frame/s.ORB-SLAM2 requires 32.15ms for the corresponding extraction and matching stage on it3f.
- Real data: On KITTI sequences 03, 04, and 10, PL-SLAM shows acceptable performance and improves over the original ORB-SLAM2 in the reported comparisons.Loop detection is disabled for the comparison, and RPE and ATE are used as evaluation criteria.
VI. CONCLUSIONS
The paper presents a graph-based visual SLAM approach that fuses point and line features, using orthonormal line representation and analytical Jacobians to improve estimation accuracy and robustness. The authors report robust real-time operation, while noting that SLSLAM source-code availability limited one KITTI comparison.
- The proposed graph-based visual SLAM fuses point and line features to improve accuracy and robustness.
- Lines are represented orthonormally during optimization, and analytical re-projection-error Jacobians are derived for line parameters.
- Experiments support more robust estimation from fusing point and line features in synthetic and real-world scenes.
- The system is reported to operate in real time.
- SLSLAM was excluded from KITTI experiments because its front-end source code was unavailable.
APPENDIX
The appendix derives the Jacobian with respect to the rotation perturbation δφ by holding δρ at zero and transforming the 3D line accordingly. The derivation uses rotation and skew-symmetric-matrix identities.
- For the Jacobian with respect to δφ, the translational perturbation δρ is set to zero.
- A transformation containing the rotation perturbation δφ produces a new 3D line denoted L∗.
- The notation [.]× denotes the skew-symmetric matrix associated with a vector.
- The derivation uses the identity for rotating cross products, (Ra) × (Rb) = R(a × b), with R ∈ SO(3).