Source-linked AI summary
UcoSLAM: Simultaneous Localization and Mapping by Fusion of KeyPoints and Squared Planar Markers
Rafael Munoz-Salinas, Rafael Medina-Carnicer
TL;DR
Monocular SLAM must handle unknown scale, weak or changing texture, repetitive environments, and unreliable relocalization. UcoSLAM fuses keypoints with squared fiducial markers, and experiments report better precision, robustness, and speed than ORB-SLAM2 and LDSO, with combined landmarks outperforming either alone in accuracy.
Problem
Monocular SLAM methods can suffer from unknown map scale, insufficient texture, pure-rotation failures, and weak relocalization under viewpoint changes or repetitive patterns.
Method
UcoSLAM fuses natural keypoints with squared fiducial markers and supports map initialization through keypoints, homography, fundamental matrices, or markers.
Results
UcoSLAM performs better than ORB-SLAM2 and LDSO in precision and speed, while combined keypoints and markers improve accuracy and reduce ambiguity in repetitive environments.
Takeaways & Limitations
Markers provide stable long-term references that support correct scale, relocalization, and tracking in repetitive environments while keypoints remain usable where markers are absent.
Takeaways & Limitations
Keypoint-based initialization can be difficult and time-consuming, and maps initialized from keypoints are scale-agnostic until a marker is found.
Abstract
from arXiv · showhide
This paper proposes a novel approach for Simultaneous Localization and Mapping by fusing natural and artificial landmarks. Most of the SLAM approaches use natural landmarks (such as keypoints). However, they are unstable over time, repetitive in many cases or insufficient for a robust tracking (e.g. in indoor buildings). On the other hand, other approaches have employed artificial landmarks (such as squared fiducial markers) placed in the environment to help tracking and relocalization. We propose a method that integrates both approaches in order to achieve long-term robust tracking in many scenarios. Our method has been compared to the start-of-the-art methods ORB-SLAM2 and LDSO in the public dataset Kitti, Euroc-MAV, TUM and SPM, obtaining better precision, robustness and speed. Our tests also show that the combination of markers and keypoints achieves better accuracy than each one of them independently.
1. Introduction
UcoSLAM combines keypoints with squared fiducial markers to address scale, texture, relocalization, and long-term tracking limitations in monocular SLAM. Experiments report improved precision, robustness, and speed compared with established methods.
- Motivation: Monocular SLAM methods can have unknown scale, fail under pure rotation, require texture, and relocalize poorly under viewpoint changes or repetitive patterns.These limitations are reported for ORB-SLAM2 and LDSO-style approaches.
- Proposed system: UcoSLAM fuses keypoints and squared fiducial markers while supporting operation with markers, keypoints, or both.This allows markerless operation while using detected markers when available.
- Proposed system: Markers automatically determine map scale, reduce visual ambiguity during relocalization, and provide stable references for long-term mapping.The paper contrasts their temporal stability with keypoints that may change substantially over time.
- Evaluation: The method was compared with ORB-SLAM2, LDSO, and SPM-SLAM on Kitti, Euroc-MAV, TUM, and SPM datasets.The evaluation considers pose-estimation accuracy and the percentage of correctly tracked frames.
- Results: The proposed method is reported as more precise, robust, and faster than ORB-SLAM2 and LDSO, while combining markers and keypoints improves precision over either alone.The implementation also supports map saving and sequential map generation for later use.
2. Related works
Related work spans natural-feature, direct photometric, and fiducial-marker SLAM. UcoSLAM combines keypoints and squared markers to retain complementary advantages while reducing their individual limitations.
- Natural-feature methods: PTAM and ORB-SLAM use natural keypoints for tracking and mapping, but monocular systems have unknown scale and can struggle with repetitive environments.ORB-SLAM uses keypoints for tracking and bag-of-words relocalization.
- Direct methods: Direct and sparse direct SLAM estimate pose by minimizing photometric error, but require camera calibration and historically offered limited relocalization.The cited discussion identifies these as continuing limitations of the approach.
- Fiducial-marker methods: Squared planar markers encode unique identities and provide four corner correspondences sufficient for camera pose estimation from one marker.Marker-based approaches can improve precision, robustness, and speed, but many estimate pose only relative to a single marker.
- UcoSLAM: UcoSLAM combines keypoints and squared markers to obtain correct scale, stable temporal references, and robust tracking in repetitive environments without requiring markers to operate.Markers can still improve tracking when available.
3. System Overview
The system represents an environment map with keyframes, triangulated map points, squared markers, connection information, and a recognition database, then processes frames through a combined keypoint-marker SLAM pipeline.
- System map: The environment map contains keyframes, map points, markers, a keyframe connection graph, and a keyframe recognition database.Map points are triangulated 3D points with viewing directions and representative descriptors.
- Map elements: Each map point records its keyframe observations, while each marker stores its size, pose, and four reference-frame corners.Marker observations record the image coordinates of the four detected corners.
- Operational pipeline: The pipeline combines keypoints and squared planar markers for initialization, tracking, mapping, and loop closure.Initialization may use homography, fundamental matrices, or one or more markers.
- Implementation: The system accepts binary or real descriptors and can run its processing components in parallel or sequentially.ORB is reported as a speed-robustness trade-off, while the number of threads is adjustable.
- Map serialization: An efficient mixed vector structure supports constant-time map-element access without copying data when storage grows.Freed positions are reused, and additional fixed-size vectors are created only when needed.
- Pose ambiguity: Pose ambiguity can arise when noisy marker-corner localization gives two camera poses with similar reprojection errors.The ambiguity is especially relevant when a marker is far from the camera.
4. Detailed system description
This section introduces the detailed components of the system described in the preceding overview.
- The section explains in detail the components involved in the previously outlined system.
- The detailed description follows the complete system process rather than a single isolated component.
- The section expands the system overview into component-level descriptions.
4.1. Map Initialization
UcoSLAM initializes from two frames using either keypoints or markers, preferring marker initialization when both succeed because it provides correct scale and faster startup.
- The system tests keypoint-based and marker-based initialization on the first two frames, selecting a successful marker method when both succeed.If both methods fail, it retries using the first frame and subsequent frames.
- Keypoint initialization: Keypoint initialization chooses between homography and essential-matrix solutions using parallax and heuristics.This process can be difficult and time-consuming, and its resulting map is initially scale-agnostic.
- Marker initialization: Marker initialization can recover relative frame-marker pose under sufficient parallax, including ambiguous detections resolved from multiple frame observations.
- Marker initialization: Marker-based initialization produces a map in the correct scale and is faster than initialization using keypoints alone.
4.2. Tracking
Tracking estimates the current camera pose from map-point and marker correspondences by jointly minimizing their reprojection errors, with weighting and robust outlier handling.
- The current camera pose is initialized from the previous pose and refined by minimizing reprojection errors for observed map points and marker corners.
- Correspondence search: Tracking first matches points from the previous frame and reference keyframe, then searches neighboring keyframes for additional correspondences.
- Match validation: Map-point matches are filtered using viewing angle, scale-invariant distance, image bounds, descriptor distance, and duplicate-assignment checks.For duplicate assignments, the match with the smallest distance is retained.
- Marker selection: Only markers with valid poses observed near the reference keyframe are used for tracking.Visible markers may be excluded when their poses are unavailable or they are not in the relevant keyframe neighborhood.
- Pose optimization: The optimization combines marker and map-point reprojection errors, using a Huber function to reduce the influence of outliers.The relative marker weight is adjusted because map points are usually more numerous than markers.
- Pose optimization: When no valid markers are available for tracking, the marker weight is set to zero; as marker support increases, the two terms become more balanced.
4.3. Keyframe insertion
Keyframes are inserted selectively to add new marker or keypoint information while supporting reliable tracking. Marker observations can be incorporated immediately, with ambiguous marker poses refined from later keyframes.
- A keyframe is added when it contributes new marker or keypoint information needed for smooth and reliable tracking.
- A newly observed marker and its frame enter the map even when the marker pose cannot yet be estimated unambiguously.The marker pose is initially marked invalid and estimated later from additional observations.
- After a keyframe is added, local or global optimization integrates the newly available information.
- An unambiguously detected marker receives an initial pose that can be refined as further keyframes are added.
- When a marker remains ambiguous, multiple keyframes can later provide its pose through observations accumulated as the camera moves.At least two ambiguous keyframe observations can be used to obtain the marker pose.
- For new keypoints, the system searches neighboring keyframes for correspondences and uses epipolar constraints to reduce false matches.The policy aims to add only robust map points and also strengthen support for existing ones.
4.4. Map optimization
Map optimization integrates new observations and removes incorrect map points by adjusting keyframe poses, map-point locations, and marker poses. Local optimization limits updates to connected neighbors, while global optimization is reserved for broader corrections such as loop closures.
- Optimization updates keyframe poses, map-point locations, and marker poses whenever a frame is added or a loop closure is detected.The update integrates new information and can remove incorrect map points.
- Global optimization is a sparse Levenberg-Marquardt problem that can run in a separate Map Manager thread.
- After a new keyframe, optimization usually updates only map keyframes sharing points with it rather than the whole map.
- This reduced optimization over connected neighboring keyframes is called local optimization.
- The local problem is defined over the keyframes connected to a selected keyframe and the observations contained in those frames.
4.5. Keyframe culling
Keyframe culling controls map growth while preserving the observations needed to optimize both markers and keypoints. It protects geometrically valuable marker views and removes redundant frames only when sufficient higher-resolution support remains.
- Keyframe culling prevents unlimited map growth while accounting for both keypoints and markers.
- Three widely separated keyframes observing each marker are retained to support good triangulation during optimization.These selected frames cannot be removed.
- Remaining keyframes are removed only when at least τc% of their matched map points are observed in three other keyframes at equal or higher pyramid scale.The removed frame is treated as redundant because the other observations can replace it.
4.6. Loop closure detection and correction
Loop closure detection combines keypoint-based recognition with immediate marker-based detection to identify and correct accumulated drift. Marker closures are prioritized because leaving drift uncorrected can disrupt subsequent camera tracking.
- Keypoint loop closures are detected in a separate Map Manager thread using a bag-of-words database of inserted keyframes.
- A candidate keypoint loop is validated by estimating a rigid transform with map-point positions and RANSAC Perspective-n-Point, then checking the inlier count.
- Marker-based loop closures are detected immediately before processing another frame because uncorrected drift can cause camera tracking problems.
- For either loop-closure type, drift is corrected by optimizing Sim(3) keyframe poses and propagating the error along the keyframe path.
4.7. Relocalisation
Relocalization first uses known markers to estimate the camera pose, then refines it with map correspondences. If marker-based estimation is ambiguous, the keyframe database is used instead.
- Known markers are checked first for unambiguous camera-pose estimation after tracking failure.The marker-based estimate is then refined using map correspondences and camera-pose reestimation.
- When markers cannot provide an accurate pose, the keyframe database supports relocalization.
5. Experiments and results
The experiments compare UcoSLAM with established monocular SLAM methods using a measure that balances pose accuracy and tracking coverage. Across benchmark and repetitive-environment tests, UcoSLAM is reported as slightly better or more robust, with combined markers and keypoints performing best in the SPM evaluation.
- Experimental setup: The evaluation compares UcoSLAM with ORB-SLAM2 and LDSO on Kitti, Euroc-MAV, TUM, and SPM datasets.
- Measures for comparison: The proposed comparison evaluates both pose precision and the number of frames tracked.The methodology compares ATE values on frames tracked by both methods and accounts for each method’s total tracked frames.
- Measures for comparison: The score prioritizes ATE and assigns non-zero credit only when a method is at least as good in ATE as its counterpart.Significance thresholds prevent small differences from determining the comparison.
- Measures for comparison: The proposed measure is non-commutative, and pairwise scores do not necessarily sum to one.
- Monocular SLAM: UcoSLAM is slightly better than ORB-SLAM2, while LDSO performs worst in almost all comparisons.
- Marker and keypoint variants: In the SPM dataset, combined keypoints and markers outperform either keypoints or markers alone.The compared variants are UcoSLAM(kp), UcoSLAM(m), and UcoSLAM(kp+m).
- Large-scale mapping: In the repetitive-environment test, only UcoSLAM using both keypoints and markers created a map, with loop closure based exclusively on markers.The resulting map received only a qualitative evaluation because ground truth was unavailable.
- Computing times: UcoSLAM’s computing-time differences relative to ORB-SLAM2 were statistically significant, while its SLAM-time difference relative to LDSO was not significant.The averages exclude sequences with zero values.
6. Conclusions and future work
UcoSLAM combines keypoints with squared fiducial markers to improve robustness, precision, and speed over established methods. The system also supports map reuse after saving.
- UcoSLAM combines keypoints and squared fiducial markers as its central SLAM approach.
- The method performs better than ORB-SLAM2 and LDSO in precision and speed, while combined landmarks improve accuracy and reduce ambiguity in repetitive environments.
- The publicly available implementation can save maps and reuse them later, a feature identified as missing from ORB-SLAM2.