Source-linked AI summary
Appearance-Based Loop Closure Detection for Online Large-Scale and Long-Term Operation
Mathieu Labbé, François Michaud
TL;DR
Large-scale, long-term appearance-based loop closure detection becomes computationally difficult as maps grow. RTAB-Map manages locations between Working and Long-Term Memory, retrieving relevant stored locations while limiting processing time. Across diverse environments, it reports strong recall at 100% precision while satisfying real-time constraints, with a documented weakness when regions are observed only once and then left for long periods.
Problem
As maps grow, comparing new observations with all stored locations increases processing time and can threaten online real-time operation.
Method
RTAB-Map keeps recent and frequently observed locations in Working Memory, transfers others to Long-Term Memory, and retrieves associated long-term locations after Working Memory matches.
Results
RTAB-Map achieves good recall at 100% precision across diverse large-scale environments while satisfying real-time constraints independently of environment scale.
Takeaways & Limitations
Retrieval supports adequate recall even when many perceived locations are transferred to Long-Term Memory, while online processing remains available for each acquired image.
Takeaways & Limitations
A region seen only once can be transferred entirely to Long-Term Memory after extended travel elsewhere, causing duplicated locations when the robot returns.
Abstract
from arXiv · showhide
In appearance-based localization and mapping, loop closure detection is the process used to determinate if the current observation comes from a previously visited location or a new one. As the size of the internal map increases, so does the time required to compare new observations with all stored locations, eventually limiting online processing. This paper presents an online loop closure detection approach for large-scale and long-term operation. The approach is based on a memory management method, which limits the number of locations used for loop closure detection so that the computation time remains under real-time constraints. The idea consists of keeping the most recent and frequently observed locations in a Working Memory (WM) used for loop closure detection, and transferring the others into a Long-Term Memory (LTM). When a match is found between the current location and one stored in WM, associated locations stored in LTM can be updated and remembered for additional loop closure detections. Results demonstrate the approach's adaptability and scalability using ten standard data sets from other appearance-based loop closure approaches, one custom data set using real images taken over a 2 km loop of our university campus, and one custom data set (7 hours) using virtual images from the racing video game ``Need for Speed: Most Wanted''.
I. INTRODUCTION
Appearance-based loop closure detection can become too slow as maps grow because new observations are compared with increasingly many stored locations. RTAB-Map addresses this through memory management that keeps processing within real-time constraints while retaining access to long-term locations.
- Loop closure detection recognizes whether a current observation corresponds to a previously visited location, enabling associations within a SLAM map.
- Growing maps increase the time required to compare new observations with all previously visited locations, potentially exceeding image acquisition time.
- RTAB-Map dynamically limits searched locations using processing time as the criterion for Working Memory capacity.
- The method keeps recent and frequently observed locations in Working Memory and transfers other locations to Long-Term Memory.
- When a Working Memory match occurs, associated Long-Term Memory locations can be retrieved and updated for further loop closure detection.
- RTAB-Map defines real-time operation as processing each acquired observation within the time available before the next acquisition, independently of mapped-environment scale.
II. RELATED WORK
Prior appearance-based methods use visual representations and selected map locations to support recognition, but their real-time scalability remains tied to explored-environment size. RTAB-Map instead manages locations across short-, working-, and long-term memories while retrieving relevant long-term locations when needed.
- Appearance-based loop closure detection: Bag-of-words represents images with visual words and links words to related images for efficient large-scale comparison.
- Appearance-based loop closure detection: Pre-trained vocabularies can make image comparison fast, but real-time satisfaction remains limited by the maximum mapped-environment size.
- Limitations of existing approaches: Perceptual aliasing, environmental changes, and weak visual information can cause missed recognition and additional map locations, affecting real-time constraints.
- RTAB-Map approach: RTAB-Map uses a limited location set for loop closure detection while retaining access to the entire map when necessary.
- RTAB-Map approach: Each location stores an image signature, time index, and weight, with graph links representing temporal neighbors or spatial loop closures.
- RTAB-Map approach: Sensory Memory creates locations, Short-Term Memory updates and merges recent similarities, and Working Memory detects spatial loop closures.
- RTAB-Map approach: The loop closure process updates a Bayesian filter in Working Memory, selects an accepted hypothesis, retrieves Long-Term Memory locations, and transfers locations when processing exceeds its threshold.
A. Location Creation
RTAB-Map creates image-based locations by extracting SURF features, quantizing their descriptors into an incrementally built visual vocabulary, and storing the resulting signature with a time index and weight.
- The method uses an incrementally constructed bag-of-words vocabulary rather than a pre-trained vocabulary to avoid environment-specific training.
- SURF features are extracted and filtered by feature response, with at most TmaxFeatures retained to keep image word counts similar.
- Features with too few detected descriptors are marked as bad signatures and excluded from loop closure processing.
- SURF descriptors are quantized into visual words using nearest-neighbor distance-ratio matching and randomized kd-trees.
- A created location contains the quantized image signature, time index, and an initial weight of zero, linked bidirectionally to the preceding location.
B. Weight Update
The method updates location weights and Bayesian loop-closure beliefs from visual similarity, while modeling both previously visited and new-location hypotheses over the changing Working Memory.
- Observation model: Visual similarity compares the current signature with each candidate signature using matched word pairs relative to their total word counts.The resulting score is used to evaluate the observation likelihood for each loop-closure state.
- Weight update: When similarity exceeds Tsimilarity, the compared location is merged into the current location and its weight, links, and vocabulary references are updated.The compared location is deleted from STM after its neighbor and loop-closure links are redirected.
- Bayesian belief update: The Bayesian filter estimates whether the current location matches a stored Working Memory location or represents a new location.Its state distribution includes loop-closure hypotheses and a new-location state, with locations drawn from WM and STM.
- New-location hypothesis: A high new-location likelihood indicates that the current signature is not similar to a particular Working Memory location.The new-location likelihood is evaluated relative to the mean and standard deviation of similarity scores.
- Transition model: The transition model predicts the next hypothesis distribution from the previous distribution and the robot’s motion between observations.The model assigns probabilities to new-location events, loop closures after new locations, and loop closures near a previous loop closure.
D. Loop Closure Hypothesis Selection
Loop-closure hypothesis selection accepts the highest-probability candidate only when the new-location probability falls below the loop-closure threshold.
- Hypothesis acceptance: The highest loop-closure hypothesis is accepted when p(St = −1|Lt) is lower than Tloop.The accepted current and old locations are linked, the current weight increases by the old weight, and the old weight is reset to 0.
E. Retrieval
After detecting a loop closure, RTAB-Map retrieves selected neighboring locations from LTM into WM and restores their vocabulary references as needed.
- Retrieval trigger: Neighbors of the highest loop-closure hypothesis that are absent from WM are transferred back from LTM after detection.LTM is implemented as a SQLite3 database whose link table distinguishes neighbor and loop-closure links.
- Vocabulary update: Retrieving a location updates the visual vocabulary with references and descriptors associated with its stored signature.Unmatched descriptors generate new vocabulary words, while matched descriptors can replace corresponding old words.
- Retrieval policy: At most two locations are retrieved per iteration, prioritizing nearby locations in time over nearby locations connected through loop closures.This ordering favors temporal neighbors while the robot is moving and eventually includes spatial neighbors when it remains stationary.
F. Transfer
The transfer mechanism keeps processing within the time threshold by moving selected WM locations and their vocabulary references to LTM, while protecting recent high-weight locations for future closures.
- Transfer trigger: When image processing exceeds Ttime, the oldest locations among the lowest-weight WM locations are transferred to LTM.Neighbors of the highest loop-closure hypothesis remain protected so the Bayesian filter can evaluate nearby hypotheses.
- Real-time constraint: If Ttime exceeds image acquisition time, the algorithm uses an image rate corresponding to Ttime with 100% CPU usage.
- Vocabulary control: Transferred locations remove their word references from the visual vocabulary, allowing vocabulary size and indirectly WM size to regulate processing time.Building the nearest-neighbor index is identified as RTAB-Map’s most expensive step.
- Protection policy: A subset of the highest-weighted recent locations remains in WM after the last detected loop closure to preserve coverage of newly explored regions.The protected subset is defined by Trecent × NWM; locations beyond that bound can be transferred using the ordinary age-and-weight criterion.
- Evaluation context: Performance is evaluated using precision-recall metrics, with recall requirements depending on the accuracy of the associated SLAM odometry.The paper notes that metric SLAM with excellent odometry may require about 1% recall, whereas less accurate or absent odometry requires more.
- Transfer implementation: The transfer process selects WM locations by weight and age, excludes recent or retrieved locations, and asynchronously saves transferred data to LTM.Locations and vocabulary words without remaining WM references are moved through a background-thread trash process.
A. Community Data Sets
RTAB-Map was evaluated on ten community data sets, with recall measured at 100% precision and precision-recall curves reported across the data sets. It achieved higher recall than comparable approaches while remaining within real-time processing limits.
- Data sets: The evaluation used ten community data sets, including indoor, outdoor, mixed, omnidirectional, and crowded environments.The listed sets include NewCollege, CityCentre, Lip6Indoor, Lip6Outdoor, 70 km, New-CollegeOmni, CrowdedCanteen, BicoccaIndoor, BovisaOutdoor, and BovisaMixed.
- Evaluation: Recall performance was defined as the maximum recall observed at 100% precision, with precision-recall curves shown for each data set.The curves are presented in Fig. 4.
- Results: RTAB-Map achieved better recall at 100% precision than other approaches using the same data sets, with improvements up to 54%.The reported comparison appears in Table II, with the largest improvement shown at the bottom of the table.
- Results: The maximum processing time remained below the image acquisition time for all community-data-set experiments.This is the paper's criterion for respecting real-time constraints.
B. Universit´e de Sherbrooke (UdeS) Data Set
The UdeS experiment tested RTAB-Map on a 2 km campus loop traversed twice under varied visual conditions. Memory management maintained online processing, while recall and loop-closure retrieval depended on the time budget and working-memory contents.
- Data set: The UdeS data set contains 5395 images captured at 1 Hz over 90 minutes during two traversals of a 2 km campus loop.The route included indoor and outdoor areas, roads, parkings, paths, trees, and a football field, with illumination and camera-orientation changes.
- Data set: The aerial-view figure marks the first traversal with a dotted line, the second with a nearby solid line, and the start/end point with a circle.Waypoint numbers include white dots indicating camera orientation; the illustrated recall result uses Ttime = 0.7 s.
- Processing time: With Ttime = ∞, maximum processing time reached 10.9 seconds, exceeding the 1-second image acquisition interval and preventing online use.Keeping all locations in Working Memory makes loop-closure detection operate over all previously visited locations.
- Performance: With Ttime <= 0.75 s, processing remained online, while recall ranged from 39% to 54% for Ttime ∈[0.95; 0.35] s.The recall variation was attributed to which locations were retained or retrieved in Working Memory.
- Loop-closure behavior: Changes in illumination or camera orientation produced insufficient loop-closure probabilities, although Retrieval could still recover appropriate transferred signatures in some cases.During Red paths, false-positive hypotheses remained below Tloop and were not accepted; Yellow paths indicated successful retrieval but insufficient probability.
- Processing time: At Ttime = 0.7 s, memory management was triggered after 444 images; average processing time was 0.67 s and maximum processing time was 0.87 s.After reaching the time threshold, processing remained close to Ttime.
C. “Need for Speed: Most Wanted” (NFSMW) Data Set
The NFSMW data set tests RTAB-Map under repeated traversals, changing illumination, and transitions to new areas. Recall reaches 60–100% at 100% precision, while processing remains within the 1 Hz real-time constraint.
- Data set: The NFSMW experiment used 25098 virtual 640×480 images acquired at 1 Hz during 7 hours of driving across two areas.Area 1 and Area 2 were each traversed about one hundred times, testing repeated observations and subsequent movement to new locations.
- Data set: Changing illumination included moving shadows, bright sunrises, and random storms, making long-term loop closure detection challenging.The same locations could appear under substantially different atmospheric and lighting conditions.
- Recall: Recall at 100% precision varied from 60% to 100% across Area 1 and Area 2 traversals.The first traversal of each area produced 0% recall as expected, while later performance varied with illumination conditions.
- Recall: After 41 traversals, average recall stabilized around 89% at 100% precision, with Tloop = 0.10.The ground truth contained 24800 loop closures, and the resulting global graph contained 2728 duplicated locations.
- Area transition: After the first Area 2 traversal, retrieval restored loop-closure performance to 100% recall at 100% precision on the second traversal.The first detected loop closure occurred at the first intersection encountered during the second traversal.
V. DISCUSSION
The discussion attributes RTAB-Map’s operation to memory management that bounds Working Memory while retaining access to Long-Term Memory. It also identifies scalability, feature-quality, and forgetting trade-offs.
- Memory management: Similarity occurrences provide a simple method for selecting locations to retain in Working Memory.The method has a limitation when an area is observed only once before the system moves elsewhere for a long period.
- Limitations: A single Area 1 traversal followed by 56 Area 2 traversals transferred all Area 1 locations to Long-Term Memory, causing later duplicate locations when Area 1 was revisited.This illustrates the real-time trade-off in which infrequently visited locations may be forgotten from Working Memory.
- Complexity: RTAB-Map’s per-step costs include O(nw) Bayesian observation and hypothesis selection, O(log[wl + nl]) database retrieval, and O(nw) transfer selection.Location creation costs O(wwlogww) for building vocabulary kd-trees and O(logww) for descriptor quantization; weight update is O(1).
- Scalability: When Ttime is reached, Working Memory remains fixed to bound costs associated with ww and nw, but retrieval and transfer still depend on unbounded Long-Term Memory.Theoretically, Working Memory could eventually become null, disabling loop closure detection, although the observed logarithmic growth was small.
- Scalability: In the 7-hour NFSMW experiment, database-access time growth was unnoticeable despite a 3.1 GB database containing 6.3 million words and 25098 locations.Working Memory-size variation increased after approximately 12000 locations, mainly during the transition between the two areas.
- Limitations: SURF-feature quality limits performance under illumination, shadows, dynamic objects, and people because common features can obscure discriminative ones.The system requires at least one discriminative feature for loop closure and prefers missed closures over increased false positives in such conditions.
- Future work: Future improvements include exploiting Bayesian-filter sparsity, using more efficient nearest-neighbor structures, and triggering active localization when hypotheses decrease.These changes could retain more locations in Working Memory or help retrieve suitable locations from Long-Term Memory after camera-orientation changes.
VI. CONCLUSION
The conclusion reports that RTAB-Map uses memory management to support online loop closure detection over large-scale and long-term operation. It maintains a relatively constant Working Memory while using retrieval to preserve recall after locations move to Long-Term Memory.
- Conclusion: RTAB-Map meets the real-time constraints required for online large-scale and long-term operation through memory management.The conclusion states that online processing is achieved while keeping a relatively constant number of locations in Working Memory.
- Conclusion: Retrieval helps maintain adequate recall even when many perceived locations are transferred to Long-Term Memory and excluded from loop-closure detection.The paper identifies combining RTAB-Map with other approaches to build a complete SLAM system as future work.