Source-linked AI summary
On the Performance of ConvNet Features for Place Recognition
Niko Sünderhauf, Feras Dayoub, Sareh Shirazi, Ben Upcroft, Michael Milford
TL;DR
The paper asks whether ConvNet features can support robust, real-time visual place recognition despite severe appearance and viewpoint changes. It systematically evaluates layers and three networks, combining semantic search-space partitioning with locality-sensitive hashing. The results show large speed-ups with limited performance loss, while different layers provide distinct robustness to appearance and viewpoint variation.
Problem
Robotic place recognition must handle appearance and viewpoint changes, but computer-vision ConvNet results do not directly establish suitability for robotic camera data and real-time operation.
Method
The paper evaluates ConvNet layers and three networks across four real-world datasets, combining semantic search-space partitioning with locality-sensitive hashing for large-scale matching.
Results
Two orders of magnitude speed-up is achieved with 99.6% feature-data compression while retaining 95% of place recognition performance; mid-level and top layers are respectively strongest for appearance and viewpoint changes.
Takeaways & Limitations
Semantic place-categorization networks and ConvNet layers provide complementary capabilities for robust place recognition under changing conditions.
Abstract
from arXiv · showhide
After the incredible success of deep learning in the computer vision domain, there has been much interest in applying Convolutional Network (ConvNet) features in robotic fields such as visual navigation and SLAM. Unfortunately, there are fundamental differences and challenges involved. Computer vision datasets are very different in character to robotic camera data, real-time performance is essential, and performance priorities can be different. This paper comprehensively evaluates and compares the utility of three state-of-the-art ConvNets on the problems of particular relevance to navigation for robots; viewpoint-invariance and condition-invariance, and for the first time enables real-time place recognition performance using ConvNets with large maps by integrating a variety of existing (locality-sensitive hashing) and novel (semantic search space partitioning) optimization techniques. We present extensive experiments on four real world datasets cultivated to evaluate each of the specific challenges in place recognition. The results demonstrate that speed-ups of two orders of magnitude can be achieved with minimal accuracy degradation, enabling real-time performance. We confirm that networks trained for semantic place categorization also perform better at (specific) place recognition when faced with severe appearance changes and provide a reference for which networks and layers are optimal for different aspects of the place recognition problem.
I. INTRODUCTION
The paper evaluates ConvNet features for robotic place recognition under appearance and viewpoint changes, while addressing the real-time demands of large-scale maps.
- Motivation: ConvNet features can serve as holistic image descriptors for visual place recognition.The paper investigates their utility across challenging robotic datasets.
- Main results: Higher ConvNet layers encode semantic place information and can partition the search space to reduce recognition time.This combines semantic representations with large-scale search optimization.
- Main results: Two orders of magnitude speed-up is achievable by replacing cosine distance with Hamming distance over hashed bitvectors, retaining 95% of place recognition performance.The hashing approach compresses feature data by 99.6%.
- Main results: Networks trained for semantic place categorization outperform an object-recognition ConvNet under severe appearance changes.The comparison covers three state-of-the-art ConvNets.
- Main results: Mid-level features are more robust to appearance changes, whereas top-layer features are more robust to viewpoint changes.The study evaluates these properties across multiple datasets and ConvNet layers.
A. Place Recognition
Place recognition research has moved from hand-crafted or pixel-level representations toward learned ConvNet features, motivating systematic evaluation for changing environments.
- Place recognition: Earlier approaches commonly rely on fixed hand-crafted features or raw pixels, while ConvNets provide learned features for recognition.ConvNets have outperformed classical hand-crafted approaches in object classification and detection.
- Place recognition: The paper extends prior ConvNet place-recognition work through systematic datasets, three-network comparisons, and algorithmic improvements for real-time performance.Its investigation separates ConvNet features from the matching strategy.
- Place recognition: AlexNet layer outputs are extracted as holistic image descriptors, with the network originally pretrained on ImageNet for object recognition.The architecture contains convolutional and fully connected layers.
B. Image Matching and Performance Measures
The evaluation uses single-image nearest-neighbor matching and precision-recall analysis across datasets selected for distinct appearance and viewpoint changes.
- Image matching: Place recognition uses single-image nearest-neighbor search with cosine distance between extracted feature vectors.The study avoids sequence search and other specialized matching improvements to measure baseline ConvNet performance.
- Performance measures: Performance is evaluated with precision-recall curves and F1 scores.A ratio-test threshold controls the precision-recall curves.
- Datasets: The four datasets are organized into groups with severe appearance changes, viewpoint changes, or both.Two datasets isolate severe appearance change, one emphasizes viewpoint change, and three include both variations.
- Performance measures: A match passes when the best-to-second-best distance ratio satisfies the ratio test; true positives fall within a dataset-dependent ground-truth frame window.Matches outside that window are false positives.
1) The Nordland Dataset:
The experiments use real-world traversals designed to expose appearance and viewpoint variation, including seasonal train footage and day/night campus recordings.
- The Nordland Dataset:: The Nordland dataset contains a 10-hour train journey recorded from the front cart across four seasons.It has no viewpoint variation, isolating seasonal condition changes.
- The Nordland Dataset:: Gardens Point contains two daytime traverses and one nighttime traverse, with recordings taken from opposite walkway sides.The setup combines appearance and viewpoint changes.
- The Nordland Dataset:: The campus dataset pairs robot Kinect footage recorded during the day with human GoPro footage recorded at dawn.The resulting appearance changes are especially significant outdoors.
4) The St. Lucia Dataset:
The St. Lucia dataset captures repeated traversals with daytime and multi-day appearance variation plus mild viewpoint changes. Under these conditions, conv3 and conv4 achieve near-perfect precision-recall performance.
- Nine St. Lucia traversals span varying daytimes and several weeks, with mild viewpoint variation from route changes.
- Conv3 and conv4 perform almost perfectly across the nine trials, with overlapping precision-recall curves near the upper-right corner.
- Conv3 remains reasonably effective despite extreme day-to-night appearance changes after nighttime images are contrast-enhanced and converted to grayscale.
IV. LAYER-BY-LAYER STUDIES
The layer-by-layer studies evaluate ConvNet features under severe appearance and viewpoint variation across several real-world datasets. Mid-level conv3 features are consistently more robust to appearance changes, while combined appearance and viewpoint changes remain difficult.
- Appearance Change Robustness: Experiments evaluate layer robustness against severe appearance changes across Nordland, Gardens Point, St. Lucia, and Campus datasets.
- Appearance Change Robustness: Combining extreme appearance and viewpoint changes causes place recognition performance to deteriorate and remains challenging for current systems.
- Appearance Change Robustness: On Campus, mid-level convolutional layers outperform fully connected layers fc6 and fc7 when recordings use different cameras and times of day.
- Appearance Change Robustness: Mid-level conv3 features are more robust to appearance changes than other layers, while lower and higher layers show inferior place recognition performance.
- Appearance Change Robustness: Single-image conv3 matching matches or exceeds SeqSLAM under severe appearance changes.
- Appearance Change Robustness: Across seasons on Nordland, conv3 performs acceptably with single-frame nearest-neighbor matching, whereas fc6 and fc7 fail completely.
B. Viewpoint Change Robustness
The viewpoint studies use synthetic shifts and real two-sided traversals to measure ConvNet robustness. Higher layers, especially fc6, handle viewpoint changes better, while all layers tolerate mild changes with approximately 90% scene overlap.
- Viewpoint Change Robustness: Viewpoint robustness is evaluated using shifted image crops from Nordland and day-left versus day-right traversals from Gardens Point.
- Viewpoint Change Robustness: fc6 and fc7 are more robust to viewpoint changes than lower layers in both experiments.
- Viewpoint Change Robustness: All layers remain robust to mild viewpoint changes when scenes retain more than 90% overlap.
- Viewpoint Change Robustness: The increased viewpoint robustness of upper layers is attributed to pooling operations after the first, second, and fifth convolutional layers.
C. Summary and Discussion
Conv3 is the strongest general-purpose layer for single-image place recognition under severe appearance changes, whereas higher layers are advantageous mainly for viewpoint changes with limited appearance variation. Feature extraction is fast, but high-dimensional nearest-neighbor search is the computational bottleneck.
- Single-image matching with conv3 performs well under severe appearance changes, while higher layers, especially fc6, help mainly with viewpoint changes and mild appearance variation.
- Higher layers encode more semantic scene information but can lose discrimination between individual places sharing the same semantic scene type.
- Conv3 balances robustness and place discrimination better than low-level features or highly semantic fully connected features.
- 15 ms are required to extract a ConvNet feature, whereas matching among 10,000 places takes 3.5 seconds using 64,896-dimensional cosine-distance search.
V. REAL-TIME LARGE-SCALE PLACE RECOGNITION
The paper targets nearest-neighbor search as the main scalability bottleneck in ConvNet-based place recognition. Mid-level conv3 features and binary hashing substantially reduce computation while retaining most recognition performance.
- Nearest-neighbor runtime scales with the number of stored places, creating a key bottleneck under robotics’ soft real-time constraints.The paper therefore seeks roughly two orders of magnitude lower search time with minimal accuracy degradation.
- Mid-level conv3 features outperform other layers under significant appearance changes.
- 8192-bit hashes retain approximately 95% of place recognition performance while compressing the original feature data by 99.6%.The hashes approximate cosine distance using Hamming distance over binary vectors.
- 13.4 ms is sufficient to find the best match among 10,000 candidates, versus 3570 ms using original cosine-distance features.This yields a 266-fold speed-up for the nearest-neighbor search.
B. Search Space Partitioning using Semantic Categorization
Semantic search-space partitioning uses high-level ConvNet features to restrict matching to similar place categories. This reduces runtime through a tunable trade-off with recognition performance.
- Search-space partitioning: High-level ConvNet features are used with a nonlinear SVM to partition searches among semantic categories such as offices, corridors, classrooms, and restaurants.The approach constrains nearest-neighbor matching to areas with similar semantic place categories.
- Search-space partitioning: 76% less nearest-neighbor search time is achieved by partitioning the search space, with a slight performance decrease.
- Search-space partitioning: Lowering the threshold to θ = 10% assesses more candidate matches, increasing recognition performance at the expense of runtime.The threshold exposes an adjustable accuracy–runtime trade-off.
VI. COMPARING DIFFERENT CONVNETS FOR PLACE RECOGNITION
The study compares AlexNet with Places205 and Hybrid networks for place recognition. Networks trained for semantic place categorization perform slightly better under severe appearance changes, while viewpoint-change results favor AlexNet slightly.
- Appearance changes: Places205 and Hybrid perform slightly better than AlexNet on the Nordland Spring-versus-Winter dataset using hashed conv3 features.
- Appearance changes: Networks perform slightly better under severe appearance changes, plausibly because scene-categorization training teaches place discrimination.
- Viewpoint changes: AlexNet has a slight performance advantage for viewpoint changes on Gardens Point left versus right, while the results are inconclusive.
- Overall comparison: Across the comparison, networks trained for semantic place categorization show slight performance advantages.