Source-linked AI summary
Deep Learning Features at Scale for Visual Place Recognition
Zetao Chen, Adam Jacobson, Niko Sunderhauf, Ben Upcroft, Lingqiao Liu, Chunhua Shen, Ian Reid, Michael Milford
TL;DR
Visual place recognition lacked sufficiently large, place-specific data and evidence about networks trained directly for the task. The paper addresses this by training two CNNs with multi-scale feature encoding on a 2.5 million-image place dataset, then evaluates them across challenging benchmarks. The trained networks show an average 10% performance increase over other place-recognition algorithms and pre-trained CNNs.
Problem
Prior studies mainly used CNN features trained for other recognition tasks, while place-recognition datasets were too small to study task-specific training at scale.
Method
The authors train two CNN architectures for place recognition using a 2.5 million-image dataset, formulate recognition as classification, and apply multi-scale feature encoding.
Results
The trained networks achieve an average 10% increase in performance over other place-recognition algorithms and pre-trained CNNs on challenging benchmark datasets.
Takeaways & Limitations
Place-specific training produces representations reported as more robust to appearance and viewpoint variations than features from object-centric networks.
Takeaways & Limitations
HybridNet consistently outperforms AMOSNet, suggesting that fine-tuning pre-trained structure can be more discriminative than training completely from scratch on this dataset.
Abstract
from arXiv · showhide
The success of deep learning techniques in the computer vision domain has triggered a range of initial investigations into their utility for visual place recognition, all using generic features from networks that were trained for other types of recognition tasks. In this paper, we train, at large scale, two CNN architectures for the specific place recognition task and employ a multi-scale feature encoding method to generate condition- and viewpoint-invariant features. To enable this training to occur, we have developed a massive Specific PlacEs Dataset (SPED) with hundreds of examples of place appearance change at thousands of different places, as opposed to the semantic place type datasets currently available. This new dataset enables us to set up a training regime that interprets place recognition as a classification problem. We comprehensively evaluate our trained networks on several challenging benchmark place recognition datasets and demonstrate that they achieve an average 10% increase in performance over other place recognition algorithms and pre-trained CNNs. By analyzing the network responses and their differences from pre-trained networks, we provide insights into what a network learns when training for place recognition, and what these results signify for future research in this area.
I. INTRODUCTION
The paper identifies gaps in applying deep learning specifically to visual place recognition and addresses them by training place-focused networks at scale. It introduces a large place-centric dataset, evaluates the trained networks against existing approaches, and examines their learned representations.
- Visual place recognition matches a current scene to a previously visited location, commonly using image retrieval and bag-of-words representations built from local features.
- Pre-trained CNN representations have achieved strong place-recognition performance, but they were trained for object recognition rather than the place-recognition task.
- The paper identifies three gaps: unknown outcomes from task-specific training, insufficiently large place-recognition datasets, and limited understanding of task-specific network characteristics.
- The authors develop a 2.5 million-image dataset spanning thousands of places and hundreds of environmental examples per place, enabling place recognition to be treated as classification.
- They evaluate trained networks on four public benchmark datasets against conventional place-recognition methods and networks trained for other recognition tasks.
- The paper analyzes CNN layer responses to compare internal representations learned specifically for place recognition with those learned for other recognition tasks.
B. Existing Place Recognition Datasets
Existing place-recognition datasets are too small and environmentally limited for broadly applicable condition-invariant CNN training. The authors therefore construct a large place-centric database with diverse environments and repeated observations of specific locations.
- Eynsham contains 9575 panoramic images from two 35 km traverses, while St. Lucia covers five times of day in suburban streets with small appearance variation.
- Nordland covers a 728 km journey across four seasons, but lacks sufficient variation in illumination, day-night cycles, and other weather phenomena.
- Existing place-recognition datasets are tiny compared with object datasets and provide only a handful of environmental exemplars for each place.
- Scene recognition differs from recognizing a specific unique place, so a scene-centric database does not directly provide the required place-recognition training data.
- The authors construct a freely available place-centric database from surveillance cameras worldwide, covering varied environments and more than 2.5 million images.
III. TRAINING CNNS FOR PLACE RECOGNITION
The paper trains classification-based CNNs specifically for place recognition and encodes convolutional responses with multi-scale pooling to improve robustness across viewpoints and conditions.
- Training architecture: The authors choose a classification architecture because Siamese and triplet training would create impractically large permutation spaces on millions of SPED images.The classification network successfully trains on millions of images, whereas Siamese or triplet networks would require exponentially large permutations.
- Network design: The ConvNet uses six convolutional layers with max-pooling, two fully connected layers, and a softmax output representing place identity.Images are resized to 256 × 256, then randomly cropped into 227 × 227 RGB patches for training.
- Optimization: Training maps input images to place probabilities and minimizes negative log-likelihood using stochastic gradient descent with back-propagation.The batch size is 50.
- Feature encoding: Multi-scale pooling searches for prominent visual patterns across locations by applying max pooling within cells of a four-scale spatial pyramid.The scales are H = [1 2 3 4], allowing responses to be matched across different viewpoints.
- Implementation: The study compares HybridNet, initialized with CaffeNet weights, against AMOSNet, trained without that initialization.Both networks were trained for 120,000 iterations, taking about two days on an NVIDIA Tesla K40 GPU.
IV. CREATING THE SPECIFIC PLACES DATASET (SPED)
SPED is a large place-centric dataset built from outdoor surveillance cameras to provide diverse scenes and repeated appearances under changing environmental conditions.
- Data source: The dataset draws from AMOS, which contains images from approximately 30,000 outdoor cameras around the world.The cameras recorded images every half hour over a period of ten years, supporting long-term environmental variation studies.
- Dataset construction: The authors selected 2,543 cameras and downloaded images from February and August 2014, producing about 2,500,000 images.These time points were chosen because they exhibit strong seasonal changes.
- Curation: Dataset curation removes pitch-black nighttime images and corrupted images caused by camera malfunction.Sample images illustrate both diversity across scenes and condition variation within scenes.
- Within-place variation: The database captures large environmental changes within each scene, including lighting, day-night, and seasonal variation.For each camera, images were collected every half hour during the selected months.
- Scene diversity: The selected cameras span outdoor settings from forest landscapes and country roads to urban scenes.This scene diversity complements the repeated observations of each camera location.
V. EXPERIMENTAL SETUP
The experiments use representations generated from images to evaluate visual place recognition performance.
- Experimental setup: This section introduces the testing datasets and explains how image representations are generated.The supplied passage provides no further experimental-design details.
A. Testing Datasets
The evaluation uses four geographically separated benchmark datasets, with separate reference and test traverses under changing conditions and viewpoints.
- Testing protocol: The benchmark datasets come from locations and environments distinct from the training data, and each uses one traverse for reference and another for testing.This setup tests generalization across geographically separated locations.
- Appearance robustness: Nordland, St. Lucia, and Eynsham evaluate robustness to appearance changes across different environmental and viewpoint conditions.Nordland spans four seasons, St. Lucia includes time-of-day and medium viewpoint variation, and Eynsham has modest appearance variation.
- Viewpoint robustness: Gardens Point evaluates viewpoint robustness using daytime-left-side and nighttime-right-side traverses of the same route.The dataset therefore combines significant viewpoint and appearance changes.
B. Ground Truth
Ground truth was established using dataset-specific procedures, including GPS-derived tolerances and manual or visual frame matching.
- Eynsham ground truth used the dataset’s 40 metre GPS-derived tolerance, matching the original FAB-MAP study.
- Nordland and Gardens Point ground truth came from manually selected frame correspondences.
- St. Lucia ground truth combined closest-GPS reference-image retrieval with visual matching of the correct image.
- Table I describes the benchmark datasets using the number of frames per traverse.
C. Feature Extraction
The evaluation extracts representations from multiple network layers and matches testing images against reference frames using layer-specific confusion matrices.
- Feature activations are compared from the third convolutional layer through the eighth fully connected layer.
- Each layer output is treated as a deep learned image representation before matching testing images to all reference frames.
- The matching process produces a separate confusion matrix for each evaluated network layer.
- The test datasets were unseen during training and therefore assess generalization to unseen data.
A. Visualization of the Deep Features
The visualization compares learned weights and activations from the place-recognition models with those from CaffeNet/ImageNet-trained features, revealing increasingly different higher-layer responses.
- First-layer weights are similar across networks, capturing oriented edges and opponent colors.
- Higher-layer activations become semantically different between the ImageNet network and the place-recognition models.
- At layer 4, ImageNet features respond more to object blobs, whereas the trained model responds more to scene-type patches.
B. Comparing Different Feature Encoding Methods
The experiments compare multi-scale encoding and layer-specific CNN features across benchmark datasets, showing advantages for the trained place-recognition networks and selected intermediate convolutional layers.
- B. Comparing Different Feature Encoding Methods: Multi-scale pooling consistently outperforms cross-layer and holistic pooling on the Stlucia, Nordland, and Garden datasets.
- C. Benchmark Place Recognition Performance: On Nordland, AMOSNet and HybridNet outperform SeqSLAM, CaffeNet, and PlaceNet across all evaluated layers.
- C. Benchmark Place Recognition Performance: The best Nordland result uses HybridNet’s fifth convolutional layer, where the advantage is most pronounced.
- C. Benchmark Place Recognition Performance: On St. Lucia, AMOSNet and HybridNet outperform CaffeNet and PlaceNet from Conv5 through fc8, indicating robustness to illumination changes.
- C. Benchmark Place Recognition Performance: On Eynsham, both trained networks perform well, with HybridNet’s Conv5 achieving the best performance.
- D. Viewpoint Change Robustness: On Gardens Point, AMOSNet and HybridNet are more robust than CaffeNet to appearance and viewpoint variations across network layers.
- D. Viewpoint Change Robustness: HybridNet is strongest at Conv4–Conv6 on Gardens Point, while PlaceNet leads at fc7 and fc8.
VII. DISCUSSION AND FUTURE WORK
The paper shows that place-centric CNN representations are more robust to appearance and viewpoint variation than object-centric features across challenging datasets. It also identifies dataset coverage and viewpoint diversity as important constraints for further progress.
- Place-centric CNN representations are more robust to appearance and viewpoint variations than features from object-centric networks such as CaffeNet.The comparison was demonstrated on several challenging place recognition datasets.
- The trained networks outperformed state-of-the-art place recognition algorithms in the reported comparisons.
- A. HybridNet vs. AMOSNet: HybridNet consistently outperformed AMOSNet, likely because ImageNet pretraining supplied useful structure that remained discriminative after fine-tuning.The authors found this result counterintuitive because AMOSNet was trained from scratch and was expected to be more place-recognition-specific.
- B. More significant viewpoint invariance: The current network has limited viewpoint variation because its training data were collected from static webcams.
- B. More significant viewpoint invariance: No sufficiently large dataset with significant viewpoint changes was available for training a deep CNN, motivating work on city-scale synthetic data with broad viewpoint and condition variation.
- Shared place-relevant data from self-driving car fleets may enable systems that surpass current state-of-the-art results after suitable curation and preprocessing.