PaperlayerView exact revision on arXiv

Source-linked AI summary

NetVLAD: CNN architecture for weakly supervised place recognition

Relja Arandjelović, Petr Gronat, Akihiko Torii, Tomas Pajdla, Josef Sivic

arXiv:1511.07247v3cs.CVcs.LG

TL;DR

Large-scale place recognition needs compact representations that distinguish visually similar places despite appearance changes. NetVLAD introduces an end-to-end CNN with trainable VLAD pooling and weakly supervised ranking, significantly outperforming off-the-shelf CNNs and improving state-of-the-art results across place-recognition and retrieval benchmarks.

  • Problem

    Place recognition requires compact representations that distinguish similarly looking places despite illumination and appearance changes, while off-the-shelf CNN descriptors offer limited instance-level recognition improvements.

  • Method

    The paper trains a CNN end-to-end for place recognition using a differentiable generalized VLAD pooling layer, NetVLAD, and weakly supervised ranking from Street View Time Machine imagery.

  • Results

    81.0% recall@1 versus 55.0% for off-the-shelf AlexNet with standard VLAD on Pitts250k-test, while trained representations outperform competing methods across place-recognition and retrieval benchmarks.

  • Takeaways & Limitations

    NetVLAD pooling and weakly supervised ranking are generic CNN building blocks applicable beyond place recognition, according to the paper’s conclusion.

  • Takeaways & Limitations

    Street View Time Machine imagery provides incomplete and noisy supervision because GPS tags give approximate locations without correspondences between depicted scene parts.

Abstract

from arXiv · show

We tackle the problem of large scale visual place recognition, where the task is to quickly and accurately recognize the location of a given query photograph. We present the following three principal contributions. First, we develop a convolutional neural network (CNN) architecture that is trainable in an end-to-end manner directly for the place recognition task. The main component of this architecture, NetVLAD, is a new generalized VLAD layer, inspired by the "Vector of Locally Aggregated Descriptors" image representation commonly used in image retrieval. The layer is readily pluggable into any CNN architecture and amenable to training via backpropagation. Second, we develop a training procedure, based on a new weakly supervised ranking loss, to learn parameters of the architecture in an end-to-end manner from images depicting the same places over time downloaded from Google Street View Time Machine. Finally, we show that the proposed architecture significantly outperforms non-learnt image representations and off-the-shelf CNN descriptors on two challenging place recognition benchmarks, and improves over current state-of-the-art compact image representations on standard image retrieval benchmarks.

INRIA ∗

The section lists Tomas Pajdla as affiliated with CTU in Prague.

  • Tomas Pajdla is affiliated with CTU in Prague.

1. Introduction

Visual place recognition remains challenging because places must be identified across viewpoint, illumination, and temporal changes while using compact representations. The paper addresses this gap with an end-to-end CNN, a trainable NetVLAD layer, weakly supervised learning from Street View imagery, and strong benchmark results.

  • Motivation and challenge: Visual place recognition supports applications including autonomous driving, augmented reality, and geo-localizing archival imagery, but remains challenging across locations, viewpoints, illumination, and time.The central representation must distinguish similarly looking places while remaining compact enough to represent entire cities or countries.
  • Prior approaches: The task has traditionally been treated as instance retrieval using local invariant features aggregated into whole-image representations such as bag-of-visual-words, VLAD, and Fisher vectors.The query location is estimated from the locations of the most visually similar images in a large geotagged database.
  • Research gap: Off-the-shelf CNN descriptors have yielded limited improvements for instance-level recognition, motivating CNN representations developed and trained directly for place recognition.The paper frames architecture design, data collection, and learning from weak supervision as three main challenges.
  • Contributions: The proposed architecture aggregates conv5 features into a compact vector and introduces NetVLAD, a trainable generalized VLAD layer that is pluggable into CNNs and trainable by backpropagation.NetVLAD is inspired by the Vector of Locally Aggregated Descriptors representation used in image retrieval.
  • Contributions: The model is trained end-to-end with weakly labelled Google Street View Time Machine panoramas depicting the same places across viewpoints and time.The data provide approximately similar positions from noisy GPS, but do not identify corresponding scene parts within panoramas.
  • Contributions: The learned representation is robust to viewpoint and lighting changes, focuses on building façades and skylines, and ignores confusing elements such as cars and people.The learning procedure is tailored to place recognition and uses weakly labelled Time Machine imagery.
  • Results: The proposed architecture significantly outperforms non-learnt representations and off-the-shelf CNN descriptors on two challenging place recognition benchmarks, and improves over state-of-the-art compact representations on image retrieval benchmarks.Earlier CNN retrieval work generally used networks as black-box descriptor extractors rather than training them for the retrieval task.

2. Method overview

The method casts place recognition as image retrieval over a large geotagged database, comparing learned query and database representations by Euclidean distance. Its representation is trained end-to-end with a deep CNN and weakly supervised Google Street View Time Machine data.

  • Image retrieval formulation: Place recognition is formulated as image retrieval: top-ranked geotagged database images provide location suggestions for an unknown query.Database representations can be extracted offline, while the query representation is extracted online.
  • Image retrieval formulation: Nearest-neighbor search ranks database images by Euclidean distance between the query representation f(q) and each database representation f(Ii).Search can be exact or use fast approximate nearest-neighbor methods.
  • Learned representation: Unlike hand-engineered SIFT, bag-of-words, and VLAD representations, the method learns fθ(I) end-to-end directly for place recognition.The representation is parameterized by network parameters θ.
  • Learned representation: The proposed representation uses a deep convolutional neural network architecture inspired by compact aggregated image descriptors for instance retrieval.The network architecture is described in section 3.
  • Weakly supervised training: Network parameters θ are learned end-to-end using weakly supervised training data from Google Street View Time Machine.The learning method is described in section 4.

3. Deep architecture for place recognition

The architecture turns the standard retrieval pipeline of dense local descriptors followed by orderless pooling into a differentiable CNN, centered on the trainable NetVLAD layer. NetVLAD replaces VLAD’s hard cluster assignment with soft assignment, enabling end-to-end learning while retaining residual aggregation.

  • Architecture: The CNN treats the last convolutional feature map as a dense local-descriptor extractor and follows it with orderless pooling for end-to-end retrieval learning.The feature map has dimensions H × W × D.
  • NetVLAD: NetVLAD is a trainable generalized VLAD layer that mimics VLAD within a CNN and produces an image representation optimized directly for place recognition.It is designed to be differentiable with respect to both its parameters and input, enabling backpropagation.
  • Differentiable pooling: Soft assignment replaces VLAD’s discontinuous hard assignment, weighting each descriptor across multiple clusters according to relative proximity.The assignment weights range from 0 to 1, and α → +∞ recovers the original hard-assignment VLAD.
  • Implementation: The layer can be decomposed into standard CNN operations—1×1 convolutions, softmax, L2-normalization—and an aggregation layer, yielding a (K × D) × 1 descriptor.The convolution and softmax compute soft assignments before aggregation and normalization.
  • NetVLAD: NetVLAD aggregates first-order residual statistics using three trainable parameter sets, {wk}, {bk}, and {ck}, rather than VLAD’s single set {ck}.This gives the generalized layer greater flexibility than the original VLAD representation.

4. Learning from Time Machine data

The paper learns its place-recognition representation end to end from weakly labeled Google Street View Time Machine imagery. Because GPS identifies nearby and faraway panoramas without scene-part correspondences, training uses potential positives, definite negatives, and a weakly supervised ranking loss.

  • Weak supervision from the Time Machine: Google Street View Time Machine provides multiple panoramic images of nearby locations captured at different times and seasons.The authors use this imagery as a source of weak supervision for learning an image representation.
  • Weak supervision from the Time Machine: GPS tags provide only approximate locations, yielding potential positives that may match the query and definite negatives that are geographically far away.The supervision is incomplete and noisy because GPS does not establish correspondences between scene parts.
  • Weakly supervised triplet ranking loss: The training objective ranks a close-by database image above faraway images by requiring dθ(q, Ii∗) < dθ(q, Ii).This directly targets place-recognition performance by learning a representation whose Euclidean distances reflect geographic proximity.
  • Weakly supervised triplet ranking loss: For each query tuple, the method selects the best matching image from the ambiguous set of potential positives and compares it against all definite negatives.The potential-positive set contains at least one matching image, but the correct positive is unknown in advance.
  • Weakly supervised triplet ranking loss: The weakly supervised ranking loss sums hinge losses over negatives, assigning zero loss when the negative is farther than the best positive by margin m and penalizing margin violations.The loss adapts triplet-loss intuition to the weakly supervised setting.

5. Experiments

Experiments evaluate NetVLAD on Pittsburgh and Tokyo 24/7 place-recognition benchmarks, comparing trained representations with off-the-shelf CNNs and compact local-feature descriptors. End-to-end training outperforms these alternatives, while the Pittsburgh-trained 256-D representation also achieves state-of-the-art compact image retrieval results.

  • Datasets and evaluation: The experiments use geographically disjoint Pittsburgh splits and the challenging Tokyo 24/7 benchmark, with Pitts30k providing smaller training, validation, and test subsets.Pittsburgh contains 250k database images and 24k queries; Tokyo 24/7 contains 76k database images and 315 mobile-phone queries.
  • Baselines and state-of-the-art: Comparisons include off-the-shelf AlexNet, VGG-16, and Places205 descriptors using either conv5 Max pooling or VLAD aggregation, plus RootSIFT+VLAD compact descriptors.The off-the-shelf networks are pretrained for ImageNet or scene classification without task-specific training.
  • Quantitative results: Representations trained end-to-end for place recognition consistently outperform off-the-shelf CNNs by a large margin on both Pittsburgh and Tokyo 24/7.The authors attribute this to training features to be comparable using Euclidean distance, unlike off-the-shelf conv5 activations.
  • Quantitative results: The trained VGG-16 fVLAD representation with whitening convincingly outperforms RootSIFT+VLAD+whitening and Torii et al., setting the state-of-the-art for compact descriptors on all benchmarks.Reducing VLAD to 4096-D with PCA, whitening, and L2-normalization performs similarly to the full-size vector.
  • Image retrieval benchmarks: The Pittsburgh-trained VGG-16 fVLAD representation with whitening achieves mAP of 63.5%, 73.5%, and 79.9% on Oxford 5k, Paris 6k, and Holidays, respectively.The 256-D representation yields a +20% relative improvement on Oxford 5k.

6. Conclusions

The paper presents an end-to-end CNN architecture for place recognition trained from weakly supervised Street View Time Machine data. Its learned representation outperforms off-the-shelf CNN models and improves over the state of the art on the 24/7 Tokyo, Oxford, and Paris benchmarks.

  • Conclusions: The proposed CNN architecture is trained end to end for place recognition using weakly supervised Street View Time Machine data.Its architecture includes the NetVLAD pooling layer.
  • Conclusions: The trained representation significantly outperforms off-the-shelf CNN models on the challenging 24/7 Tokyo dataset.The conclusion describes this as a significant performance advantage.
  • Conclusions: The trained representation significantly improves over the state of the art on the Oxford and Paris image retrieval benchmarks.The conclusion also reports state-of-the-art improvement on the challenging 24/7 Tokyo dataset.

Appendices

The appendices cover implementation details, Google Street View Time Machine datasets, and additional results.

  • Appendices: The appendices describe implementation details, dataset details, and additional results.Appendix A covers implementation, appendix B covers Google Street View Time Machine datasets, and appendix C provides additional results.

A. Implementation details

The implementation extends AlexNet and VGG-16 with Max pooling or NetVLAD, using descriptor normalization for VLAD variants and dataset-specific NetVLAD initialization. Training uses weakly supervised tuples with hard-negative mining, scheduled optimization, cached representations, and validation-based model selection.

  • Network architecture: AlexNet and VGG-16 are cropped at conv5 and extended with Max pooling or NetVLAD; VLAD and NetVLAD receive descriptor-wise L2 normalization, unlike Max.Max uses raw conv5 descriptors without normalization.
  • NetVLAD initialization: K = 64 clusters are used in all VLAD and NetVLAD experiments, with NetVLAD initialized to reproduce conventional VLAD vectors from clustered conv5 descriptors.The initialization uses a large α to make soft assignment weights sparse and mimic conventional VLAD.
  • Optimization: Training uses margin m = 0.1, learning rate 0.001 or 0.0001, halving every 5 epochs, momentum 0.9, weight decay 0.001, and batches of 4 tuples.Training lasts at most 30 epochs, and the model with the best validation recall@5 is tested.
  • Network architecture: Because VGG-16 is deeper and more GPU-memory hungry than AlexNet, it is trained only down to the conv5 layer.This restriction follows experiments reported in table 1 of the main paper.
  • Tuple construction: Each query tuple includes all positives within 10 meters and randomized hard negatives beyond 25 meters, retaining the 10 hardest from 1000 random negatives and the previous epoch.Remembering previous hard negatives adds stability to training.
  • Efficient mining: Cached representations of the full training query and database sets make hard-negative mining tractable, limiting forward and backward passes to the selected 10 examples.The cached representations are slightly stale but avoid processing more than 10^10 full-resolution images per tuple.

B. Google Street View Time Machine datasets

The study introduces Tokyo Time Machine (TokyoTM), built from downloaded Time Machine panoramas represented as 12 orientation-sampled perspective images. Query positives and negatives are separated temporally by at least one month, and geographic splits are mutually disjoint across training, validation, and testing.

  • Dataset construction: TokyoTM is generated from downloaded Time Machine panoramas, with each panorama represented by 12 perspective images sampled evenly across orientations.Figure 7 provides example images from the dataset.
  • Dataset splits: Training, validation, and testing datasets are mutually disjoint geographically.This condition is stated for all datasets used in the experiments.
  • Temporal sampling: For every query, positive and negative database samples have timestamps at least one month away from the query timestamp during training, validation, and testing.The same temporal separation rule is applied to both sampling and evaluation.
  • Availability: Pitts250k and Tokyo 24/7 are publicly available from their authors, while TokyoTM and Pitts30k are shared on request.The datasets are provided by the authors or made available upon request.

C. Additional results and discussions

Additional experiments show that NetVLAD is more compact than Max pooling at comparable performance, while trained pooling improves place recognition and compact retrieval results. Training with Time Machine imagery is important for generalization, and NetVLAD achieves strong performance on standard retrieval benchmarks.

  • VLAD versus Max: 128-D NetVLAD matches 512-D Max on Tokyo 24/7, yielding a four-fold more compact representation at similar performance.On Pitts250k-test, NetVLAD also achieves a two-fold memory saving compared with Max.
  • Max versus Sum: 67.9% versus 59.3% recall@5 shows that Sum outperforms Max off the shelf on Pitts250k-test for VGG-16, but not AlexNet.After training, Max reaches 88.7% versus Sum at 82.3%, although Sum gains 21% relative to its off-the-shelf performance.
  • Further results: Training with Google Street View Time Machine imagery is essential for generalization on Pitts30k-val.Table 3 compares recall of fmax with and without Time Machine data for AlexNet.
  • Image retrieval experiments: +20% relative improvement on Oxford 5k (crop) is achieved by the 256-D NetVLAD representation over compact image-representation baselines.The representation achieves the best mAP on Oxford and Paris by a large margin and sets the state of the art on Holidays, although training is detrimental there.
  • Benefits of end-to-end training for place recognition: Trained representations outperform off-the-shelf networks by a large margin, while VGG-16 NetVLAD with whitening sets the state of the art across the reported datasets.The comparison also reports that fVLAD outperforms fmax.
Loading 1511.07247v3…