Source-linked AI summary
CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples
Filip Radenović, Giorgos Tolias, Ondřej Chum
TL;DR
The paper addresses the need to fine-tune CNNs for image retrieval without manual annotation. It selects training pairs from automated 3D reconstructions and trains with hard positive and negative examples. The resulting compact descriptors outperform prior compact-representation methods across standard benchmarks and approach local-feature systems while using less memory and time.
Problem
Manual annotation is costly for CNN training and fine-tuning, motivating an automated approach tailored to image retrieval.
Method
The method uses automated 3D reconstructions to select matching and non-matching training pairs for siamese CNN fine-tuning, followed by learned whitening.
Results
The method outperforms the state of the art on standard benchmarks across 16D to 512D descriptors and reaches the level of leading local-feature systems.
Takeaways & Limitations
Hard positive and negative examples enhance compact retrieval representations, with 32D vectors matching existing 256D representations in the reported comparisons.
Takeaways & Limitations
Some reported comparisons are not directly comparable because full images were used as queries on Oxford and Paris.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) achieve state-of-the-art performance in many computer vision tasks. However, this achievement is preceded by extreme manual annotation in order to perform either training from scratch or fine-tuning for the target task. In this work, we propose to fine-tune CNN for image retrieval from a large collection of unordered images in a fully automated manner. We employ state-of-the-art retrieval and Structure-from-Motion (SfM) methods to obtain 3D models, which are used to guide the selection of the training data for CNN fine-tuning. We show that both hard positive and hard negative examples enhance the final performance in particular object retrieval with compact codes.
1 Introduction
Image retrieval has advanced through local-feature systems and compact representations, while CNN adaptation remains constrained by costly annotation and classification-oriented training. This paper proposes unsupervised fine-tuning from 3D reconstructions, using hard matching and non-matching examples to learn compact retrieval representations.
- Local-feature retrieval systems combine large visual codebooks, spatial verification, and query expansion for particular object retrieval.
- Compact representations such as Fisher vectors and VLAD reduce memory requirements and improve search efficiency.
- CNN activations provide effective off-the-shelf image descriptors, but their success depends on large manually annotated datasets.
- Classification-based fine-tuning improves retrieval accuracy but is less aligned with instance retrieval than the desired matching behavior.
- Structure-from-Motion pipelines use reliable image clusters and matching graphs to build 3D models from unordered collections.
- The proposed method uses 3D reconstructions to select training data and teaches CNNs to mimic local-feature retrieval with spatial verification.
- Hard positive and hard negative examples enhance the learned representation, while learned whitening complements fine-tuning and 32D vectors reach the level of existing 256D representations.
2 Related work
Prior CNN retrieval methods rely on classification or weak supervision, whereas this work discovers matching and non-matching pairs through 3D reconstruction without manual annotation. The reconstruction also supplies harder positive examples than those selected by the current CNN representation.
- CNN retrieval accuracy demonstrates some generalization from ImageNet classification, while landmark-specific retraining improves standard retrieval benchmarks.
- Constructing task-specific training datasets for CNN fine-tuning requires manual effort.
- Geo-tagged weak supervision selects non-matching pairs by distance and matching pairs by nearby-image similarity under the current CNN representation.
- This work discovers matching and non-matching pairs unsupervisedly from 3D reconstruction, enabling harder matching examples than those identified by the current network.
3 Network architecture and image representation
The architecture converts convolutional activations into normalized MAC descriptors and fine-tunes them with a shared-weight siamese network using automatically generated pairs. A learned projection further whitens and reduces the descriptors using matching and non-matching covariance structure.
- 3.1 Image representation: MAC represents an image by max-pooling each feature map of the final convolutional tensor, producing one value per feature map.
- 3.1 Image representation: MAC vectors are non-negative, ℓ2-normalized, and compared with an inner product; their dimensionality equals the number of feature maps.
- 3.2 Network and siamese learning: Fully connected layers are discarded, and a MAC layer followed by ℓ2-normalization produces the descriptor used throughout the method.
- 3.2 Network and siamese learning: A siamese network shares parameters across two branches that process image pairs labeled as matching or non-matching.
- 3.2 Network and siamese learning: Contrastive loss pulls matching normalized MAC vectors together and penalizes non-matching pairs only when their distance is below threshold τ.
- 3.2 Network and siamese learning: The network is trained with stochastic gradient descent on a large automatically created training set.
- 3.2 Network and siamese learning: Figure 1 highlights patches associated with MAC components contributing most to pairwise similarity before and after VGG fine-tuning.
- 3.3 Whitening and dimensionality reduction: The learned projection whitens using matching-pair covariance, rotates using interclass covariance in whitened space, and retains the D largest-eigenvalue directions.
4 Training dataset
The training dataset is built automatically from unordered images by combining BoW retrieval with SfM-based 3D models. These models guide selection of challenging positive and negative pairs for CNN fine-tuning.
- Automatic dataset construction: BoW retrieval clusters landmark images into matching graphs that seed SfM reconstruction of 3D models.SfM filters mismatches and estimates camera positions and registered local features for clustered images.
- Automatic dataset construction: Each 3D model is represented as a bipartite visibility graph linking images to the 3D points they observe.For an image i, P(i) contains the model points visible in that image.
- Pair construction: Training tuples contain a query q, one matching image m(q), and a set of non-matching images N(q), yielding |N(q)| + 1 training pairs.Positive candidates are drawn from images with nearby camera centers in the query's cluster, although viewpoint variation can make them depict different objects.
- Positive selection: The MAC-distance strategy selects the candidate positive closest to the query, but its low descriptor distance makes the example less challenging.This choice can therefore provide limited pressure for the network to learn from positive samples.
- Positive selection: Maximum-inlier selection chooses the image sharing the most co-observed 3D points, producing harder positives independently of the CNN descriptor.The measure corresponds to spatially verified features used in BoW-based retrieval.
- Positive selection: Relaxed-inlier selection samples an image that co-observes enough points while limiting scale change, guaranteeing the same object with greater viewpoint variability.Method m3 differs from m1 for 86.5% of queries and increases viewpoint variability.
- Negative selection: Hard negatives come from clusters different from the query's, using nearest-neighbor selection either globally or with at most one image per cluster.N1 may select multiple similar instances of one object, whereas N2 provides greater variety.
5 Experiments
Experiments evaluate unsupervised CNN fine-tuning across retrieval datasets, training choices, learned whitening, descriptor dimensionality, generalization, and state-of-the-art comparisons. Hard positive and negative selection, diverse 3D-model training data, and learned whitening consistently support compact retrieval representations.
- Test datasets and evaluation protocol: The evaluation uses Oxford buildings, Paris, and Holidays, with Oxford100k distractors for larger-scale testing and mAP as the metric.Queries are evaluated using either cropped images (CropI) or cropped activations (CropX).
- Learning: Positive examples with larger viewpoint variability and negative examples with higher content variability acquire better retrieval representations during fine-tuning.Methods are compared across training epochs against the off-the-shelf network using AlexNet MAC on Oxford105k and Paris106k.
- Dataset variability: Training with all 551 available 3D models performs best because greater dataset variability improves the training set, although few clusters can cause over-fitting.Significant gains are still obtained with 10 or 100 models when the number of training queries is fixed.
- Learned projections: Learned whitening achieves the best performance in most cases and is never worst, whereas PCA-whitening often reduces performance substantially.For VGG, learned whitening always outperforms the no-post-processing baseline; for AlexNet, its occasional drops are negligible relative to PCA-whitening drops.
- Dimensionality reduction: Learned whitening works better at higher descriptor dimensionalities, while PCA-whitening works slightly better at lower dimensionalities.A 16D MAC descriptor reaches 45.5 mAP on Oxford105k, exceeding the 41.4 mAP of state-of-the-art BoW-based 128D compact codes.
- Over-fitting and generalization: The network generalizes well and is relatively insensitive to over-fitting when training includes reconstructions of Oxford and Paris landmarks.This comparison uses the same number of training queries while changing whether those landmark reconstructions are included.
- Comparison with the state of the art: Fine-tuned MAC and R-MAC outperform prior compact-representation methods on Paris and Oxford across descriptor sizes from 16D through 512D, with and without distractors.On Holidays, Neural codes lead for extreme short codes, while off-the-shelf NetVLAD performs best at 256D and higher.
- Comparison with the state of the art: Localization-based re-ranking further boosts MAC and R-MAC, bringing their scores into competition with local-feature and query-expansion systems that require more memory and query time.The comparison concerns the fine-tuned CNN representations combined with a recent localization method.
6 Conclusions
The method fine-tunes CNNs for image retrieval using training data selected from automated 3D reconstruction, without manual annotation. It outperforms state-of-the-art methods across descriptor dimensionalities, while remaining faster and less memory-intensive than the best local-feature systems.
- The method selects CNN fine-tuning data from automated 3D reconstruction of a large unordered photo collection, eliminating manual annotation.Training data, fine-tuned networks, and evaluation code are publicly available.
- The proposed approach outperforms the state of the art on standard benchmarks across descriptor dimensionalities from 16 to 512.Table 2 compares AlexNet and VGG variants, with fine-tuned networks marked separately.
- The resulting systems reach the level of the best local-feature systems while being faster and requiring less memory.The conclusion contrasts the CNN-based approach with systems using local features, spatial matching, and query expansion.
- Some Oxford and Paris results are not directly comparable because full images were used as queries.This evaluation caveat is marked with a dagger in the reported results.