Source-linked AI summary
Fine-tuning CNN Image Retrieval with No Human Annotation
Filip Radenović, Giorgos Tolias, Ondřej Chum
TL;DR
CNN retrieval fine-tuning typically depends on costly, potentially error-prone annotation, especially because particular-object matching requires pair-level supervision. The paper uses automatically reconstructed 3D models to mine hard matching and non-matching pairs, alongside discriminative whitening and trainable GeM pooling. The resulting VGG-based method achieves top performance on standard retrieval benchmarks without manual annotation.
Problem
Particular-object retrieval requires pair-level matching labels, while fine-tuning CNNs traditionally depends on costly manual annotation.
Method
The method fine-tunes a siamese CNN using automatically selected matching and non-matching pairs from 3D reconstructions, with discriminative whitening and trainable GeM pooling.
Results
The method achieves top performance on standard benchmarks without manual annotation and improves retrieval through its pooling layer.
Takeaways & Limitations
Automatically reconstructed 3D models can supply training data for effective CNN fine-tuning on particular-object retrieval.
Takeaways & Limitations
Extremely hard positive pairs, such as minimal overlap or extreme scale changes, can prevent generalization and lead to over-fitting.
Abstract
from arXiv · showhide
Image descriptors based on activations of Convolutional Neural Networks (CNNs) have become dominant in image retrieval due to their discriminative power, compactness of representation, and search efficiency. Training of CNNs, either from scratch or fine-tuning, requires a large amount of annotated data, where a high quality of annotation is often crucial. In this work, we propose to fine-tune CNNs for image retrieval on a large collection of unordered images in a fully automated manner. Reconstructed 3D models obtained by the state-of-the-art retrieval and structure-from-motion methods guide the selection of the training data. We show that both hard-positive and hard-negative examples, selected by exploiting the geometry and the camera positions available from the 3D models, enhance the performance of particular-object retrieval. CNN descriptor whitening discriminatively learned from the same training data outperforms commonly used PCA whitening. We propose a novel trainable Generalized-Mean (GeM) pooling layer that generalizes max and average pooling and show that it boosts retrieval performance. Applying the proposed method to the VGG network achieves state-of-the-art performance on the standard benchmarks: Oxford Buildings, Paris, and Holidays datasets.
1 INTRODUCTION
The paper fine-tunes CNNs for instance image retrieval without manual annotation by using automatically reconstructed 3D models to select training pairs. It also introduces trainable pooling and discriminative whitening to improve compact image representations.
- Instance retrieval seeks a particular query object in a large, unordered image collection, where CNN descriptors offer compact and accurate representations.
- Fine-tuning improves adaptation to retrieval but traditionally requires additional annotated training data.
- Automatically reconstructed 3D models provide geometry, camera positions, and matching graphs for selecting matching and non-matching training pairs without manual annotation.
- A trainable generalized-mean pooling layer includes max and average pooling as special cases and significantly boosts performance over standard non-trainable pooling.
- The paper uses SfM information to enforce hard-positive and hard-negative examples, learns whitening from the same training data, and proposes trainable pooling for retrieval.
- The manuscript evaluates its proposed methods quantitatively and qualitatively across different CNN architectures against state-of-the-art systems.
2 RELATED WORK
Prior work uses CNN activations, manually collected or weakly supervised data, and diverse pooling and whitening strategies for image retrieval. This paper replaces manual pair discovery with 3D-geometry-based mining and learns whitening discriminatively.
- CNN representations provide compact image descriptors and have outperformed mature local-feature systems using codebooks, spatial verification, and query expansion.
- Instance retrieval is formulated as metric learning, but particular-object supervision requires image-pair labels because category labels do not specify matching viewpoints or objects.
- Earlier retrieval systems used classification activations, manually collected landmark data, and several global-pooling schemes including max, sum, regional, and mixed pooling.
- The proposed method discovers matching and non-matching pairs unsupervised from 3D reconstructions, enabling harder examples than geo-tagged weak supervision.
- Hard-positive examples require careful sampling because extremely difficult pairs can prevent generalization and cause over-fitting.
- The paper learns whitening discriminatively from reconstructed training data, while end-to-end whitening was no better than post-processing and converged more slowly.
3 ARCHITECTURE, LEARNING, SEARCH
The paper presents a fully convolutional CNN retrieval architecture using trainable GeM pooling, contrastive siamese learning, discriminative whitening, and α-weighted query expansion. Its representation and post-processing are designed for compact image retrieval, with fine-tuning and visualization analyses showing how pooling and learned transformations affect similarity.
- Architecture: The architecture discards fully connected layers from fully convolutional CNNs and uses them as initialization for retrieval fine-tuning.The methodology applies to CNNs such as AlexNet, VGG, and ResNet.
- Generalized-mean pooling: GeM pooling produces one generalized-mean value per feature map and includes max pooling and average pooling as special cases.The pooling parameter can be learned through back-propagation, either separately per feature map or as one shared value.
- Generalized-mean pooling: 2.92 is the converged shared pooling parameter in the fine-tuned VGG GeM example.Larger p values produce more localized feature-map responses.
- Siamese learning and loss function: Contrastive loss trains a two-branch siamese network on matching and non-matching image pairs represented by ℓ2-normalized GeM vectors.The margin parameter determines when sufficiently distant non-matching pairs are ignored, and the authors report better generalization and convergence than triplet loss.
- Whitening and dimensionality reduction: The same training data supports discriminative whitening based on intraclass whitening and interclass PCA rotation, followed by dimensionality reduction and ℓ2 normalization.The procedure is applied after GeM optimization rather than end-to-end; the reported comparison favors this post-processing approach over end-to-end whitening.
- Image representation and search: α-weighted query expansion weights retrieved descriptors by their similarity to the query, with AQE recovered at α = 0.The method is proposed to avoid the difficulty of tuning a fixed number of equally weighted retrieved images across datasets.
4 TRAINING DATASET
The training dataset is selected automatically from BoW retrieval and SfM reconstructions, which provide camera and 3D visibility information for constructing challenging matching and non-matching pairs. Positive selection increases viewpoint variability while negative selection increases instance variability.
- 4.1 BoW and 3D reconstruction: BoW retrieval and SfM reconstruction provide the 3D models, camera positions, and visibility information used to select training data.The system clusters an unannotated image collection, reconstructs a model per cluster, and represents each model with a bipartite image–point visibility graph.
- 4.2 Selection of training image pairs: Training tuples contain a query image, one matching image, and non-matching images, producing |N(q)| + 1 training pairs per query.Candidate positives are initially drawn from images with camera centers closest to the query, although these images may not depict the same object.
- 4.2 Selection of training image pairs: The maximum-inliers strategy selects the image sharing the most co-observed 3D points with the query, independently of the CNN descriptor.Because the measure counts spatially verified features, it provides more challenging positive examples than descriptor-based selection.
- 4.2 Selection of training image pairs: The relaxed-inliers strategy randomly selects a same-object image with sufficient shared 3D points and bounded scale change, increasing viewpoint variability.Method m3 differs from m1 for 86.5% of queries while retaining the guarantee that the images depict the same object.
- 4.2 Selection of training image pairs: Negative strategy N2 selects k-nearest non-matching images with at most one image per 3D model, increasing negative-example variability over N1.N1 selects nearest neighbors across all non-matching images and can return multiple similar instances of the same object.
5 EXPERIMENTS
The experiments evaluate automated CNN fine-tuning across training-data selection, descriptor processing, pooling, multi-scale representation, query expansion, and benchmark performance. Results show benefits from learned GeM pooling and whitening, while broader training-data variability and unsupervised training support strong retrieval performance.
- Training setup: Training uses automatically reconstructed 3D models, with 551 models selected for training and 162 for validation.The source collection contains 713 non-overlapping reconstructed models with more than 163k unique images.
- Training setup: Each training tuple contains one query, one positive, and five negatives, with positives selected from the 100 closest camera centers.Hard negatives are re-mined three times per epoch.
- Descriptor whitening: The learned whitening transform Lw achieves the best performance in most cases and always outperforms no post-processing for VGG.PCA whitening often reduces performance, while end-to-end whitening converges more slowly and does not surpass Lw.
- Multi-scale representation: Multi-scale GeM provides a significant benefit over descriptor averaging and is adopted for subsequent experiments.The multi-scale representation is constructed at test time without additional learning.
- Query expansion: The proposed αQE is more stable than AQE across Oxford and Paris because the datasets differ in the number of relevant images per query.Oxford averages 52 positive images per query, compared with 163 for Paris.
- Comparison with the state of the art: The proposed methods outperform the state of the art on all datasets with VGG, achieve the state-of-the-art score on Oxford with ResNet, and are on par on Holidays.On Paris with ResNet, the method is outperformed by Gordo et al.; no manual labeling or cleaning is used.
6 CONCLUSIONS
The paper fine-tunes CNNs for image retrieval using training data selected automatically from 3D reconstructions of unordered photo collections. Without manual annotation, the method achieves top benchmark performance, while GeM pooling improves retrieval accuracy and multi-scale representation.
- 6 CONCLUSIONS: Training data are selected automatically from 3D reconstructions of a large unordered photo collection without manual annotation.The process is applicable to rigid 3D objects, although the reconstructions used here consist of buildings and popular landmarks.
- 6 CONCLUSIONS: The method achieves top performance on standard image-retrieval benchmarks without manually annotated or cleaned training data.The results reach the level of the best systems based on local features with spatial matching and query expansion.
- 6 CONCLUSIONS: The proposed pooling layer improves retrieval accuracy and supports construction of a joint multi-scale representation.Training data, trained models, and code are publicly available.