Source-linked AI summary
Visual Instance Retrieval with Deep Convolutional Networks
Ali Sharif Razavian, Josephine Sullivan, Stefan Carlsson, Atsuto Maki
TL;DR
Visual instance retrieval needs compact representations that remain effective across changes in viewpoint, scale, and position. This paper studies generic ConvNet representations and builds a multi-scale local-feature pipeline with spatially consistent similarity computation. Across five standard datasets, suitably extracted ConvNet representations outperform state-of-the-art methods, while the authors position the result as a baseline for further improvements.
Problem
Visual instance retrieval requires compact, descriptive representations and efficient search for large image collections, while earlier ConvNet representations had not surpassed state-of-the-art holistic methods.
Method
The paper uses generic ImageNet-trained ConvNets, last-convolutional-layer features, spatial max-pooling, and multi-scale local-feature extraction with geometric invariance explicitly considered.
Results
Across five standard retrieval datasets, suitably extracted generic ConvNet representations outperform other state-of-the-art methods.
Takeaways & Limitations
ConvNet representations can support visual instance retrieval effectively when layer choice, pooling, local features, geometric invariance, and similarity computation are designed together.
Takeaways & Limitations
The reported result should be viewed as a baseline because fine-tuning and concatenating multi-scale, multilayer, or architectural representations can improve performance.
Abstract
from arXiv · showhide
This paper provides an extensive study on the availability of image representations based on convolutional networks (ConvNets) for the task of visual instance retrieval. Besides the choice of convolutional layers, we present an efficient pipeline exploiting multi-scale schemes to extract local features, in particular, by taking geometric invariance into explicit account, i.e. positions, scales and spatial consistency. In our experiments using five standard image retrieval datasets, we demonstrate that generic ConvNet image representations can outperform other state-of-the-art methods if they are extracted appropriately.
1. Introduction
Visual instance retrieval requires compact, descriptive representations that remain useful across changes in viewpoint, lighting, scale, and location. This paper studies ConvNet representations and proposes a multi-scale retrieval pipeline addressing geometric invariance, layer choice, spatial consistency, and search efficiency.
- Task and challenges: Visual instance retrieval matches unlabeled query images to reference images containing the same object or scene.The task uses labeled reference images and must support collections containing millions or billions of images.
- Approach: The paper prioritizes geometric invariance by studying multi-scale local features and designing the similarity measure around positions, scales, and spatial consistency.The pipeline is intended to handle items appearing at different viewpoints, scales, and locations.
- Task and challenges: Representations must balance compactness with descriptiveness, with sizes ranging from 1k dimensions to less than 16 bytes depending on the task.The paper distinguishes small-footprint and medium-footprint settings according to memory and collection size.
- Motivation: ConvNet retrieval performance had not yet matched state-of-the-art holistic methods, motivating further study of representation and similarity choices.Earlier work examined different ConvNet layers, while Fisher vectors remained a stronger holistic baseline.
- Contributions: Across five standard retrieval datasets, suitably extracted ConvNet representations outperform state-of-the-art methods.The datasets span patterns from texture-less to repeated ones.
- Contributions: The contributions include a ConvNet-based local-feature pipeline and spatial pooling that reduces dimensionality while preserving spatial consistency.These design choices target efficient retrieval without discarding useful spatial information.
3. The ConvNet representation of an image
The paper uses generic ImageNet-trained ConvNets to represent images or sub-patches, emphasizing a representation suitable for a general retrieval pipeline. It also avoids dataset-specific bias and supports parallelized processing.
- Representation design: The pipeline extracts representations of either whole images or sub-patches using generic ConvNets trained on ImageNet.This representation is the basis for the subsequent retrieval pipeline.
- Representation design: The pipeline does not rely on dataset bias and can therefore be highly parallelized.The only specialized data use is estimating the distribution of the data for the representation.
2. Related work
Earlier retrieval work motivates compact, spatially informative representations, while this paper develops ConvNet-based alternatives using convolutional responses, spatial pooling, and multi-resolution local features.
- ConvNet representations attracted attention for image retrieval after deep convolutional networks succeeded in image classification.
- Retrieval methods include holistic image vectors and vector-aggregation approaches such as BOW, Fisher Vector, and VLAD.
- The final convolutional layer is favored because its responses preserve appearance patterns and information useful for instance retrieval.
- Fully connected layers are less suitable for rectangular local features because cropping loses information or breaking aspect ratio harms retrieval.
- Spatial pooling reduces response-map dimensionality while preserving spatial consistency, but excessive grid cells can reduce performance.
- The pipeline extracts local features across multiple patch sizes and computes image distances by pooling distances between query and reference patches.
4. Pipeline for measuring similarities of images
The pipeline measures image similarity using multi-resolution local patches, query-image jittering, and feature preprocessing. It explicitly addresses variation in object position and scale while controlling patch representation and dimensionality.
- Patch extraction: Sub-patches are resized to equal area, using square patches wherever possible to reduce variation in patch aspect ratio.
- Multi-resolution search: Multi-resolution search extracts sub-patches at different sizes and locations to handle objects occurring at arbitrary positions and scales.With L = 4, the method extracts 30 sub-patches from each image.
- Jittering: Jittering extracts multiple query patches instead of one, improving robustness when the reference contains only part of an item or depicts it at a larger scale.
- Feature preprocessing: PCA whitening reduces the representation dimensionality by half after computing a covariance matrix from training images.
- Similarity measurement: The pipeline computes similarity between query and reference images from sets of feature vectors generated by reference-image sub-patches.
5. Results
The evaluation reports results on five standard image retrieval datasets, including building and sculpture benchmarks.
- Results are reported on five standard image retrieval datasets.
- Oxford5k and Paris6k are included as building retrieval datasets.
- Sculptures6k is included among the evaluated retrieval datasets.
Medium footprint representation & choice of parameters
The pipeline uses convolutional features with multi-scale search, spatially consistent pooling, and distance aggregation to handle changes in scale, position, and viewpoint. Performance varies with image structure and scale, while the resulting medium and small representations outperform or compare favorably with state-of-the-art alternatives.
- Distance computation: The final distance sums each query sub-patch’s minimum distance to the reference image.The vector distance is typically L2 normalized; similarity measures replace the minimum with a maximum.
- Choice of parameters: Increasing input resolution helps Oxford5k by describing distinctive small sub-patches, but hurts Sculpture6k when global structure is more informative.The experiments resize images to an area of 600×600 pixels.
- Pooling and post-processing: Spatial pooling preserves spatial consistency and boosts Sculpture performance when the sculpture boundary is the primary information source.The reported results exclude query expansion and re-ranking post-processing.
- Medium footprint representation: The ConvNet pipeline outperforms hand-crafted representations such as VLAD and IFV for medium-sized representations.VLAD and IFV generally require additional learning iterations from specialized training data.
- Small footprint representations: Small-memory ConvNet representations can use aggressive quantization to reduce memory while being compared with methods using 128 or more dimensions.The small-footprint setting targets memory-constrained retrieval scenarios.
- Choice of parameters: Multi-resolution search helps most when query items appear smaller in reference images, but degrades UKB performance when item scales are uniform.Jittering and PCA whitening are almost always useful according to the reported pipeline ablation.
6. Conclusion
The paper presents an efficient ConvNet-based retrieval pipeline that explicitly incorporates multi-scale local features and geometric invariance. Across five standard datasets, appropriately extracted generic ConvNet representations outperform state-of-the-art methods, while the reported system remains a baseline open to domain adaptation and representation combinations.
- 6. Conclusion: The proposed pipeline uses multi-scale local ConvNet features to account explicitly for geometric invariance.The approach selects the last convolutional layer and adapts max-pooling.
- 6. Conclusion: Across five standard image retrieval datasets, generic ConvNet representations outperform other state-of-the-art methods in all tested cases when extracted appropriately.The pipeline does not rely on dataset bias; specialized training data is used only for PCA whitening.
- 6. Conclusion: The reported result is a baseline, with domain adaptation and concatenating multi-scale, multilayer, or different-architecture representations identified as routes to higher performance.Fine-tuning with a landmark dataset reaches 85.3 on Oxford5k, while concatenation reaches 87.2.