Source-linked AI summary
Exploiting Local Features from Deep Networks for Image Retrieval
Joe Yue-Hei Ng, Fan Yang, Larry S. Davis
TL;DR
Instance-level image retrieval requires distinguishing objects within the same category, but classification-oriented CNN practice typically favors final layers that may lose local detail. The paper extracts convolutional features across layers, encodes them with VLAD, and evaluates layer depth and input scale using OxfordNet and GoogLeNet. Intermediate layers or higher layers at finer scales perform better, and compressed 128-D VLAD descriptors achieve state-of-the-art results on two of three datasets.
Problem
Final CNN layers are not necessarily optimal for instance-level retrieval because higher-level features may fail to preserve local characteristics needed to separate same-category objects.
Method
The paper extracts convolutional features from multiple layers of pretrained OxfordNet and GoogLeNet networks and encodes them into image descriptors with VLAD.
Results
Intermediate layers perform best across architectures and datasets, while higher layers with finer-scale inputs can capture local characteristics and improve retrieval.
Takeaways & Limitations
Compressed 128-D VLAD descriptors achieve state-of-the-art retrieval results on two of three instance image retrieval datasets.
Takeaways & Limitations
The authors identify combining multiple layers from the best scales in spatial search as future work.
Abstract
from arXiv · showhide
Deep convolutional neural networks have been successfully applied to image classification tasks. When these same networks have been applied to image retrieval, the assumption has been made that the last layers would give the best performance, as they do in classification. We show that for instance-level image retrieval, lower layers often perform better than the last layers in convolutional neural networks. We present an approach for extracting convolutional features from different layers of the networks, and adopt VLAD encoding to encode features into a single vector for each image. We investigate the effect of different layers and scales of input images on the performance of convolutional features using the recent deep networks OxfordNet and GoogLeNet. Experiments demonstrate that intermediate layers or higher layers with finer scales produce better results for image retrieval, compared to the last layer. When using compressed 128-D VLAD descriptors, our method obtains state-of-the-art results and outperforms other VLAD and CNN based approaches on two out of three test datasets. Our work provides guidance for transferring deep networks trained on image classification to image retrieval tasks.
1. Introduction
The paper questions whether final CNN layers are best for instance-level retrieval and studies layer depth and input scale as key design factors. It introduces VLAD encoding of convolutional features to preserve local instance information while retaining CNN representations.
- Final CNN layers capture semantic category information but may lose local characteristics needed to distinguish instances within the same category.
- The study extracts convolutional activations across layers and encodes them with VLAD for instance-level image retrieval.
- Higher-resolution inputs allow higher-layer filters to capture local characteristics alongside semantic concepts, producing better retrieval features.
- Systematic experiments vary convolutional layers and input scales using OxfordNet and GoogLeNet.
- The resulting framework outperforms other VLAD- and CNN-based approaches with relatively low-dimensional representations.
2. Related Work
Related work established CNN features as transferable visual representations but often relied on final-layer activations or spatial pooling. This paper instead evaluates lower convolutional layers and preserves local information for instance retrieval.
- Traditional retrieval methods use SIFT descriptors encoded with BoW, VLAD, or related representations.
- Early CNN retrieval results using final-layer features lagged behind simple SIFT-based BoW and VLAD methods without additional spatial information.
- Prior work used VLAD at the last convolutional layer with spatial pooling for latent concept descriptors.
- This paper extends convolutional-feature extraction to lower layers without additional pooling and evaluates different layers for instance-level retrieval.
3. Approach
The approach extracts local feature vectors from convolutional maps at multiple layers, encodes them with VLAD, and compresses the resulting descriptors for efficient retrieval. It is evaluated with pretrained OxfordNet and GoogLeNet networks.
- The method applies VLAD to CNN features and compresses the descriptors into compact representations for image retrieval.
- Experiments use pretrained OxfordNet and GoogLeNet convolutional networks, with OxfordNet and GoogLeNet providing alternative architectures.
- An input image is warped to an n×n square, passed through the network, and represented at layer Ll by an nl × nl × dl feature map Ml.
- Each spatial location in a convolutional feature map provides a dl-dimensional local feature vector, yielding nl^2 local vectors per layer.
- The method extracts feature sets from all convolutional layers rather than only the last convolutional layer.
- Local features are used because they preserve smaller image parts and can generalize across object categories while retaining instance-level information.
- VLAD assigns local features to visual words and accumulates residuals into a single descriptor describing their distribution.
- The original VLAD descriptor is intra-normalized, reduced with PCA, and whitened to improve efficiency and robustness.
4. Experiments
Experiments across Holidays, Oxford, and Paris compare convolutional layers and input scales in OxfordNet and GoogLeNet. Intermediate layers generally outperform the last layer, while finer scales can make higher layers more effective by preserving local detail.
- Comparison of layers: The experiments evaluate layer-wise convolutional features on Holidays, Oxford, and Paris using OxfordNet and GoogLeNet.The datasets contain 1491, 5062, and 6412 images, respectively.
- Comparison of layers: 82.0% versus 68.5% mAP on Holidays shows that GoogLeNet’s intermediate Inception 3a layer outperforms its last layer at the original scale.The best original-scale layers are Inception 3a, Inception 4a, and Inception 4e for Holidays, Oxford, and Paris.
- Comparison of layers: Intermediate layers outperform the last layer across network architectures and datasets, with OxfordNet’s conv5 1 best on Oxford and Paris rather than its final layer.The results indicate that increasing generalization at higher layers is not always useful for instance-level retrieval.
- Scales: Higher-resolution inputs generally improve performance, and on Oxford and Paris they make higher GoogLeNet layers outperform lower layers.The reported exception is OxfordNet on Holidays; OxfordNet’s multi-scale training yields similar layer behavior across scales.
- Feature visualization: Feature correspondence declines from GoogLeNet Inception 4a to Inception 5b at scale 1, where local appearance differences become blurred.At scale 2, Inception 5b retrieves more semantically relevant yet more locally diverse patches than at scale 1.
- Feature visualization: Inception 3a produces SIFT-like clusters of edges, corners, and textures, whereas finer-scale higher layers preserve more instance-level detail.These preserved details help explain why higher layers at finer scales are more useful for retrieval.
- Comparison to state-of-the-art: The multi-scale VLAD descriptors concatenate features from the best-performing layers at both scales, and OxfordNet performs slightly better than GoogLeNet.The selected layers differ by dataset and network, reflecting the layer-dependent retrieval behavior.
5. Conclusion
The work systematically evaluates convolutional features across network layers and input-image scales for instance-level retrieval, using VLAD encoding. It achieves state-of-the-art results with low-dimensional representations on two datasets.
- VLAD-encoded convolutional responses achieve state-of-the-art retrieval results with low-dimensional representations on two instance image retrieval datasets.