Source-linked AI summary
Faster R-CNN Features for Instance Search
Amaia Salvador, Xavier Giro-i-Nieto, Ferran Marques, Shin'ichi Satoh
TL;DR
This paper studies instance search using global and local representations from Faster R-CNN, including off-the-shelf and fine-tuned features. It builds a filtering-and-spatial-reranking pipeline and reports competitive results, with fine-tuning greatly improving off-the-shelf performance at added training cost.
Problem
The paper examines how object-detection CNN representations can support effective and efficient instance retrieval from visual databases.
Method
The system extracts global and local CNN features from Faster R-CNN, uses RPN proposals for spatial reranking, and fine-tunes the detector for target query objects.
Results
Fine-tuning greatly improves the off-the-shelf system's performance, while the proposed pipeline achieves competitive results on Oxford Buildings, Paris Buildings, and TRECVid Instance Search.
Takeaways & Limitations
Faster R-CNN features provide a simple baseline for describing whole images and sub-parts, with stronger performance available when the CNN is fine-tuned for the queried objects.
Takeaways & Limitations
For TRECVid Instance Search 2013, fine-tuning only fully connected layers may be insufficient to effectively detect challenging query objects.
Abstract
from arXiv · showhide
Image representations derived from pre-trained Convolutional Neural Networks (CNNs) have become the new state of the art in computer vision tasks such as instance retrieval. This work explores the suitability for instance retrieval of image- and region-wise representations pooled from an object detection CNN such as Faster R-CNN. We take advantage of the object proposals learned by a Region Proposal Network (RPN) and their associated CNN features to build an instance search pipeline composed of a first filtering stage followed by a spatial reranking. We further investigate the suitability of Faster R-CNN features when the network is fine-tuned for the same objects one wants to retrieve. We assess the performance of our proposed system with the Oxford Buildings 5k, Paris Buildings 6k and a subset of TRECVid Instance Search 2013, achieving competitive results.
1. Introduction
The paper addresses instance search by combining Faster R-CNN’s global and local features with spatial reranking and fine-tuning. It evaluates this approach across three retrieval benchmarks.
- The work targets instance search, retrieving database images that contain a queried instance.
- The proposed system extracts global and local convolutional features from an object-detection CNN in a single image pass.
- Spatial reranking uses Region Proposal Network locations to provide rough object localization for top-ranked images.
- The study fine-tunes the detection CNN for the target instances and finds this suitable for learning better image representations.
- The resulting pipeline combines local and global object-detection features, with rankings illustrated across TRECVid INS 2013, Paris Buildings, and Oxford Buildings.
2. Related Work
Prior instance-search work established CNN-based retrieval and increasingly favored convolutional features and pooled region descriptors. This paper applies those ideas using Faster R-CNN to obtain image- and region-based features in one forward pass.
- Earlier retrieval systems used fully connected features, while later work found convolutional layers substantially stronger for image retrieval.
- Sum-pooled convolutional descriptors and region-aggregation methods provided compact representations for image retrieval.
- Unlike prior approaches, this work uses an object-detection CNN to extract image- and region-based convolutional features in a single forward pass.
- The method leverages Faster R-CNN’s end-to-end architecture to extract both image and region features for instance search.
3. Methodology
The methodology uses Faster R-CNN to pool global image and local proposal features, then performs filtering, spatial reranking, and query expansion. Fine-tuning adapts representations and reranking scores to the queried instances.
- 3.1. CNN-based Representations: Faster R-CNN combines a Region Proposal Network that learns window locations with a classifier that labels each proposed window.
- 3.1. CNN-based Representations: Image-wise pooling aggregates last-convolutional-layer activations into a global descriptor, while region-wise pooling aggregates activations for RPN proposals.
- 3.2. Fine-tuning Faster R-CNN: Fine-tuning either updates only fully connected classification layers or updates nearly all layers after the first two convolutions.
- 3.3. Instance Retrieval Pipeline: The pipeline first ranks all database images by cosine similarity between image descriptors, then locally analyzes and reranks the top N results.
- 3.3. Instance Retrieval Pipeline: Class-agnostic reranking compares proposal descriptors with the query region, whereas class-specific reranking uses fine-tuned proposal classification scores.
- 3.3. Instance Retrieval Pipeline: Query expansion averages the descriptors of the top M ranked images with the query descriptor to perform a new search.
4. Experiments
Experiments evaluate Faster R-CNN image- and region-wise descriptors on Oxford Buildings, Paris Buildings, and INS13, comparing pooling, architectures, query expansion, and fine-tuning. Fine-tuning deeper network layers improves retrieval and proposal localization, while the overall pipeline remains competitive with prior systems.
- Off-the-shelf Faster R-CNN features: The off-the-shelf pipeline uses IPA-sum descriptors for filtering and RPA-max descriptors for spatial reranking.Sum pooling is stronger for filtering, whereas max pooling is stronger for region-wise reranking.
- Off-the-shelf Faster R-CNN features: Sum pooling significantly outperforms max pooling during filtering, while max-pooled region descriptors improve Oxford and Paris reranking by 0.10 and 0.03 mAP points.Spatial reranking has little or no mAP effect on INS13, although max pooling produces more accurate object locations qualitatively.
- Off-the-shelf Faster R-CNN features: VGG16 features generally outperform ZF features, and query expansion after spatial reranking provides significant gains on all tested datasets.Query expansion uses the M = 5 top retrieved images in the reported architecture comparison.
- Fine-tuning Faster R-CNN: Fine-tuning Strategy #2 changes the last convolutional layer, increases activation responses to query-object patterns, and produces tighter object locations than Strategy #1.Adapting the RPN makes its proposals more suitable for objects in the test datasets.
- Fine-tuning Faster R-CNN: Fine-tuned features significantly outperform raw Faster R-CNN features across all datasets, with mAP gains of approximately 20% for Oxford and Paris and 8% for INS13.The benefit of spatial reranking and query expansion varies by dataset, with INS13 gaining most from fine-tuned features.
- Comparison with state-of-the-art: The Faster R-CNN pipeline achieves competitive state-of-the-art results, while fine-tuned features are especially effective when retrieval time is not a constraint.The reported comparison attributes remaining performance differences partly to architecture, training data, and input image size.
5. Conclusion
The paper presents strategies for using object-detection CNN features in instance search and shows that fine-tuning can greatly improve off-the-shelf performance.
- Fine-tuning the CNN for query images greatly improves the performance of an off-the-shelf Faster R-CNN feature system.The approach incurs the cost of fine-tuning the CNN for the target objects.