Source-linked AI summary
Bags of Local Convolutional Features for Scalable Instance Search
Eva Mohedano, Amaia Salvador, Kevin McGuinness, Ferran Marques, Noel E. O'Connor, Xavier Giro-i-Nieto
TL;DR
Instance retrieval needs representations that retain CNN effectiveness while scaling to challenging search settings. The paper encodes local convolutional features with BoW, uses assignment maps for spatial reranking and query expansion, and reports competitive Oxford and Paris results plus significant gains over sum pooling on TRECVid INS.
Problem
Challenging TRECVid INS systems had not adopted pipelines relying solely on CNN features, despite strong CNN performance on Oxford and Paris.
Method
BLCF assigns local convolutional activations to visual words, uses the assignment map for spatial reranking and object localization, then performs query expansion.
Results
The method achieves competitive performance on Oxford and Paris and significantly outperforms sum pooling on the TRECVid INS subset.
Takeaways & Limitations
BoW aggregation of local CNN features is suitable for scalable instance retrieval, with benefits in index size, indexing cost, and search time.
Abstract
from arXiv · showhide
This work proposes a simple instance retrieval pipeline based on encoding the convolutional features of CNN using the bag of words aggregation scheme (BoW). Assigning each local array of activations in a convolutional layer to a visual word produces an \textit{assignment map}, a compact representation that relates regions of an image with a visual word. We use the assignment map for fast spatial reranking, obtaining object localizations that are used for query expansion. We demonstrate the suitability of the BoW representation based on local CNN features for instance retrieval, achieving competitive performance on the Oxford and Paris buildings benchmarks. We show that our proposed system for CNN feature aggregation with BoW outperforms state-of-the-art techniques using sum pooling at a subset of the challenging TRECVid INS benchmark.
1. INTRODUCTION
The paper revisits BoW encoding for local CNN features to address scalable instance search beyond conventional CNN-based pipelines. It introduces a sparse descriptor, assignment map, and localization-driven query expansion, reporting state-of-the-art performance across several benchmarks.
- CNN descriptors perform well on Oxford and Paris, but challenging TRECVid INS systems still rely on aggregated local hand-crafted features.
- BLCF encodes local convolutional features as a sparse visual descriptor that supports fast retrieval through an inverted index.
- The assignment map compactly maps image pixels to visual words, enabling fast BoW construction for arbitrary image regions.
- Local analysis of multiple regions supports spatial reranking and object localization, followed by query expansion using the localized regions.
- The resulting retrieval system achieves state-of-the-art performance in several instance retrieval benchmarks.
2. RELATED WORK
Prior work progresses from fully connected CNN descriptors to pooled convolutional features and localized search. The paper positions BLCF as a scalable alternative to approaches requiring many patches or costly spatial searches.
- Early CNN retrieval methods replaced hand-crafted descriptors with fully connected features from classification-pretrained networks.
- Later methods improved retrieval by using sum- or max-pooled convolutional features.
- Local CNN retrieval methods aggregate features from image patches with average pooling, BoW, or VLAD, but patch extraction slows indexing and retrieval-time feature extraction.
- Spatial search can improve performance, yet arbitrary multiscale windows are computationally costly and do not scale well to large datasets or real-time search.
3. BAG OF WORDS FRAMEWORK
BLCF treats activations at convolutional-layer locations as local descriptors and aggregates them with BoW. The resulting assignment map preserves spatial word assignments for efficient regional representations.
- Activations at different convolutional-layer locations are used as local features after discarding the network’s softmax and fully connected layers.
- Each convolutional layer contains D feature maps of size N × M, viewed as N × M local descriptors of dimension D.
- BoW produces sparse, high-dimensional codes that can be stored in inverted indices and are compact, fast to compute, and interpretable.
- K-means learns a visual codebook, and each local CNN feature is assigned to its nearest visual word to form an N × M assignment map.
- The assignment map relates image pixels to visual words and enables rapid BoW vectors for the full image and its regions.
- The pipeline encodes each image as a sparse high-dimensional descriptor for instance retrieval.
4. IMAGE RETRIEVAL
The retrieval pipeline combines scalable initial BoW search with assignment-map spatial reranking and query expansion. Regional cosine similarity identifies rough object locations for refining the ranking and query representation.
- Initial search: Initial search compares query and database BoW vectors with cosine similarity using a sparse inverted index and GPU-based sparse matrix multiplication.
- Initial search: Global search uses all query-image local CNN features, whereas local search uses only features inside the query bounding box.
- Local reranking: Local reranking applies sliding windows directly to the assignment map across multiple widths and heights with 50% overlap.
- Local reranking: Windows with aspect-ratio similarity below threshold th are discarded before regional comparison.
- Local reranking: Regional BoW vectors are compared with the query using cosine similarity, and the highest-scoring window provides a rough object localization.
- Query expansion: Global query expansion averages top-ranked image BoW vectors with the query, while local query expansion averages localized-region vectors with the query bounding-box BoW.
5. EXPERIMENTS
Experiments evaluate the pipeline across Oxford, Paris, and TRECVid INS, including comparisons with CNN retrieval methods and sum pooling. The results show competitive performance on Oxford and Paris and a stronger advantage for BoW on the challenging TRECVid INS subset.
- Datasets: The evaluation covers Oxford Buildings, Paris Buildings, and a TRECVid INS subset, with additional Oxford 105k and Paris 106k distractor-image settings.The distractor settings add 100,000 Flickr images to the original Oxford and Paris datasets.
- Pipeline: The pipeline combines initial ranking, spatial reranking, and query expansion, with local reranking applied to the top-100 images and query expansion to the top-10.The local reranking stage uses a sliding-window approach, followed by query expansion over the resulting ranking.
- Reranking and query expansion: 0.652 to 0.769: Oxford 5k performance after spatial reranking and query expansion, comparable to the 0.66 to 0.77 improvement reported for R-MAC.Local reranking is beneficial when the initial ranking comes from global BoW search, whereas it is redundant after local search.
- Reranking and query expansion: Query expansion greatly improves Oxford 5k, while Paris 6k shows no significant gain after reranking and query expansion despite its best result occurring with LS + R + LQE.The reported effect of the later pipeline stages differs across the two datasets.
- Comparison with the state of the art: The sparse BoW representation achieves consistently high mAP across tested Oxford and Paris datasets while supporting fast retrieval through inverted indices.The paper contrasts this scalability with denser or computationally costly alternatives such as VLAD and multiscale spatial search.
- TRECVid INS: Bag of Words significantly outperforms sum pooling on TRECVid INS, whereas the two methods have similar performance on Oxford and Paris.The paper relates the TRECVid advantage to queries with non-centered objects and variable size and appearance.
6. CONCLUSION
The conclusion presents BoW aggregation of CNN convolutional features as a sparse, scalable representation for instance search. It reports competitive Oxford and Paris performance and consistent superiority over sum pooling on TRECVid INS, while noting greater sensitivity to many distractors.
- Contribution: The proposed aggregation strategy encodes CNN convolutional features with Bag of Words into a sparse representation for instance search.The resulting representation supports the paper’s retrieval pipeline based on local convolutional features.
- Results: The method achieves competitive performance against other CNN representations on Oxford and Paris while reducing index size, indexing cost, and search time.The conclusion characterizes the scalability advantage relative to the compared CNN-based representations.
- Results: The BoW encoding consistently and significantly outperforms sum pooling on the more challenging TRECVid instance search task.This comparison is presented as evidence for the suitability of local CNN features with BoW aggregation on TRECVid INS.
- Limitation: The method appears more sensitive to large numbers of distractor images than sum- and max-pooling methods.The authors speculate that a larger codebook may better represent the diversity of distractor-image visual words.