PaperlayerView exact revision on arXiv

Source-linked AI summary

Particular object retrieval with integral max-pooling of CNN activations

Giorgos Tolias, Ronan Sicre, Hervé Jégou

arXiv:1511.05879v2cs.CV

TL;DR

CNN descriptors were limited in compatibility with geometry-aware re-ranking and could trail traditional systems on particular-object retrieval benchmarks. The paper uses compact multi-region CNN representations and integral-image-based localization for both retrieval stages, producing results competitive with traditional methods on Oxford5k and Paris6k.

  • Problem

    CNN representations were not compatible with geometry-aware re-ranking and could be outperformed by traditional particular-object retrieval systems.

  • Method

    The paper uses compact convolutional representations for multiple regions, generalized-mean integral images for max-pooling localization, and localization-based re-ranking.

  • Results

    The combined approach competes with state-of-the-art local-feature re-ranking methods on Oxford and Paris benchmarks and substantially outperforms previous CNN-based methods.

  • Takeaways & Limitations

    The same CNN-derived information can support both compact initial retrieval and localization-based re-ranking, with the highest reported performance on the Paris dataset.

  • Takeaways & Limitations

    The representation assumes objects are upright and relies on CNN-provided tolerance to rotation and scale changes.

Abstract

from arXiv · show

Recently, image representation built upon Convolutional Neural Network (CNN) has been shown to provide effective descriptors for image search, outperforming pre-CNN features as short-vector representations. Yet such models are not compatible with geometry-aware re-ranking methods and still outperformed, on some particular object retrieval benchmarks, by traditional image search systems relying on precise descriptor matching, geometric re-ranking, or query expansion. This work revisits both retrieval stages, namely initial search and re-ranking, by employing the same primitive information derived from the CNN. We build compact feature vectors that encode several image regions without the need to feed multiple inputs to the network. Furthermore, we extend integral images to handle max-pooling on convolutional layer activations, allowing us to efficiently localize matching objects. The resulting bounding box is finally used for image re-ranking. As a result, this paper significantly improves existing CNN-based recognition pipeline: We report for the first time results competing with traditional methods on the challenging Oxford5k and Paris6k datasets.

1 INTRODUCTION

The paper revisits CNN-based image retrieval by using convolutional activations for both initial filtering and re-ranking. It introduces compact regional representations and efficient localization, achieving competition with local-feature re-ranking methods on Oxford and Paris benchmarks.

  • The work revisits both filtering and re-ranking stages using CNN convolutional-layer activations.
  • The proposed compact representation encodes multiple image regions without repeatedly feeding inputs to the CNN.The same primitive representation supports initial retrieval and re-ranking.
  • Generalized mean pooling enables integral images for efficient max-pooling and direct object localization in CNN activation maps.
  • Localization supports image re-ranking and a simple query-expansion method.
  • The combined system competes with state-of-the-art local-feature re-ranking approaches on the Oxford and Paris building benchmarks.It also outperforms previous CNN-based methods by a large margin while being more efficient in practice.

2 RELATED WORK

Related retrieval systems commonly use Bag-of-Words pipelines with locally invariant features, large codebooks, and geometric refinement. CNN approaches offer alternative representations, while prior localization methods face computational or pooling limitations addressed by this work.

  • CNN representations derive image features from fully connected or convolutional activations for image and particular-object retrieval.
  • Convolutional representations use stacking, spatial max-pooling, or sum-pooling, and can generalize better to data far from training data.
  • Fully connected features are costly for large-scale re-ranking and are not optimal for particular-object matching.
  • Integral images efficiently evaluate sum-pooled regions, whereas exhaustive sliding-window localization remains costly and prior methods differ in their handling of max-pooling.

3 BACKGROUND

The method discards fully connected CNN layers and represents convolutional responses as non-negative spatial feature maps. MAC summarizes each channel by its maximum response, yielding a translation-invariant image descriptor but discarding activation locations.

  • A pre-trained CNN is used after discarding its fully connected layers, leaving a W × H × K convolutional activation tensor.K denotes output feature channels, while W × H depends on the network, layer, and input resolution.
  • ReLU activations ensure that all convolutional response values are non-negative.
  • The tensor is represented as K two-dimensional feature-channel response maps, from which spatial max-pooling constructs the descriptor.
  • MAC compares images using cosine similarity between K-dimensional vectors formed by channel-wise maximum activations.
  • Global max-pooling makes MAC translation-invariant but removes activation-location information.The representation captures each convolutional filter’s maximum local response.
  • The approach assumes upright objects and relies on CNN-provided tolerance to rotation and scale changes.

4 ENCODING REGIONS INTO SHORT VECTORS

The paper encodes rectangular CNN-map regions with channel-wise maxima, then aggregates normalized regional vectors into a compact R-MAC image signature. Regions are sampled across multiple scales without re-feeding additional inputs to the CNN.

  • Regional feature vectors are used to represent image regions and are aggregated into a short signature for retrieval filtering.
  • For a rectangular region R, each feature component f_R,i is the maximum activation of channel i within R.
  • Regions are defined on CNN response-map coordinates rather than directly on the input image plane.
  • Multiple regional representations are constructed without re-feeding additional inputs to the CNN, reducing processing cost.
  • R-MAC samples square regions at multiple scales with approximately 40% overlap between consecutive regions.The largest-scale region is as large as possible, with dimensions based on min(W, H).
  • Each regional vector is ℓ2-normalized, PCA-whitened, ℓ2-normalized again, summed across regions, and finally ℓ2-normalized.The resulting dimensionality equals the number of feature channels.

5 OBJECT LOCALIZATION

The method extends integral images to approximate max-pooling over CNN response maps, enabling efficient regional feature computation and object localization. It then searches and refines candidate windows to identify regions matching a query object while reducing background influence.

  • Approximate integral max-pooling: The method approximates max-pooling over non-negative CNN response maps with generalized means, enabling integral-image computation for regional feature vectors.The generalized-mean estimate converges to the maximum as α increases, and integral images reduce each regional sum to four terms per channel.
  • Approximate integral max-pooling: A high exponent improves approximation accuracy, whereas applying the approximation over more response elements makes it less precise.The approximation error is evaluated over image regions with different response-set sizes.
  • Approximate integral max-pooling: The approximate feature vectors closely match exact vectors in cosine similarity, and the experiments use α = 10.The reported similarity distribution is measured over all possible regions in 10 randomly selected images.
  • Window detection: For a query represented by MAC, the method selects the CNN-activation region in a database image with maximum cosine similarity to the query.The selected region is mapped back to the original image with rough spatial precision and reduces the influence of background clutter.
  • Window detection: AML samples candidate regions uniformly, discards incompatible aspect ratios, and refines the best candidates by coordinate descent with bounded updates.The refinement allows a maximum change of 3 units and is repeated up to 5 times.

6 RETRIEVAL, LOCALIZATION AND RE-RANKING

The pipeline uses MAC or R-MAC vectors for initial ranking, then applies approximate maximum-localization to a shortlist for re-ranking. A subsequent query-expansion step averages the top five retrieved images with the query vector.

  • Initial retrieval: Initial retrieval ranks database images by cosine similarity between their MAC or R-MAC vectors and the query vector.The corresponding compact feature vector is computed for every database image and at query time.
  • Re-ranking: Re-ranking applies AML to query–database pairs in a shortlist of N top-ranked images and scores each image using its most similar localized region.The query uses MAC, while the database image is represented by its convolutional activation maps.
  • Re-ranking: Filtering may use whitened MAC or R-MAC, while localization uses similarity to l2-normalized MAC before the selected region is rescored with the filtering representation.The resulting similarity score determines the re-ranking.
  • Query expansion: Query expansion averages the query vector with the 5 top-ranked images and uses the mean vector to re-rank the top N images again.The expansion is performed after re-ranking has brought positive images to the highest positions.

7 IMPLEMENTATION DETAILS

Implementation accelerates approximate max-pooling with response quantization, lookup tables, and sparse storage. Table 1 compares localization efficiency against exhaustive search and reports MAC and R-MAC retrieval performance on Oxford5k.

  • Acceleration: Thresholding responses above 128 and flooring them causes insignificant losses while enabling lookup-table computation of α-th powers and approximate roots.Binary search on the same lookup table accelerates the α-th root computation.
  • Memory: On Oxford buildings, 81% of CNN response values are zero, and quantizing responses into 8 values further reduces memory requirements.Nonzero positions are stored using delta coding.
  • Evaluation: Table 1 reports average IoU and evaluated-window percentage for exhaustive versus sampled-and-refined localization, alongside MAC and R-MAC mAP at different input resolutions on Oxford5k.The table’s right side uses mAP, and resolution denotes the maximum input-image dimension.

8 EXPERIMENTS

The experiments evaluate compact CNN representations, AML localization, retrieval re-ranking, and comparisons with established compact and local-feature methods on Oxford and Paris benchmarks. The results show improved CNN-based retrieval, efficient approximate localization, and competitive performance with local-feature approaches.

  • Experimental setup: 5063 Oxford5k images and 6412 Paris6k images, expanded with 100k Flickr images and a 1 million-image distractor set, were evaluated using mAP.The study follows the standard protocol, including query bounding boxes and cross-dataset PCA learning.
  • Localization accuracy: 51.3% IoU is achieved by speeded-up AML on Oxford5k, compared with 52.6% for exhaustive evaluation, while localization becomes approximately 180 times faster.On Paris6k, the corresponding IoU values are 51.4% and 52.9%.
  • Retrieval and re-ranking: R-MAC reaches 66.9 mAP on Oxford5k with VGG16 after aggregating regions from three scales, versus 63.0 using only the largest scale.Adding the second scale increases performance to 65.4 before the finest scale produces the final 66.9 score.
  • Retrieval and re-ranking: 13% mAP improvement is obtained when AML re-ranks retrieval results with 1 million distractor images.On Oxford105k, performance consistently improves as up to 1000 images are re-ranked, and query expansion adds benefit at low extra cost.
  • Localization accuracy: AML provides rough object localization intended to improve image similarity, with accuracy sufficient for re-ranking rather than precise localization.The approach uses convolutional activations and efficiently evaluates candidate regions through max-pooling-compatible integral images.
  • Comparison to the state of the art: The proposed short-sized R-MAC outperforms other compact representations, while AML competes with local-feature methods using geometric matching and query expansion.AML achieves the best reported performance on Paris6k in the cited comparison, while results remain lower on Oxford.

9 CONCLUSIONS

The method reuses convolutional CNN activations across filtering and re-ranking, combining compact regional representations with efficient localization. It achieves the highest performance on the Paris dataset and competes with methods using costly geometric matching or query expansion.

  • 9 CONCLUSIONS: The pipeline reuses convolutional-layer activations for both initial filtering and image re-ranking.Its compact representation and localization operate within the same CNN-based retrieval system.
  • 9 CONCLUSIONS: The compact vector representation encodes several image regions and outperforms state-of-the-art competitors.The representation is built from CNN activations using a simple aggregation method.
  • 9 CONCLUSIONS: The localization component improves retrieval performance after the system is initially built around the compact representation.Localization supplies the basis for the re-ranking stage.
  • 9 CONCLUSIONS: The approach competes with state-of-the-art methods that use costly geometric matching or query expansion.The paper reports the highest performance on the Paris dataset.
Loading 1511.05879v2…