Source-linked AI summary
Cross-dimensional Weighting for Aggregated Deep Convolutional Features
Yannis Kalantidis, Clayton Mellina, Simon Osindero
TL;DR
Compact image search representations require effective aggregation of deep convolutional features. The paper introduces a generalized cross-dimensional weighting framework with non-parametric spatial and channel weighting, and reports state-of-the-art performance for pre-trained-network approaches with minimal overhead.
Problem
Existing image-search methods use varied pooling and aggregation strategies, motivating a general framework for combining cross-dimensional weighting with deep convolutional features.
Method
The method weights convolutional features spatially and by channel before sum-pooling, using non-parametric schemes that emphasize active responses and regulate bursty channels.
Results
Over 10% higher mean average precision is reported on Oxford and Paris than the previous state-of-the-art for compact visual representations.
Takeaways & Limitations
CroW features achieve state-of-the-art image-retrieval results with minimal computational overhead and can be combined for query expansion.
Takeaways & Limitations
Learning weights for a particular task and fine-tuning the full end-to-end process are identified as promising future directions requiring sufficient ground-truth data.
Abstract
from arXiv · showhide
We propose a simple and straightforward way of creating powerful image representations via cross-dimensional weighting and aggregation of deep convolutional neural network layer outputs. We first present a generalized framework that encompasses a broad family of approaches and includes cross-dimensional pooling and weighting steps. We then propose specific non-parametric schemes for both spatial- and channel-wise weighting that boost the effect of highly active spatial responses and at the same time regulate burstiness effects. We experiment on different public datasets for image search and show that our approach outperforms the current state-of-the-art for approaches based on pre-trained networks. We also provide an easy-to-use, open source implementation that reproduces our results.
1 Introduction
The paper introduces cross-dimensional weighting and aggregation of deep convolutional features to create compact image representations, reporting improved visual image-search performance with minimal overhead.
- Deep CNN features have become dominant in visual image search and have surpassed many domain-specific approaches.
- The paper presents a generalized framework for cross-dimensional weighting and aggregation, including spatial and channel-wise weighting steps.
- CroW features apply spatial and channel weighting to convolutional outputs before sum-pooling into a final aggregation.
- The proposed non-parametric schemes boost highly active spatial responses and regulate channel burstiness.
- Over 10% higher mean average precision is reported on Oxford and Paris than the previous state-of-the-art for compact visual representations.
- The implementation is open source and reproduces the reported results.
2 Related work
Related work progresses from scalable challenges in local-feature image search toward global and deep-feature representations, with several pooling and aggregation strategies establishing strong benchmarks.
- Bag-of-words approaches using local features perform well for object retrieval but are difficult to scale because images contain hundreds of patches.
- Global representations such as VLAD and Fisher Vectors narrowed the performance gap with bag-of-words methods for image search.
- Pre-trained CNN features achieved state-of-the-art results across computer-vision tasks, including image search.
- CNN image-search methods explored aggregating fully connected and convolutional responses, with max-pooling, normalization, and whitening producing strong low-dimensional codes.
- SPoC uses sum-pooling and is closely related to the paper’s framework, while R-MAC provides complementary cross-region max-pooling.
- Fine-tuned CNN retrieval methods became state of the art using annotated data and training losses, whereas the proposed features do not require training or clean annotated data.
3 Framework for Aggregation of Convolutional Features
The framework transforms a convolutional feature tensor through local pooling, spatial and channel weighting, aggregation, normalization, dimensionality reduction, and final normalization.
- The framework begins with spatially local sum- or max-pooling over each channel, with global pooling and no pooling as limiting cases.
- The weighted tensor is sum-pooled channel-wise to produce an aggregate feature vector.
- Spatial weighting assigns α_ij to each location, while channel weighting assigns β_k to every location in channel k.
- The resulting vector is normalized and power-transformed, reduced with PCA that may include whitening, and normalized again.
- The input is a three-dimensional tensor X with K channels and spatial dimensions W and H, which may vary with the original image size.
- Algorithm 1 specifies the framework’s inputs, including tensor X, pooling parameters, spatial and channel weight functions, normalization settings, whitening parameters, and final dimensionality.
4 Feature Weighting Schemes
The paper proposes non-parametric spatial and channel weighting schemes within a cross-dimensional aggregation framework. Spatial weighting emphasizes locations with strong, co-occurring responses, while sparsity-sensitive channel weighting reduces bursty channel effects and preserves infrequent signals.
- Spatial weighting: Spatial weighting is derived from normalized aggregate responses across channels and applied independently at every spatial location and channel.The aggregate response map is normalized and power-scaled before its values become spatial weights αij.
- Spatial weighting: L2 normalization with b = 2 was selected because alternative normalization schemes produced image-search performance differences usually below 0.01 mAP.
- Spatial weighting: Spatial weighting boosts salient locations and locations where multiple channels are active, while down-weighting non-salient locations.The authors describe this as a computationally cheap way to favor feature co-occurrence while accounting for response strength.
- Channel weighting: Channel weighting uses channel sparsity, with sparsity patterns highly correlated within landmark classes and less correlated across different landmarks.The Paris query set contains 55 images: five images for each of 11 landmark classes.
- Channel weighting: Sparsity-sensitive channel weighting down-weights bursty filters while boosting infrequently occurring features that may provide class-consistent signal.Channels producing non-zero responses across many image regions are treated as spatially recurring visual elements.
- Framework comparison: The framework distinguishes CroW, uCroW, and SPoC through their pooling, spatial weighting, and channel weighting choices.CroW uses learned-from-activation spatial weighting and sparsity-sensitive channel weighting, while SPoC uses a centering prior and uniform channel weighting.
5 Experiments
Experiments evaluate CroW and its weighting variants on public image-search datasets, examining preprocessing, layer choice, dimensionality, whitening, and query expansion. CroW generally outperforms competing aggregation methods, with gains over uCroW increasing at lower dimensions and smaller advantages on Holidays.
- Experimental setup: Experiments use Oxford, Paris, Oxford100k distractors, and Holidays, measuring mean average precision with pre-trained VGG16 features.Oxford uses cropped queries, while Holidays uses upright images; cited table rows report results from prior papers.
- Pooling: Max pooling is beneficial before aggregation, while sum pooling is preferred over max pooling for final aggregation when whitening is used.Without whitening, max pooling can perform better, but unwhitened mAP is sometimes more than 10% lower across tested datasets.
- Image size and layer selection: Keeping original image size improves performance, while conv5-3 and especially pool5 outperform earlier conv4 and pool4 layers.The figure compares original-size inputs with images resized to 586 × 586 and reports 0.58 mAP for conv4 and pool4 at d = 128.
- Feature weighting and dimensionality: CroW combines spatial and sparsity-sensitive channel weighting, whereas uCroW applies uniform or no weighting; uCroW +SSW is more affected by dimensionality reduction.The stated explanation is that sparsity-sensitive weights use all dimensions before final reduction, which discards some of them.
- Image-search results: CroW and uCroW consistently outperform other aggregation methods across Paris, Oxford, and Holidays at different representation sizes, except that R-MAC is very strong on Paris at 512 dimensions.CroW improves over the strong uCroW baseline in all cases, with larger gains as final dimensionality decreases.
- Image-search results: On Holidays, CroW outperforms the best competing method by about 4.9% at d = 256 and 1.2% at d = 512.
6 Conclusions
The paper presents a generalized framework for cross-dimensional weighting of aggregated deep convolutional features and instantiates it as CroW. CroW achieves state-of-the-art image-retrieval performance with minimal overhead, while the framework also supports future learned weighting and end-to-end fine-tuning.
- Contributions: The framework encompasses aggregated convolutional features with cross-dimensional weighting and includes non-parametric spatial- and channel-wise weighting schemes.
- Conclusions: CroW is one instantiation of the generic aggregation framework and achieves current state-of-the-art image-retrieval results with minimal overhead.
- Future directions: The framework provides a scaffold for exploring weighting schemes independently and could support task-specific learned weights or end-to-end fine-tuning with sufficient ground-truth data.
- Conclusions: The Paris benchmark visualization reports 3 false results across top-10 results for all 55 queries using 512-dimensional CroW features.