Source-linked AI summary
Multi-scale Orderless Pooling of Deep Convolutional Activation Features
Yunchao Gong, Liwei Wang, Ruiqi Guo, Svetlana Lazebnik
TL;DR
Global CNN activations retain too much spatial information for robust recognition under geometric variation. MOP-CNN combines whole-image activations with separately VLAD-pooled local activations across scales, substantially outperforming global CNN features across supervised classification and unsupervised retrieval tasks.
Problem
Global CNN activations lack geometric invariance, while CNN features retain substantial global spatial information that can limit robustness for variable recognition tasks.
Method
MOP-CNN extracts CNN activations from whole images and local patches at multiple scales, separately VLAD-pools finer-scale responses, and concatenates the resulting representations.
Results
MOP-CNN substantially improves over global CNN activations across classification and retrieval, achieving 51.98% accuracy on SUN397 with lower feature dimensionality than the cited Fisher Vector method.
Takeaways & Limitations
The resulting low-dimensional representation generalizes across supervised classification and unsupervised retrieval using the same parameters.
Takeaways & Limitations
The feature-extraction stage is somewhat slow, and ILSVRC results do not achieve state of the art relative to stronger CNN baselines and ensembles.
Abstract
from arXiv · showhide
Deep convolutional neural networks (CNN) have shown their promise as a universal representation for recognition. However, global CNN activations lack geometric invariance, which limits their robustness for classification and matching of highly variable scenes. To improve the invariance of CNN activations without degrading their discriminative power, this paper presents a simple but effective scheme called multi-scale orderless pooling (MOP-CNN). This scheme extracts CNN activations for local patches at multiple scale levels, performs orderless VLAD pooling of these activations at each level separately, and concatenates the result. The resulting MOP-CNN representation can be used as a generic feature for either supervised or unsupervised recognition tasks, from image classification to instance-level retrieval; it consistently outperforms global CNN activations without requiring any joint training of prediction layers for a particular target dataset. In absolute terms, it achieves state-of-the-art results on the challenging SUN397 and MIT Indoor Scenes classification datasets, and competitive results on ILSVRC2012/2013 classification and INRIA Holidays retrieval datasets.
1 Introduction
MOP-CNN addresses the tension between globally ordered CNN activations and orderless representations by pooling CNN features from local patches at multiple scales. It improves robustness and discriminative performance across classification and retrieval without target-dataset fine-tuning.
- The proposed scheme significantly outperforms global CNN activations on supervised classification and unsupervised retrieval without fine-tuning on target datasets.
- CNN activations preserve substantial global spatial information, whereas orderless bag-of-features representations provide greater tolerance to geometric variation.The paper positions the two representation types toward opposite ends of an orderless-to-globally-ordered spectrum.
- MOP-CNN extracts deep activation features from local patches at multiple scales, including the whole image, then aggregates finer-scale responses with VLAD.The whole-image level preserves global layout, while finer scales capture local details and VLAD supplies orderless aggregation.
- The introduction reports comprehensive evaluations on SUN397, MIT Indoor Scenes, ILSVRC2012/2013, and Holidays, with substantial performance gains across these benchmarks.
2 The Proposed Method
The method builds MOP-CNN from global and local CNN activations at three scales, separately pools local responses with VLAD, reduces dimensions, and concatenates the normalized level representations.
- MOP-CNN uses a reverse-SPM strategy that extracts patches from the whole image down to finer scales and pools each scale without spatial information.
- The representation has three levels: global 256 × 256 activations, 128 × 128 patch activations, and 64 × 64 patch activations.
- For the two local levels, 4096-dimensional patch activations are reduced to 500 dimensions with PCA and pooled using separate 100-center k-means codebooks.
- After power and L2 normalization, pooled vectors are reduced to 4096 dimensions, unit-normalized with the global vector, and concatenated into the final representation.
3 Analysis of Invariance
The invariance study compares global CNN activations with MOP-CNN under image transformations and local-window variation. It evaluates whether orderless, multi-scale pooling improves robustness while preserving useful classification signals.
- Transformation invariance: 397-way SUN classification evaluates global CNN and MOP-CNN under translation, scaling, flipping, and rotation transformations.The study trains one-vs-all linear SVMs and applies these transformations to test images.
- Local-window robustness: Global CNN predictions can change drastically across small image sub-windows, including incorrect labels for overlapping windows and correct labels for selected sub-windows.Examples include “alp” versus “garfish” and a whole-image error corrected by a sub-window labeled “schooner.”
- Local-window robustness: A sliding-window alternative improves ILSVRC2012/2013 accuracy from 54.34% for global image windows to 56.30%.The protocol samples windows at multiple scales and locations, then selects the maximum class score.
- Generic representation: Window-sampling methods improve robustness over single global CNN activations but depend on final prediction-layer responses and task-specific training.This limits their direct use as generic feature vectors for other datasets, tasks, or unsupervised retrieval.
4 Large-Scale Evaluation
Across classification and retrieval benchmarks, MOP-CNN consistently benefits from VLAD pooling and combining multiple scale levels, achieving strong results while remaining sensitive to dataset characteristics and the underlying CNN.
- Pooling baselines: VLAD pooling outperforms average and maximum pooling, while concatenating scale levels separately outperforms pooling all patches together.These comparisons are reported as baseline analyses for SUN397 and are described as consistent on ILSVRC2012/2013.
- SUN397: 51.98% accuracy on SUN397 surpasses the 47.2% Fisher Vector result with an order-of-magnitude lower feature dimensionality.MOP-CNN also substantially improves over the 39.57% global Caffe activation baseline.
- Class-level behavior: MOP-CNN improves most on SUN classes with high spatial variability or no clear focal object, but can hurt classes containing a centered object.The figure identifies classes with the biggest decreases and increases relative to level1 global features.
- MIT Indoor: 68.88% accuracy on MIT Indoor exceeds Doersch et al.'s 64.03%, with level2 and level3 features outperforming level1.The authors attribute this pattern to indoor scenes being better described by distinctive local patches whose locations vary.
- ILSVRC2012/2013: 57.93% accuracy on ILSVRC2012/2013 exceeds full pre-trained Caffe tested globally or on ten sub-windows, although MOP-CNN is not state of the art there.The authors relate the lower absolute standing to the Caffe baseline and suggest stronger CNN models could improve performance.
- Holidays retrieval: 80.8% mAP on Holidays is obtained with PCA and whitening using a 2048-dimensional feature, while the uncompressed three-level representation reaches 78.82%.The method is described as comparable to or better than compact-descriptor state-of-the-art results, but below methods using very large inverted-file vocabularies.
5 Discussion
MOP-CNN builds a generic low-dimensional representation from deep activations of local image patches, improving over global CNN activations across challenging datasets. Its consistent parameters support generalization, while feature extraction remains a practical efficiency concern.
- The scheme is built on deep activation features extracted from local image patches.
- MOP-CNN substantially improves over global CNN activations on four challenging datasets and sometimes outperforms the state of the art.The gains use the same patch sizes, sampling, codebook size, and PCA dimension across datasets.
- The representation generalizes across supervised classification and unsupervised retrieval tasks.Its low-dimensional descriptor is not restricted to image classification.
- Feature extraction is somewhat slow, motivating future work that exploits convolutional-network structure to accelerate the pipeline.The paper points to ongoing progress in faster multi-scale and dense architectures.