Source-linked AI summary
Good Practice in CNN Feature Transfer
Liang Zheng, Yali Zhao, Shengjin Wang, Jingdong Wang, Qi Tian
TL;DR
The paper asks how pretrained CNN features can be transferred effectively for image search and classification despite limited labeled data. It studies input-image size, pooling across CNN layers, and multi-layer feature fusion, reporting improved recognition and large-margin state-of-the-art gains across benchmarks. The resulting practices emphasize larger inputs, pooled intermediate features, and combining features across layers.
Problem
The paper addresses effective CNN feature transfer for image search and classification when collecting sufficient labeled training data is difficult.
Method
The paper evaluates larger input images, average/max pooling of intermediate feature maps, and concatenation of pooled features across CNN layers.
Results
The practices improve recognition performance, with pooled Conv5 features superior or competitive to fully connected features and state-of-the-art gains across multiple benchmarks.
Takeaways & Limitations
Larger images, pooled intermediate features, and multi-layer feature combinations are supported as effective practices for CNN feature transfer.
Takeaways & Limitations
The conclusion identifies further work on exploiting bottom-level CNN features through spatial constraints, multiple feature fusion, and effective encoding methods.
Abstract
from arXiv · showhide
The objective of this paper is the effective transfer of the Convolutional Neural Network (CNN) feature in image search and classification. Systematically, we study three facts in CNN transfer. 1) We demonstrate the advantage of using images with a properly large size as input to CNN instead of the conventionally resized one. 2) We benchmark the performance of different CNN layers improved by average/max pooling on the feature maps. Our observation suggests that the Conv5 feature yields very competitive accuracy under such pooling step. 3) We find that the simple combination of pooled features extracted across various CNN layers is effective in collecting evidences from both low and high level descriptors. Following these good practices, we are capable of improving the state of the art on a number of benchmarks to a large margin.
1. Introduction
The paper studies how to use pretrained CNN features effectively for image search and classification when large labeled datasets are difficult to obtain. It proposes larger input images, pooled intermediate features, and multi-layer fusion as practices that improve transfer performance.
- CNN transfer addresses image search and fine-grained classification settings where collecting sufficient labeled data is difficult or expensive.The paper motivates transfer by the wide variety of query content and the need for expert annotations.
- Larger input images can reduce information loss from down-sampling and yield consistent improvement during CNN transfer.The paper contrasts this with conventional fixed-size resizing and studies the issue across transfer datasets.
- Average or max pooling makes intermediate CNN features more invariant to image translations, and pooled Conv5 features achieve competitive accuracy with FC features.Pooling also supports lower-dimensional representations while preserving useful intermediate-layer information.
- Fusing features from multiple CNN layers combines local and global information across the hierarchy's varying receptive fields.The paper uses single-layer pooled features or concatenated pooled features from all layers for search and classification.
- The proposed practices improve results over state-of-the-art methods on three image-search and seven image-classification benchmarks.The paper presents these findings as good practices for CNN feature transfer.
2. Related Works
Prior CNN transfer work largely used fully connected features, while evaluations of intermediate layers and multi-layer combinations remained limited. This paper addresses that gap for CNN-based image search and classification.
- Fully connected features are widely preferred for their generalization and semantic descriptions, whereas intermediate features can better represent local patterns.Local information is especially relevant when images contain occlusion or truncation.
- Prior results disagree on the relative value of Conv5 and FC6 depending on whether spatial pooling or encoding is applied.Conv5 can outperform FC6 with Spatial Pyramid Pooling, while unpooled Conv5 may perform worse; VLAD-encoded Conv5 also shows strong accuracy in another benchmark.
- Multi-scale CNN cues have improved recognition, but prior approaches often combine features from multiple image scales or use same-layer representations.This paper instead focuses on fusing pooled features from different CNN layers for holistic image recognition.
- CNN features support global, regional, and local image-search representations, but memory efficiency remains important.Encoding or hashing schemes can make CNN features more practical for search.
- A detailed evaluation of CNN layers and their combinations across image search and classification was still lacking.The paper positions its empirical study as filling this evaluation gap.
3. Method
The method extracts pooled CNN features from multiple layers while preserving more input-image detail than conventional resizing. It uses layer-specific pooling and task-specific fusion to combine local and global evidence.
- Pooling: The method uses a pre-trained CNN with convolutional and fully connected layers represented as a sequence of feature-producing layers.Fully connected layers are treated as convolutional layers because full connection is a special case of convolution.
- Pooling: Average or max pooling compresses each layer’s feature maps into a 1 × 1 × c_k vector.For VGGNet Conv5, pooled features have dimension 1 × 1 × 512.
- Pooling: Pooling intermediate features emphasizes local visual structures and increases invariance to translation, occlusion, and truncation.For VGGNet, the resulting Conv5 representation is only 512-dimensional, improving computational efficiency.
- Pooling: Average-pooled Conv5 can preserve common local structures across relevant image pairs and improve their retrieval rank despite positional variation.The paper attributes this example to average pooling alleviating translation variance.
- Fusion: The method fuses features from low to high levels, spanning small to large receptive fields.For image search, adaptive late fusion assigns query-dependent weights to feature scores; for classification, features are concatenated.
- Fusion: For classification, pooled features from Conv1 through Conv5, FC6, and FC7 are concatenated into one representation.The resulting dimensions are 9,568 for AlexNet and 9,664 for VGGNet.
- Fusion: Shorter pooled representations enable fusion across increasing semantic levels without relying on prohibitively high-dimensional low-level feature vectors.The paper contrasts this with raw pool1 and pool2 features whose dimensions can already reach 69,984 and 43,264 for AlexNet inputs of 227 × 227.
- Image resizing: The method uses relatively large input images, resizing them according to the training-set average while preserving aspect ratio.The longer side is set to the larger of the average training-image height and width, rather than conventionally forcing 224 × 224 or 227 × 227 inputs.
4. Experiments
Experiments evaluate CNN feature transfer across image search and classification benchmarks, testing image scale, CNN layers, pooling, feature fusion, and comparison with state-of-the-art methods. Larger inputs, pooled intermediate features, and multi-layer combinations consistently improve performance, with strong benchmark results across tasks.
- Datasets: The study evaluates three image-search datasets and seven image-classification datasets using CNN features and standard task-specific metrics.Image search uses mAP on Holidays and Oxford5k and N-S score on Ukbench; classification includes generic, scene, and fine-grained datasets.
- CNN layers: Conv5 with average pooling improves search accuracy over FC6 and FC7 by +1.28% mAP on Holidays and +0.04 N-S on Ukbench.Conv5 is superior in 4 out of 7 classification datasets, particularly fine-grained and scene classification tasks.
- Image size: Larger image scales consistently improve search and classification accuracy, while performance stabilizes around scale 1.0.Images preserve aspect ratio, and scales above 1.0 add memory consumption without much improvement.
- Pooling: Pooling generally improves features from all CNN layers by reducing sensitivity to translation and occlusion while lowering feature dimensionality.Pooling aggregates local activations into a global representation, improving invariance and computational efficiency.
- Pooling: Average pooling outperforms max pooling on 8 out of 9 datasets for Conv5 features, except Bird-200.Average pooling performs well for scenes or large objects, whereas sparse activations from small birds favor max pooling.
- Feature fusion: Multi-layer feature fusion improves all three image-search datasets and all seven image-classification datasets.Search gains are +7.49% mAP on Holidays, +0.07 N-S on Ukbench, and +11.12% mAP on Oxford5k; classification gains range from +0.07% to +4.83%.
- State-of-the-art comparison: The system reports 76.4% and 95.6% accuracy on Bird-200 and Flower-102, exceeding prior methods by +6.7% and +6.1%.It also reports 92.3%, 86.0%, and 83.7% accuracy on Caltech-101, Caltech-256, and PASCAL VOC’07.
- State-of-the-art comparison: For image search, the method reports mAP = 84.2% and 71.3% on Holidays and Oxford5k and N-S = 3.75 on Ukbench.Graph Re-ranking further raises these results to 89.3%, 80.5%, and 3.90, respectively.
5. Conclusion
The paper concludes that larger inputs, pooling intermediate CNN features, and combining features across layers improve CNN feature transfer. It reports that pooled Conv5 can match or outperform fully connected features, while future work targets the discriminative power of bottom-level features.
- Larger images than 224 × 224 yield superior accuracy in CNN feature transfer.
- Pooling activation maps of intermediate CNN features consistently improves recognition over raw features.
- Pooled Conv5 features produce superior or competitive performance to fully connected features.
- Combining features across multiple CNN layers further promotes recognition accuracy and advances state-of-the-art performance.
- Future work focuses on improving the discriminative power of bottom-level CNN features through spatial constraints, multiple feature fusion, and effective encoding.