Source-linked AI summary
The Treasure beneath Convolutional Layers: Cross-convolutional-layer Pooling for Image Classification
Lingqiao Liu, Chunhua Shen, Anton van den Hengel
TL;DR
The paper addresses the limited use of convolutional activations for image representation and the cost and domain mismatch of regional DCNN features. It introduces cross-convolutional-layer pooling, which uses convolutional subarrays and successive-layer feature maps, and reports comparable or better classification with lower computational cost across four tasks.
Problem
Existing pretrained-DCNN image and regional representations primarily use fully connected activations, while convolutional activations are considered less discriminative and regional extraction incurs computational and domain-mismatch issues.
Method
Cross-convolutional-layer pooling extracts subarrays from one convolutional layer as local features and pools them using feature maps from the successive layer as guidance.
Results
Across four visual classification tasks, the proposed method achieves comparable or sometimes significantly better performance than competitive methods while creating image representations considerably faster.
Takeaways & Limitations
Appropriately used convolutional layers provide useful image information with advantages over fully connected representations, including lower computational cost.
Takeaways & Limitations
The basic cross-layer pooling method operates at a single spatial resolution, which can limit capture of small objects or fine details.
Abstract
from arXiv · showhide
A number of recent studies have shown that a Deep Convolutional Neural Network (DCNN) pretrained on a large dataset can be adopted as a universal image description which leads to astounding performance in many visual classification tasks. Most of these studies, if not all, adopt activations of the fully-connected layer of a DCNN as the image or region representation and it is believed that convolutional layer activations are less discriminative. This paper, however, advocates that if used appropriately convolutional layer activations can be turned into a powerful image representation which enjoys many advantages over fully-connected layer activations. This is achieved by adopting a new technique proposed in this paper called cross-convolutional-layer pooling. More specifically, it extracts subarrays of feature maps of one convolutional layer as local features and pools the extracted features with the guidance of feature maps of the successive convolutional layer. Compared with exising methods that apply DCNNs in the local feature setting, the proposed method is significantly faster since it requires much fewer times of DCNN forward computation. Moreover, it avoids the domain mismatch issue which is usually encountered when applying fully connected layer activations to describe local regions. By applying our method to four popular visual classification tasks, it is demonstrated that the proposed method can achieve comparable or in some cases significantly better performance than existing fully-connected layer based image representations while incurring much lower computational cost.
1. Introduction
The paper proposes cross-convolutional-layer pooling to make convolutional activations useful as local image features, combining subarrays from one layer with guidance from the next. It targets stronger representations while reducing the computation and domain mismatch associated with regional DCNN methods.
- Cross-convolutional-layer pooling extracts subarrays of convolutional activations as regional descriptors and pools them using cross-layer information.
- The method uses a single or few DCNN forward computations for all local regions instead of running the network separately for each region.
- Feature maps from deeper convolutional layers serve as indicator maps that weight regional descriptors from the preceding layer.
- Experiments cover scene, fine-grained object, generic object, and attribute classification across four datasets.
2. Current strategies for creating image representations from a pretrained DCNN
Pretrained DCNN representations are commonly built from fully connected activations, either globally or from multiple image regions. Regional approaches can improve performance but require repeated computation and introduce a mismatch between local inputs and whole-image training data.
- Pretrained DCNNs are used either as global-feature extractors or as regional-feature extractors for image representation.
- Global methods commonly use fc-6 or fc-7 activations from the whole image, sometimes averaging activations from jittered image versions.
- Regional-DCNN methods describe each region separately and aggregate the resulting activations, often with unsupervised encoding or multiple-scale extraction.
- Most established approaches use fully connected activations, while preliminary studies have suggested weaker discriminative power for convolutional activations.
3. Proposed method
The proposed cross-convolutional-layer pooling method turns convolutional activations into local descriptors and pools them using feature maps from the successive layer. It avoids domain mismatch and reduces computation while supporting multi-resolution representations.
- Convolutional layer vs. fully-connected layer: Convolutional activations preserve spatial information as D-dimensional feature vectors arranged across H × W spatial units.Fully connected activations represent the whole image and lose spatial information, preventing explicit recovery of individual spatial-unit features.
- Local feature extraction: The method extracts subarrays of convolutional activations from a single whole-image DCNN forward pass as local features.These subarrays correspond to local image regions and avoid repeatedly applying the network to separate regions.
- Cross-convolutional-layer pooling: Using feature vectors from multiple spatial units significantly improves classification performance over relying on one spatial unit.A single spatial unit may not describe the visual pattern within a local region sufficiently.
- Cross-convolutional-layer pooling: Cross-convolutional-layer pooling uses successive-layer feature maps as indicator maps that weight regional descriptors into multiple pooling channels.Deep feature maps are used because their sparse activations can indicate semantically meaningful regions without predefined part annotations.
- Implementation details: Power normalization and coarse ternary quantization are implementation options, with the latter causing only a slight performance drop while reducing storage to 2 bits per feature dimension.PCA is applied to pooled features before representation construction.
- Creating finer resolutions of spatial units partitioning: Multi-resolution pooling addresses the limitation that the basic method operates at only one spatial resolution.The approach applies the DCNN to image blocks to obtain finer spatial partitioning, though this introduces additional forward computations.
4. Experiments
The evaluation covers four datasets spanning scene, fine-grained object, generic object, and attribute classification tasks.
- Datasets: The proposed method is evaluated on MIT-67, Birds-200, Pascal-07, and H3D datasets.Together, these datasets cover scene classification, fine-grained object classification, generic object classification, and attribute classification.
4.1. Experimental protocol
The experiments compare the proposed representations with fully connected baselines and use multi-resolution variants across convolutional-layer combinations.
- Baselines: The baselines include global fully connected activations, transformed-image averaging, and fully connected activations extracted from multiple regions.These establish comparisons with common global and regional DCNN representation strategies.
- Multi-resolution protocol: The proposed representation concatenates whole-image and block-based outputs, using 2 × 2 blocks except for H3D, which uses 2 × 1 blocks.The H3D setting reflects that its images are generally taller than wide.
- Layer and variant settings: The main experiments use the 4th and 5th convolutional layers, while additional experiments use the 3rd and 4th layers.CL-45, CL-45F, and CL-45C denote whole-image, finer-resolution, and combined two-resolution variants, respectively.
4.2. Performance evaluation
Across four visual classification tasks, cross-convolutional-layer pooling generally matches or outperforms competing DCNN representations, with especially strong results on MIT-67, Birds-200, and human attributes. The method also offers substantial computational savings, including about a 10× speedup over SCFV.
- Scene classification: MIT-67: 71.5% classification performance is achieved on MIT-67 by combining CL-C with CNN-Jitter.All method variations outperform CNN-Global and CNN-Jitter, while CL-C matches regional-CNN methods with fewer CNN forward computations.
- Fine-grained image classification: Birds-200: 72.4% accuracy from CL-45 improves to 73.5% with CL-45F on Birds-200.CL-45 exceeds R-CNN SCFV by 6% without using parts annotations, and the combined result approaches methods relying on strong parts annotation.
- Object classification: Pascal-2007: The best Pascal-2007 method achieves performance comparable to the state of the art, but gains over competing methods are smaller than on MIT-67 and Birds-200.CL-45C alone outperforms CNN-Global and CNN-Jitter but does not outperform R-CNN; the paper attributes the limited improvement partly to ImageNet category overlap.
- Attribute classification: H3D: 77.3% performance on human-attribute classification exceeds R-CNN SCFV by 4%.Combining the method with CNN-Jitter makes it comparable to PANDA, which requires poselet annotations and detections.
- Computational cost: Around 10× speedup is obtained over SCFV when extracting image representations.The method’s computational cost is comparable to CNN-Global and CNN-Jitter, based on a naive MATLAB implementation that could be further accelerated.
4.3. Analysis of components of our method
Component analyses show that deeper convolutional layers and cross-layer pooling are central to performance, while feature-sign quantization substantially reduces memory with little accuracy loss.
- Using different convolutional layers: Using the 4th–5th convolutional layers outperforms using the 3rd–4th layers, consistent with deeper layers having greater discriminative power.The comparison is reported in Table 6.
- Comparison of different pooling schemes: Cross-layer pooling significantly outperforms max-pooling, sum-pooling, and spatial-pyramid pooling across the evaluated single-resolution settings.The alternatives include direct sum-pooling with square operation, max-pooling, spatial pyramid pooling, and SCFV encoding.
- Comparison of different pooling schemes: SCFV improves classification accuracy over direct pooling in many cases but remains inferior to the proposed method on MIT-67, Pascal-07, and Birds-200.H3D is the exception, where SCFV performs slightly better but requires additional codebook learning and encoding computation.
- Feature sign quantization: Feature-sign quantization uses 2 bits per pooled-feature dimension and achieves nearly the original performance on MIT-67, Pascal-07, and H3D.The scheme assigns 1 to positive values, -1 to negative values, and 0 to zero values, greatly reducing memory usage.
5. Conclusion
The paper proposes cross-convolutional-layer pooling to form image representations from pretrained CNN convolutional activations. Experiments show good classification performance at low computational cost, suggesting convolutional activations can be highly useful when appropriately processed.
- The paper proposes cross-convolutional-layer pooling to create image representations from convolutional activations of a pretrained CNN.
- Extensive experiments show that the method provides good classification performance and low computational cost.
- Appropriately used convolutional layers contain useful information and offer advantages over image representations based on fully connected activations.