PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

arXiv:1406.4729v4cs.CV

TL;DR

Fixed-size CNN inputs can lose content or distort geometry when images vary in scale, aspect ratio, or size. SPP-net adds spatial pyramid pooling to produce fixed-length representations, achieving strong classification and detection results while ranking #2 in detection and #3 in classification at ILSVRC 2014.

  • Problem

    Fixed-size CNN inputs can lose object content or introduce distortion when image scale, aspect ratio, or size varies.

  • Method

    SPP-net adds a spatial pyramid pooling layer after convolutional features to generate fixed-length outputs without initial cropping or warping.

  • Results

    SPP-net achieves state-of-the-art classification on Caltech101 and Pascal VOC 2007 and ranks #2 in detection and #3 in classification among 38 ILSVRC 2014 teams.

  • Takeaways & Limitations

    SPP provides a flexible solution for varying scales, sizes, and aspect ratios while greatly accelerating deep-network-based detection.

  • Takeaways & Limitations

    Although theoretically trainable for arbitrary image sizes, practical GPU implementations are preferably run on fixed-size inputs.

Abstract

from arXiv · show

Existing deep convolutional neural networks (CNNs) require a fixed-size (e.g., 224x224) input image. This requirement is "artificial" and may reduce the recognition accuracy for the images or sub-images of an arbitrary size/scale. In this work, we equip the networks with another pooling strategy, "spatial pyramid pooling", to eliminate the above requirement. The new network structure, called SPP-net, can generate a fixed-length representation regardless of image size/scale. Pyramid pooling is also robust to object deformations. With these advantages, SPP-net should in general improve all CNN-based image classification methods. On the ImageNet 2012 dataset, we demonstrate that SPP-net boosts the accuracy of a variety of CNN architectures despite their different designs. On the Pascal VOC 2007 and Caltech101 datasets, SPP-net achieves state-of-the-art classification results using a single full-image representation and no fine-tuning. The power of SPP-net is also significant in object detection. Using SPP-net, we compute the feature maps from the entire image only once, and then pool features in arbitrary regions (sub-images) to generate fixed-length representations for training the detectors. This method avoids repeatedly computing the convolutional features. In processing test images, our method is 24-102x faster than the R-CNN method, while achieving better or comparable accuracy on Pascal VOC 2007. In ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2014, our methods rank #2 in object detection and #3 in image classification among all 38 teams. This manuscript also introduces the improvement made for this competition.

1 INTRODUCTION

SPP-net removes CNNs’ artificial fixed-input-size constraint by inserting spatial pyramid pooling between convolutional and fully connected layers, producing fixed-length representations for arbitrary-sized inputs. The approach improves classification across CNN designs and supports faster object detection by reusing convolutional feature maps.

  • Motivation: Fixed-size CNN inputs limit image aspect ratio and scale, while cropping can omit objects and warping can distort content.Fixing input sizes also overlooks cases where object scales vary.
  • Motivation: Convolutional layers accept arbitrary image sizes, but fully connected layers require fixed-size inputs, creating the fixed-input constraint.The paper identifies this mismatch as the architectural source of the requirement.
  • Method: SPP-net adds an SPP layer after the last convolutional layer to pool features into fixed-length outputs before fully connected layers or other classifiers.This deeper-stage aggregation avoids cropping or warping at the image-input stage.
  • Method: SPP-net enables representations from arbitrarily sized images or windows and variable-size training, which increases scale-invariance and reduces over-fitting.The authors develop a multi-size training method using multiple parameter-sharing networks trained at fixed sizes.
  • Results: SPP improves four different CNN architectures on ImageNet 2012 and achieves state-of-the-art classification on Caltech101 and Pascal VOC 2007 without fine-tuning.It uses only a single full-image representation for the latter datasets.
  • Object detection: SPP-net reuses convolutional feature maps for candidate regions instead of repeatedly processing thousands of warped regions, strengthening object detection efficiency.In ILSVRC 2014, the methods ranked #2 in object detection and #3 in image classification among all 38 teams.

2 DEEP NETWORKS WITH SPATIAL PYRAMID POOLING

SPP-net removes the fixed-size input constraint caused by fully connected layers by replacing the final pooling layer with spatial pyramid pooling. The resulting fixed-length representation supports arbitrary image sizes, aspect ratios, and scales, while practical training uses fixed-size implementations.

  • Motivation: Fixed-size inputs are required because fully connected layers and classifiers need fixed-length vectors, whereas convolutional layers accept arbitrary image sizes but produce variable-size outputs.Convolutional outputs preserve roughly the input aspect ratio and form feature maps.
  • Spatial pyramid pooling: Spatial pyramid pooling preserves spatial information through local bins whose sizes scale with the image, while keeping the number of bins fixed.The coarsest pyramid level is a single global bin covering the entire image.
  • Spatial pyramid pooling: SPP replaces the last pooling layer and pools each filter’s responses within fixed spatial bins, producing a kM-dimensional vector independent of image size.M is the number of spatial bins and k is the number of filters in the last convolutional layer; the paper uses max pooling.
  • Arbitrary inputs: With SPP, images can have arbitrary sizes, aspect ratios, and scales, allowing the same network to extract features at different scales.The input may be resized to scales such as min(w, h)=180 or 224 while retaining the same filter sizes.
  • Training: Standard back-propagation theoretically supports arbitrary image sizes, but training uses fixed-size GPU implementations; single-size training enables multilevel pooling, while multi-size training simulates varying inputs.The multi-size procedure alternates 224×224 and 180×180 networks while preserving weights, and experiments find convergence similar to single-size training.

3 SPP-NET FOR IMAGE CLASSIFICATION · 3.1 Experiments on ImageNet 2012 Classification

On ImageNet 2012, SPP improves multiple CNN architectures through multi-level pooling, supports training across input sizes, and enables flexible full-image and feature-map multi-view testing. The resulting single network achieves 9.14% top-5 validation error, while an eleven-model combination ranks third among 38 ILSVRC 2014 teams.

  • 3.1.1 Baseline Network Architectures: SPP improves the accuracy of all four investigated CNN architectures, showing that its advantages are independent of network architecture.The baselines include ZF-5, Convnet*-5, and Overfeat-5/7.
  • 3.1.1 Baseline Network Architectures: The baseline networks use 6×6 feature maps after the final convolutional pooling layer, followed by two 4096-d fully connected layers and a 1000-way softmax.The authors trained 70 epochs for ZF-5 and 90 epochs for the other replicated baselines.
  • 3.1.2 Multi-level Pooling Improves Accuracy: A 4-level SPP pyramid with {6×6, 3×3, 2×2, 1×1} bins, totaling 50 bins, replaces final pooling while preserving standard 10-view prediction.The authors attribute multi-level pooling gains to robustness against object deformation and spatial-layout variance, not simply to additional parameters.
  • 3.1.3 Multi-size Training Improves Accuracy: Multi-size training with sizes 224 and 180 reduces SPP-net (Overfeat-7) top-1 error to 29.68%, 2.33% below no SPP and 0.68% below single-size training.Testing remains at size 224 with standard 10-view prediction.
  • 3.1.3 Multi-size Training Improves Accuracy: With a uniformly sampled training size from [180, 224], SPP-net (Overfeat-7) obtains 30.06%/10.96% top-1/5 error and still outperforms single-size training.The top-1 error is slightly worse than the two-size version, possibly because testing size 224 is sampled less often.
  • 3.1.3 Multi-size Training Improves Accuracy: The method is presented as the first single network trained with input images of multiple sizes, whereas earlier solutions mainly handled scale variation during testing.Overfeat and Howard’s method apply one or more networks at multiple testing scales and average scores.
  • 3.1.4 Full-image Representations Improve Accuracy: Full-image SPP representations retain useful accuracy, and adding two flipped full-image views improves predictions by about 0.2% even alongside dozens of other views.The comparisons use images resized with min(w, h)=256 and compare full-image views with central 224×224 crops.

3.2 Experiments on VOC 2007 Classification

On Pascal VOC 2007, SPP-net improves classification over a no-SPP baseline, with full-image and scale-adjusted representations reaching 82.44% mAP. This result is comparable to state-of-the-art methods while using a single full-image representation and no fine-tuning.

  • Dataset and evaluation: Pascal VOC 2007 contains 9,963 images across 20 categories, with 5,011 for training and the remainder for testing; performance is measured by mAP.The classification results are summarized in Table 6.
  • SPP-net comparisons: Replacing the no-SPP ZF-5 baseline with SPP-net improves the results of the fully connected layers, even on the center 224×224 crop.The baseline resizes images so the smaller dimension is 224 and uses the center 224×224 region.
  • Full-image representation: 78.39% vs. 76.45%: full-image representations considerably improve classification over center-crop processing by maintaining complete image content.Images are resized so their shorter side is 224.
  • Scale selection: s = 392 gives the best results on the validation set, showing that relative object scale matters in VOC 2007 classification.SPP-net permits resizing images to different smaller dimensions while using the same network.
  • State-of-the-art comparison: 82.44% mAP: the best Overfeat-7, multi-size-trained architecture raises the classification result to this level.The result is comparable to Oquab et al.’s 77.7% and Chatfield et al.’s 82.42%, which use fine-tuning and multi-view testing; SPP-net uses one full-image representation without fine-tuning.

3.3 Experiments on Caltech101

On Caltech101, SPP-net improves over the no-SPP network and benefits from full-image representations. It achieves 93.42% accuracy, exceeding the previous record by 4.88%.

  • 3.3 Experiments on Caltech101: The evaluation uses 30 training images per category, up to 50 testing images per category, and averages accuracy over 10 random splits.Caltech101 contains 9,144 images across 102 categories, including one background category.
  • 3.3 Experiments on Caltech101: SPP-net outperforms the no-SPP network, while full-view representations outperform cropped representations.These patterns are shared with the Pascal VOC 2007 results.
  • 3.3 Experiments on Caltech101: On Caltech101, fully connected layers are less accurate and SPP layers are better than on the corresponding Pascal VOC results.The paper suggests this may reflect weaker alignment between Caltech101 categories and ImageNet categories.
  • 3.3 Experiments on Caltech101: 91.44% accuracy with the undistorted full image exceeds 89.91% accuracy after warping to 224×224 using SPP-layer features.Warping preserves complete content but introduces distortion.
  • 3.3 Experiments on Caltech101: 93.42% accuracy exceeds the previous Caltech101 record of 88.54% by 4.88%.The comparison is against the previous state-of-the-art result.

4 SPP-NET FOR OBJECT DETECTION

SPP-net computes convolutional feature maps once from the entire image and pools arbitrary candidate windows into fixed-length representations, avoiding R-CNN’s repeated convolution. On Pascal VOC 2007, it achieves comparable detection accuracy while substantially improving speed, including practical multi-proposal inference and strong ILSVRC 2014 results.

  • Method: SPP-net computes entire-image feature maps once, then applies spatial pyramid pooling within arbitrary candidate windows to produce fixed-length representations.This avoids repeatedly computing convolutional features for each detection window.
  • Method: 12,800d representations result from a 4-level pyramid with 50 bins applied to each candidate window.The representation uses 256 channels across the 50 pooled bins.
  • Pascal VOC 2007: 44.9% mAP is achieved with pooled pool5 features, compared with 44.2% for R-CNN on Pascal VOC 2007.The non-fine-tuned fc6 results are inferior, while multi-scale extraction is also evaluated.
  • Efficiency: 1/160 of R-CNN’s convolutional complexity is required by the single-scale method, versus 1/24 for the 5-scale version.The comparison uses s=688 for the single-scale configuration and approximately 2,000 windows for R-CNN.
  • Efficiency: 24× to 64× faster than R-CNN is the reported running-time advantage when R-CNN uses AlexNet.The comparable accuracy is attributed partly to the ZF-5 architecture and SPPnet’s multi-level pooling.
  • Practical detection: 56.3 mAP without bounding box regression and ∼0.5s per image are achieved using SS and EdgeBox proposals for training and EdgeBoxes for testing.The result exceeds 55.2 mAP and is presented as practical for real-world applications.

5 CONCLUSION

The paper presents spatial pyramid pooling as a flexible way to handle varying image scales, sizes, and aspect ratios in deep networks. SPP-net achieves outstanding classification and detection accuracy while greatly accelerating deep-network-based detection.

  • 5 CONCLUSION: SPP addresses varying scales, sizes, and aspect ratios, issues that received little consideration in deep networks.The authors frame these variations as important visual-recognition challenges.
  • 5 CONCLUSION: The authors propose training a deep network with a spatial pyramid pooling layer.This layer produces the resulting SPP-net.
  • 5 CONCLUSION: SPP-net delivers outstanding accuracy on classification and detection tasks while greatly accelerating deep-network-based detection.The conclusion summarizes both recognition performance and detection speed as the method’s main outcomes.

APPENDIX A

Appendix A specifies preprocessing and feature-map alignment details for variable-size inputs and detection windows. It also defines how spatial pyramid pooling bins handle boundary rounding and how image-domain windows map onto subsampled feature maps.

  • Preprocessing: Variable-size ImageNet inputs use a warped 224×224 mean image, while Pascal VOC 2007 and Caltech101 experiments subtract the constant mean 128.The fixed-size mean image is warped to the desired input size for ImageNet.
  • Pooling Bins: For an n×n pyramid level, pooling bins are defined over conv5 feature-map dimensions w and h, with floor rounding on left/top boundaries and ceiling rounding on right/bottom boundaries.The implementation applies these boundary rules whenever rounding is needed.
  • Mapping a Window to Feature Maps: Detection windows specified in the image domain are aligned to cropped convolutional feature maps, including conv5, which have been subsampled several times.This alignment is also used for multi-view testing on feature maps.
  • Mapping a Window to Feature Maps: Window corners are projected to feature-map pixels whose receptive-field centers are closest to the corresponding image-domain corners.Padding complicates the mapping, so deployment pads floor(p/2) pixels for a layer with filter size p.
  • Mapping a Window to Feature Maps: The effective receptive-field center follows (x, y) = (Sx′, Sy′), with S = 16 for ZF-5 on conv5 and S = 12 for Overfeat-5/7 on conv5/7.S is the product of all previous strides.

CHANGELOG

The manuscript progressed from an initial ECCV 2014 technical report to TPAMI versions adding experiments, ILSVRC 2014 details, R-CNN and EdgeBoxes comparisons, and implementation clarification.

  • arXiv v1 was the initial technical report for the ECCV 2014 paper.
  • arXiv v2 was the TPAMI submitted version, adding experiments of SPP on various architectures and details for ILSVRC 2014.
  • arXiv v3 was the TPAMI accepted version, adding same-architecture comparisons with R-CNN and detection experiments using EdgeBoxes.
  • arXiv v4 revised “Mapping a Window to Feature Maps” in the Appendix for easier implementation.
Loading 1406.4729v4…