Source-linked AI summary
No More Strided Convolutions or Pooling: A New CNN Building Block for Low-Resolution Images and Small Objects
Raja Sunkara, Tie Luo
TL;DR
CNN performance degrades on low-resolution images and small objects, which the paper attributes to information loss from strided convolution and pooling. It introduces SPD-Conv, a space-to-depth layer followed by non-strided convolution, and reports improved performance across detection and classification, especially for small objects and low-resolution inputs.
Problem
CNNs lose fine-grained feature information on low-resolution images and small objects when using strided convolution and pooling.
Method
SPD-Conv replaces strided convolution and pooling with a space-to-depth layer followed by a non-strided convolution.
Results
The SPD variants significantly improve AP and top-1 accuracy, especially on small objects and low-resolution images.
Takeaways & Limitations
Replacing strided convolution and pooling can improve accuracy while maintaining the same level of parameter size.
Takeaways & Limitations
YOLOv5-SPD used YOLOv5 hyperparameters without retuning, while non-YOLO baselines and PP-YOLO used transfer learning from high-quality images.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) have made resounding success in many computer vision tasks such as image classification and object detection. However, their performance degrades rapidly on tougher tasks where images are of low resolution or objects are small. In this paper, we point out that this roots in a defective yet common design in existing CNN architectures, namely the use of strided convolution and/or pooling layers, which results in a loss of fine-grained information and learning of less effective feature representations. To this end, we propose a new CNN building block called SPD-Conv in place of each strided convolution layer and each pooling layer (thus eliminates them altogether). SPD-Conv is comprised of a space-to-depth (SPD) layer followed by a non-strided convolution (Conv) layer, and can be applied in most if not all CNN architectures. We explain this new design under two most representative computer vision tasks: object detection and image classification. We then create new CNN architectures by applying SPD-Conv to YOLOv5 and ResNet, and empirically show that our approach significantly outperforms state-of-the-art deep learning models, especially on tougher tasks with low-resolution images and small objects. We have open-sourced our code at https://github.com/LabSAINT/SPD-Conv.
1 Introduction
The paper argues that strided convolution and pooling discard fine-grained information needed for low-resolution images and small objects. It proposes SPD-Conv to replace both operations and reports improvements in detection and classification.
- Low-resolution images and small objects challenge CNNs because they provide limited resolution and context, while large objects can dominate feature learning.
- Strided convolution and pooling are identified as common CNN design choices that lose fine-grained information in tougher visual tasks.
- SPD-Conv replaces strided convolution and pooling with a space-to-depth layer followed by a non-strided convolution.
- The approach is evaluated by constructing YOLOv5-SPD, ResNet18-SPD, and ResNet50-SPD on object detection and image classification benchmarks.
- The reported improvements are strongest for small objects and low-resolution images, with gains in AP and top-1 accuracy.
2 Preliminaries and Related Work
The paper reviews CNN-based object detection pipelines and prior approaches for multi-scale and low-resolution recognition. It motivates its focus on one-stage, anchor-based detection and contrasts SPD-Conv with methods requiring high-resolution supervision or specialized labels.
- Object Detection: Object detectors are categorized by stages and anchor usage, with one-stage models detecting objects directly over dense locations.
- Object Detection: The paper focuses on one-stage, anchor-based models because it characterizes them as faster and more accurate, respectively, than their alternatives.
- Object Detection: A typical one-stage detector combines a CNN backbone, a multi-scale feature neck, and a detection head for classes and bounding boxes.
- Low-Resolution Classification: Prior low-resolution methods use super-resolution, high-resolution training images, or attribute-level labels, each imposing additional requirements.
3 A New Building Block: SPD-Conv
SPD-Conv downsamples feature maps by rearranging spatial information into channels, then applies a non-strided convolution. This preserves the downsampled information while reducing the expanded channel count with learnable filters.
- SPD-Conv consists of a space-to-depth layer followed by a non-strided convolution layer.
- The SPD layer extracts interleaved sub-feature maps from an intermediate feature map using a chosen scale factor.
- For scale = 2, SPD creates four sub-maps, each spatially downsampled by two, before concatenating them along the channel dimension.
- The transformation changes X(S, S, C1) into X′(S/scale, S/scale, scale^2C1), preserving the rearranged information in more channels.
- A stride-1 convolution maps the expanded representation to C2 channels, with C2 < scale^2C1, while retaining discriminative feature information.
4 How to Use SPD-Conv: Case Studies
The paper applies SPD-Conv to object detection and classification architectures by replacing strided downsampling operations. YOLOv5-SPD retains YOLOv5’s scalable design, while SPD variants of ResNet remove or replace early downsampling for small images.
- SPD-Conv is presented as a general replacement for strided convolution and pooling across CNN architectures.
- Object Detection: The YOLOv5 case study uses a CSPDarknet53 backbone, PANet neck, and YOLOv3 detection head as its architectural basis.
- Object Detection: YOLOv5-SPD replaces seven stride-2 convolutions in YOLOv5, including five backbone layers and two neck layers.
- Object Detection: YOLOv5-SPD preserves YOLOv5 scalability by adjusting convolution filters or repeated C3 modules to create nano, small, medium, and large versions.
- Image Classification: ResNet18-SPD and ResNet50-SPD replace four stride-2 convolutions, while max pooling is removed for the 64 × 64 and 32 × 32 datasets.
5 Experiments
The experiments evaluate SPD-Conv on object detection and image classification, using YOLOv5-SPD and ResNet-SPD models on low-resolution or small-object settings. SPD-Conv generally improves performance, with especially consistent gains for small-object detection.
- 5.1 Object Detection: The evaluation covers object detection on COCO val2017 and test-dev2017, reporting AP across IoU thresholds and object sizes.The experiments train YOLOv5-SPD and baselines on COCO train2017, including small-, medium-, and large-object AP metrics.
- 5.1 Object Detection: 13.15% higher APS and 10.7% higher overall AP make YOLOv5-SPD-n the best nano-model performer on val2017 versus YOLOv5n.These are the reported margins over the runner-up in the nano category.
- 5.1 Object Detection: 8.6% higher APS gives YOLOv5-SPD-m a larger small-object margin than the runner-up, while its overall AP remains competitive among medium models.The paper identifies this pattern as evidence that SPD-Conv is especially advantageous for smaller objects and lower resolutions.
- 5.1 Object Detection: YOLOv5-SPD models remain strongest on APS across large-model and test-dev comparisons, while YOLOv5-SPD-l closely matches YOLOX-L on AP.On test-dev2017, YOLOv5-SPD-n leads APS by 19% over YOLOv5n, and YOLOv5-SPD-l is best on APS while closely matching YOLOX-L on AP.
- 5 Experiments: The summary reports significant accuracy gains at the same parameter size, with improvements most prominent for small objects and consistent APS leadership.The visual examples also show SPD detecting an occluded giraffe and very small objects missed by YOLOv5m.
- 5.2 Image Classification: ResNet18-SPD and ResNet50-SPD clearly outperform the baseline models on top-1 accuracy in image classification experiments.The classification experiments use Tiny ImageNet and CIFAR-10, with low-resolution images of 64×64×3 and 32×32×3 pixels.
6 Conclusion
The paper identifies strided convolution and pooling as problematic for low-resolution images and small objects, then proposes SPD-Conv as a replacement. Across object detection and image classification, the approach improves accuracy while retaining discriminative feature information.
- Strided convolution and pooling can lose fine-grained feature information, especially for low-resolution images and small objects.
- SPD-Conv replaces these operations with a space-to-depth layer followed by a non-strided convolution.
- The design downsamples feature maps while retaining discriminative feature information.
- Evaluations on object detection and image classification show significant improvements in detection and classification accuracy.