PaperlayerView exact revision on arXiv

Source-linked AI summary

Pyramidal Convolution: Rethinking Convolutional Neural Networks for Visual Recognition

Ionut Cosmin Duta, Li Liu, Fan Zhu, Ling Shao

arXiv:2006.11538v1cs.CVcs.LGeess.IV

TL;DR

The paper addresses the limited multi-scale processing of standard convolution in visual recognition. It introduces pyramidal convolution, which combines kernels of varying sizes and depths while maintaining similar cost, and reports improvements across visual-recognition tasks, including state-of-the-art semantic segmentation results.

  • Problem

    Standard convolution commonly uses small kernels, limiting direct multi-scale processing despite scenes containing objects and details at different scales.

  • Method

    PyConv processes inputs in parallel through pyramid levels with varying kernel sizes and depths, and the paper builds architectures for classification, recognition, detection, and segmentation.

  • Results

    PyConv architectures show significant improvements over baselines across visual-recognition tasks, including a semantic-segmentation system that outperforms PSPNet with a heavier backbone.

  • Takeaways & Limitations

    PyConv provides an efficient and flexible basis for visual-recognition architectures with multi-scale processing and similar standard-convolution cost.

  • Takeaways & Limitations

    Image-classification configurations may depend on input resolution, and stronger ImageNet results in other works use substantially different training settings and datasets.

Abstract

from arXiv · show

This work introduces pyramidal convolution (PyConv), which is capable of processing the input at multiple filter scales. PyConv contains a pyramid of kernels, where each level involves different types of filters with varying size and depth, which are able to capture different levels of details in the scene. On top of these improved recognition capabilities, PyConv is also efficient and, with our formulation, it does not increase the computational cost and parameters compared to standard convolution. Moreover, it is very flexible and extensible, providing a large space of potential network architectures for different applications. PyConv has the potential to impact nearly every computer vision task and, in this work, we present different architectures based on PyConv for four main tasks on visual recognition: image classification, video action classification/recognition, object detection and semantic image segmentation/parsing. Our approach shows significant improvements over all these core tasks in comparison with the baselines. For instance, on image recognition, our 50-layers network outperforms in terms of recognition performance on ImageNet dataset its counterpart baseline ResNet with 152 layers, while having 2.39 times less parameters, 2.52 times lower computational complexity and more than 3 times less layers. On image segmentation, our novel framework sets a new state-of-the-art on the challenging ADE20K benchmark for scene parsing. Code is available at: https://github.com/iduta/pyconv

1 Introduction

CNNs dominate visual recognition, but small kernels and downsampling limit effective context and may miss the wide range of object scales and scene details. The paper introduces PyConv to address these challenges with multi-scale processing.

  • CNNs are widely used across current computer vision applications and visual-recognition tasks.
  • 2.7 times smaller empirical receptive fields than theoretical ones expose a limitation of relying on stacked small kernels.
  • Scene images contain categories at substantially different scales, including large structures and small objects or object parts.
  • A single kernel type and spatial size may not adequately capture the diversity of object scales and scene complexity.
  • PyConv uses kernel levels with varying sizes and depths to enlarge receptive fields and capture different levels of detail without increasing standard-convolution cost.

2 Related Work

The paper situates PyConv among CNN architectures and context modules used for recognition and dense prediction. It uses ResNets as baselines and contrasts its segmentation head with pyramid-based alternatives.

  • ResNets use shortcut connections and serve as influential backbones for complex tasks such as object detection and instance segmentation.
  • Grouped convolution has been used both to distribute computation across GPUs and to improve recognition in ResNeXt architectures.
  • Squeeze-and-excitation and non-local blocks capture context information as additional modules inserted into CNNs.
  • PSPNet and another competitive architecture use pyramid pooling or atrous spatial pyramid pooling heads for semantic segmentation.

3 Pyramidal Convolution

PyConv combines parallel kernels with increasing spatial sizes and decreasing depths, using grouped convolution to control connectivity. This design targets multi-scale feature extraction while preserving efficiency and architectural flexibility.

  • PyConv contains multiple kernel levels with increasing spatial sizes and decreasing depths across the pyramid.
  • Grouped convolution splits input feature maps into groups so kernels at different PyConv levels can use different depths independently.
  • Each PyConv level produces output feature maps, whose counts sum to the total output-channel count.
  • The formulation keeps PyConv’s parameters and computational cost similar to standard convolution despite multiple levels and increasing kernel sizes.
  • Smaller kernels target fine details and smaller objects, while larger kernels provide information about larger objects and contextual regions.
  • PyConv levels can be computed independently in parallel, enabling flexible architectures and distribution across computing units.
  • Users can choose pyramid levels, kernel sizes, depths, and per-level output-map counts without increasing parameters or computational costs.

4 PyConv Networks for Image Classification

PyConv-based classification networks use parallel kernels with varying spatial sizes and depths to capture multi-scale details while preserving efficiency. Their architectures also support alternative grouping and multi-scale downsampling designs.

  • 4 PyConv Networks for Image Classification: The PyConv bottleneck applies 9×9, 7×7, 5×5, and 3×3 kernels in parallel, with varying grouping depths and 16 output feature maps per level.A 1×1 convolution first reduces the input to 64 feature maps before PyConv processes it.
  • 4 PyConv Networks for Image Classification: PyConvResNet reduces the number of pyramid levels across stages as feature-map spatial resolution decreases.The network uses four main stages organized by feature-map spatial size.
  • 4 PyConv Networks for Image Classification: PyConvHGResNet increases grouping to 32–64 groups and doubles spatial-convolution feature maps, producing a slight increase in FLOPs.The first stage is limited to 32 groups because each level has 32 input and output feature maps.
  • 4 PyConv Networks for Image Classification: Different kernel sizes perform downsampling in parallel, preserving access to varied spatial context while avoiding a parameter increase and significant computational changes.This design replaces the baseline’s initial max-pooling arrangement with convolutional downsampling across multiple scales.

5 PyConv Network on Semantic Segmentation

PyConvPH is a semantic-segmentation head that combines local and global multi-scale context. It processes fine details and large objects before merging the resulting representations for pixel classification.

  • 5 PyConv Network on Semantic Segmentation: PyConvPH combines local and global information at multiple kernel scales to parse backbone feature maps for scene segmentation.The head is designed to address both fine details and broader scene context.
  • 5 PyConv Network on Semantic Segmentation: LocalPyConv targets smaller objects and local fine details using kernels with different spatial sizes and depths.It reduces backbone features to 512 channels before applying PyConv.
  • 5 PyConv Network on Semantic Segmentation: GlobalPyConv adaptively pools features to 9×9, allowing its 9×9 kernel to cover the entire pooled input and capture global information.The output is fused and bilinearly upsampled to the pre-pooling spatial size.
  • 5 PyConv Network on Semantic Segmentation: The Merge Local-Global PyConv block concatenates local and global outputs, applies a fully connected 3×3 convolution, then upsamples for final classification.Its final 1×1 convolution produces outputs with dimensionality equal to the number of classes.

6 Experiments

Experiments evaluate PyConv across image classification and semantic segmentation, showing improved recognition and segmentation performance under comparable or more efficient configurations.

  • Image classification: 1000 classes, 1.28 million training images, and 50K validation images define the ImageNet image-classification evaluation.Models use SGD with momentum 0.9, weight decay 0.0001, and a 90-epoch schedule across 8 V100 GPUs.
  • Image classification: 0.76 percentage points lower top-1 error results when two PyConv levels replace the baseline configuration, improving from 23.88 to 23.12.The two-level design uses 5×5 and 3×3 kernels while requiring fewer parameters and FLOPs than the baseline.
  • Image classification: 22.12 top-1 error is achieved by the final 50-layer PyConvResNet with 24.85 million parameters and 3.88 GFLOPs.Moving max pooling to the shortcut and retaining projection-shortcut pooling further improves the result.
  • Image classification: Less than 10 epochs are needed for 50-layer PyConvResNet to outperform the best ResNet results across the first 30 epochs.Across 50, 101, and 152 layers, PyConv networks outperform baseline ResNet; the 50-layer PyConvHGResNet also beats 152-layer ResNet on top-1 error.
  • Training settings: 300 training epochs, CutMix, cosine scheduling, batch size 1024, and mixed precision provide an alternative more complex training setup for PyConvResNet.The paper notes that stronger results reported by other works use substantially larger datasets, computational resources, or more complex training settings.
  • Semantic segmentation: 152-layer PyConvSegNet outperforms PSPNet with a 269-layer backbone on ADE20K while requiring fewer resources for the PPM head.Tables report single-scale and multi-scale inference across different ResNet and PyConvResNet backbone depths, including validation and testing comparisons.

7 Conclusion

The paper concludes that PyConv combines multi-scale kernels with efficiency and flexibility, producing improvements across visual recognition tasks and state-of-the-art image segmentation results.

  • Conclusion: PyConv uses several kernel levels with varying scales and supports a broad pool of potential network architectures.The conclusion identifies visual recognition improvements alongside efficiency and flexibility.
  • Conclusion: PyConv-based architectures show significant improvements across different visual recognition tasks, while the proposed segmentation framework achieves state-of-the-art results.The paper also identifies image restoration, inpainting, artifact removal, enhancement, and super-resolution as potential application directions.

A Appendix

The appendix provides supplementary architectural details, video-classification pipeline information, and visual examples for image segmentation.

  • Appendix: Three appendix sections cover object-detection architecture details, the video-classification pipeline, and visual examples on image segmentation.The object-detection material is presented in Section A.1, video classification in A.2, and segmentation examples in A.3.

A.1 PyConv on object detection

For object detection, PyConv is integrated into SSD as PyConvSSD, combining a PyConvResNet backbone with multi-scale detection features and reporting better, more efficient results than the ResNet-based baseline.

  • Framework: PyConvSSD integrates PyConv into the efficient single-stage SSD detector, which performs detection over feature maps at multiple resolutions.Parallel kernel levels address objects appearing at different image scales.
  • PyConvResNet backbone: 1024 output feature maps at 38×38 resolution are produced by the truncated PyConvResNet stage 3 for 300×300 input images.Layers after stage 3 are removed and stage-3 strides are set to 1 to maintain framework efficiency.
  • PyConvSSD head: 9×9, 7×7, 5×5, and 3×3 kernels operate in parallel in the four-level PyConvSSD head, including stride-2 downsampling.The number of PyConv levels decreases as feature-map resolution decreases.
  • Detection outputs: 4, 6, 6, 6, 4, and 4 default boxes are assigned per spatial location across the six detection feature maps.Each box predicts four localization offsets and class confidences through separate 3×3 convolutions.
  • Baseline: The baseline replaces SSD’s VGG backbone with ResNet and uses standard 3×3 convolutions in the head for a direct comparison with PyConvSSD.This isolates the proposed PyConv-based design from the original SSD backbone choice.
  • Results: PyConvSSD improves all reported metrics over the baseline while using fewer parameters and FLOPs, and its 50-layer backbone is competitive with the baseline’s 101-layer backbone.The comparison uses 300×300 input images and COCO val2017 results.

A.2 PyConv on video classification

PyConvResNet3D applies pyramidal convolution across spatial and temporal dimensions for video classification, improving performance over a baseline while using fewer resources. On Kinetics-400, the 50-layer model also shows improved training convergence.

  • PyConv enlarges the receptive field and processes multiple kernel scales in parallel across spatial and temporal dimensions for video tasks.
  • Kinetics-400 contains approximately 246k training videos, 20k validation videos, and 400 action classes.
  • 37.01% to 34.56% top-1 error: PyConvResNet3D improves over the 50-layer baseline on video recognition.The PyConvResNet3D model also requires fewer parameters and FLOPs than the baseline.
  • PyConvResNet3D improves training convergence according to the training and validation curves.The curves are computed during training over independent clips on Kinetics-400.

A.3 Qualitative examples on image segmentation

PyConSegNet is qualitatively compared with PSPNet and DeepLabv3 on ADE20K validation images, with PyConvPH providing the proposed parsing head. The examples show competitive detail and efficiency, alongside a difficult door–window confusion case.

  • PyConSegNet with the PyConvPH head is visually compared with PSPNet using PPM and DeepLabv3 using ASPP on ADE20K validation images.
  • PyConvPH competes at a high level with PSPNet and DeepLabv3 while using fewer parameters and lower computational complexity.PSPNet requires over 40% more parameters and FLOPs, while DeepLabv3 requires over 20% more parameters and close to 30% more FLOPs than PyConSegNet.
  • The visual examples highlight differences in segmentation details between PyConSegNet and the state-of-the-art comparison methods.
  • PyConSegNet confuses a door with a window in one difficult example.The passage describes this case as confusing even for a human observer.
  • Segmentation quality for a ceiling-mounted fan improves as the PyConvResNet backbone increases from 50 to 101 to 152 layers.The comparison uses PyConSegNet with 50-, 101-, and 152-layer PyConvResNet backbones.
Loading 2006.11538v1…