PaperlayerView exact revision on arXiv

Source-linked AI summary

Selective Kernel Networks

Xiang Li, Wenhai Wang, Xiaolin Hu, Jian Yang

arXiv:1903.06586v2cs.CV

TL;DR

CNNs typically use fixed receptive-field sizes, despite evidence that visual-cortex receptive fields adapt to stimuli. SKNets address this gap with Selective Kernel convolutions that select among differently sized kernels using fused branch information. The resulting networks achieve strong benchmark performance with modest complexity and exhibit input-dependent receptive-field selection.

  • Problem

    CNNs commonly assign fixed receptive-field sizes, while stimulus-modulated receptive fields in visual cortex have been rarely incorporated into CNN design.

  • Method

    Selective Kernel convolution splits inputs across different kernel sizes, fuses branch information, and selects their feature maps using selection weights within stacked SK units.

  • Results

    SKNets achieve state-of-the-art performance across ImageNet and CIFAR benchmarks with lower or similar complexity, while larger objects shift neuron responses toward larger-kernel paths.

  • Takeaways & Limitations

    Input-dependent kernel selection gives SKNet neurons effective receptive fields that adapt to target-object scale across large and small models.

Abstract

from arXiv · show

In standard Convolutional Neural Networks (CNNs), the receptive fields of artificial neurons in each layer are designed to share the same size. It is well-known in the neuroscience community that the receptive field size of visual cortical neurons are modulated by the stimulus, which has been rarely considered in constructing CNNs. We propose a dynamic selection mechanism in CNNs that allows each neuron to adaptively adjust its receptive field size based on multiple scales of input information. A building block called Selective Kernel (SK) unit is designed, in which multiple branches with different kernel sizes are fused using softmax attention that is guided by the information in these branches. Different attentions on these branches yield different sizes of the effective receptive fields of neurons in the fusion layer. Multiple SK units are stacked to a deep network termed Selective Kernel Networks (SKNets). On the ImageNet and CIFAR benchmarks, we empirically show that SKNet outperforms the existing state-of-the-art architectures with lower model complexity. Detailed analyses show that the neurons in SKNet can capture target objects with different scales, which verifies the capability of neurons for adaptively adjusting their receptive field sizes according to the input. The code and models are available at https://github.com/implus/SKNet.

1. Introduction

The paper targets CNNs’ limited ability to adapt receptive-field size to stimulus content. It introduces Selective Kernel convolutions, which combine multi-scale paths with learned selection to support adaptive receptive fields while maintaining efficient models.

  • Motivation: Visual-cortex neurons adapt their receptive-field sizes to stimulus properties, but this behavior has received limited attention in CNN design.Cortical receptive fields can enlarge or vary with stimulus contrast and surrounding stimulation.
  • Motivation: Linear aggregation of multi-scale branches may not provide sufficient adaptation, motivating a nonlinear aggregation approach.
  • Method: Selective Kernel convolution uses Split, Fuse, and Select operators to combine paths with different kernel sizes according to learned selection weights.Split creates different receptive-field paths, Fuse builds a global representation, and Select aggregates their feature maps.
  • Results: SK convolutions add only slight parameter and computational cost while SKNets outperform prior state-of-the-art models at similar ImageNet complexity.
  • Results: When target objects become larger, most SKNet neurons increasingly collect information from the larger-kernel path.This stimulus-scaling experiment supports adaptive receptive-field sizes in the proposed network.

2. Related Work

The related work situates SKNets among multi-branch and efficient convolutional architectures. SKNets retain multiple filters but add simpler design and adaptive branch selection for receptive-field adjustment.

  • Multi-branch convolutional networks: Prior multi-branch CNNs use bypasses, repeated paths, tree structures, fractal expansion, or customized filters to aggregate features.
  • Multi-branch convolutional networks: SKNets differ from InceptionNets through simpler branch schemes and adaptive selection across branches to realize adaptive receptive-field sizes.
  • Grouped/depthwise/dilated convolutions: Grouped convolutions reduce parameters and computation by dividing them by the group size G relative to ordinary convolution.
  • Grouped/depthwise/dilated convolutions: Depthwise separable convolutions decompose ordinary convolution into depthwise and pointwise convolution, supporting compact models such as Xception and MobileNetV1.

3. Methods

SK convolution adaptively selects among branches with different receptive-field sizes using fused global information and softmax channel attention. SK units stack these operations into efficient networks that support adaptive receptive fields while remaining applicable to compact architectures.

  • Selective Kernel Convolution: SK convolution uses Split, Fuse, and Select operators to choose among multiple kernels with different receptive-field sizes.Split creates branches, Fuse aggregates their information, and Select applies adaptive attention across the branches.
  • Selective Kernel Convolution: The default Split operator creates kernel-3 and kernel-5 paths, replacing the conventional 5×5 convolution with a dilated 3×3 convolution for efficiency.The branches use grouped or depthwise convolutions, Batch Normalization, and ReLU.
  • Selective Kernel Convolution: Fuse sums branch outputs and applies global average pooling to produce channel-wise statistics that guide the selection weights.A compact feature is then formed with a fully connected layer and dimensionality reduction.
  • Selective Kernel Convolution: Select uses softmax attention across channels to combine differently scaled feature maps, with two-branch weights satisfying ac + bc = 1.The final feature map is obtained by weighting the branch outputs according to these attention values.
  • Network Architecture: SK units replace large-kernel convolutions in ResNeXt-style bottleneck blocks, and SKNet-50 increases parameters by 10% and computational cost by 5% over ResNeXt-50.The architecture uses repeated 1×1 convolution, SK convolution, and 1×1 convolution sequences.
  • Network Architecture: SK convolutions can also replace 3×3 depthwise convolutions in lightweight networks such as MobileNet and ShuffleNet.The paper reports appealing results from embedding SK into compact architectures.

4. Experiments

Experiments show that SKNets improve ImageNet performance over strong architectures at similar complexity, while ablations support adaptive multi-kernel selection and analyses show receptive-field adaptation to object scale.

  • ImageNet Classification: SKNets consistently improve performance over state-of-the-art attention-based CNNs under similar ImageNet model budgets.SKNet-50 outperforms ResNeXt-101 by above absolute 0.32% despite 60% fewer parameters and 80% less computation.
  • ImageNet Classification: With comparable or lower complexity than InceptionNets, SKNets achieve above absolute 1.5% gain of performance.
  • ImageNet Classification: Using slightly fewer parameters, SKNets obtain 0.3∼0.4% gains over SENet counterparts at both 224×224 and 320×320 evaluation resolutions.
  • Selective Kernel vs. Depth/Width/Cardinality: Increasing ResNeXt depth, width, or cardinality yields marginal gains of 0.19%, 0.1%, and 0.23%, whereas SKNet-50 improves 1.44% over ResNeXt-50.These comparisons match overall computational complexity to SKNet.
  • Ablation Studies: Under similar complexity, the optimal second branch uses a 5×5 kernel rather than the fixed 3×3 kernel, supporting aggregation of multi-scale information.Dilated 3×3 kernels with various dilation rates are empirically moderately superior to corresponding undilated large kernels in performance and complexity.
  • Ablation Studies: SK attention consistently outperforms simple path summation, while increasing paths from M = 2 to M = 3 reduces top-1 error only from 20.79% to 20.76%.The authors therefore prefer M = 2 for the performance-efficiency trade-off.
  • Receptive-Field Analysis: When target objects enlarge, most channels increase attention to the 5×5 path, indicating that neurons adaptively enlarge their receptive fields.
  • Receptive-Field Analysis: The same increase in 5×5 importance occurs across all 1,000 ImageNet categories, but the pattern disappears in much higher layers where scale information is lost.

5. Conclusion

SKNets use adaptive kernel selection to adjust receptive field sizes, improving object-recognition efficiency and effectiveness across benchmarks and model scales.

  • SKNets introduce Selective Kernel convolution, which adaptively selects among differently sized kernels using soft attention.The mechanism is inspired by adaptive receptive fields in visual cortex.
  • SKNets achieve state-of-the-art performance across benchmarks, from large models to tiny models.The conclusion also reports meaningful kernel-selection behavior across channel, depth, and category.
  • The study empirically validates that SKNets adapt receptive field sizes and links this behavior to better object recognition.

A. Details of the Compared Models in Table 3

The compared ResNeXt variants are modified in width, depth, or cardinality so their overall complexity matches SKNet-50 for fair evaluation.

  • ResNeXt-50, wider adds 16 channels in every bottleneck block.
  • ResNeXt-56, deeper adds two blocks at the end of ResNeXt-50’s fourth stage.
  • ResNeXt-50 (36×4d) uses cardinality 36 instead of 32.
  • All three variants match SKNet-50’s overall complexity, supporting fair comparisons.

B. Details of the Models in Table 4

The lightweight-model comparisons reimplement ShuffleNetV2 with altered stage depths and evaluate integrations of SE and SK modules, including activation-function settings.

  • ShuffleNetV2 is reimplemented with 0.5× and 1.0× settings for fair comparison.
  • The three stage block counts change from {4,8,4} to {4,6,6}, altering reported performance and computational costs.
  • In the lightweight comparison, “+ SE” integrates SE after each shuffle layer, while “+ SK” replaces each 3×3 depthwise convolution with an SK unit.The SK unit uses two paths with K3 and K5 kernels, r = 4, and G equal to the corresponding stage’s channel count.
  • The best ShuffleNet integration omits ReLU activations in both SK paths.
  • Evaluation uses a single 224×224 crop on the ImageNet validation set.

C. Details of the Compared Models in Figure 2

Figure 2 compares state-of-the-art CNN families and their variants by parameter count and ImageNet validation performance, with SENets based on corresponding ResNeXts.

  • Figure 2 includes variants of ResNet, ResNeXt, DenseNet, DPN, and SENet models.
  • Each plotted dot represents one variant of a model family.
  • Table S9 provides variant settings, parameter counts, and ImageNet validation results.SENets are based on the corresponding ResNeXts.

D. Implementation Details on CIFAR Datasets (Section 4.2)

The CIFAR experiments use a fixed training setup across two datasets, with dataset-specific learning rates and scheduled reductions. The setup also specifies optimization hyperparameters and model initialization details.

  • CIFAR-10 and CIFAR-100 networks are trained on 2 GPUs with mini-batches of 128 for 300 epochs.
  • The initial learning rates are 0.1 for CIFAR-10 and 0.05 for CIFAR-100, reduced tenfold at 50% and 75% of training.
  • Training uses weight decay of 5e-4, momentum of 0.9, and the initialization method introduced in [8].

E. More Examples of Dynamic Selection

Supplementary analyses examine attention across images containing targets at three scales and report additional ImageNet validation results under single-crop testing. Larger targets receive more attention from larger kernels in low- and middle-level stages.

  • The dynamic selection mechanism assigns more attention to larger kernels as the target object becomes larger in low- and middle-level stages.
  • Table S9 reports top-1 error rates on the ImageNet validation set using single 224×224 crop testing.
Loading 1903.06586v2…