PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs

Xiaohan Ding, Xiangyu Zhang, Yizhuang Zhou, Jungong Han, Guiguang Ding, Jian Sun

arXiv:2203.06717v4cs.CVcs.AIcs.LG

TL;DR

CNNs usually build receptive fields by stacking small convolutions, leaving open whether a few large kernels could provide a stronger alternative to transformer-style large-region aggregation. The paper develops RepLKNet with re-parameterized large depth-wise convolutions up to 31×31 and reports competitive classification and downstream results, while identifying a scaling limitation against Swin at larger data and model scales.

  • Problem

    CNNs typically use stacks of small convolutions, while ViTs gather information from larger regions; the paper investigates whether a few large kernels can provide a more powerful CNN design.

  • Method

    The paper systematically studies large convolutions, formulates five design guidelines, and builds RepLKNet as a pure CNN using re-parameterized large depth-wise convolutions.

  • Results

    RepLKNet closes much of the CNN–ViT gap, including 84.8% top-1 accuracy on ImageNet-1K, 4.4% higher COCO detection performance, and 6.1% higher ADE20K segmentation performance than comparable CNN backbones.

  • Takeaways & Limitations

    Large-kernel CNNs obtain larger effective receptive fields and use more shape information than conventional CNNs, supporting large kernels as an alternative to multi-head self-attention.

  • Takeaways & Limitations

    At larger data and model scales, RepLKNet-31L is 0.7% lower in ImageNet top-1 accuracy than Swin-L with ImageNet-22K pretraining, while downstream scores remain comparable.

Abstract

from arXiv · show

We revisit large kernel design in modern convolutional neural networks (CNNs). Inspired by recent advances in vision transformers (ViTs), in this paper, we demonstrate that using a few large convolutional kernels instead of a stack of small kernels could be a more powerful paradigm. We suggested five guidelines, e.g., applying re-parameterized large depth-wise convolutions, to design efficient high-performance large-kernel CNNs. Following the guidelines, we propose RepLKNet, a pure CNN architecture whose kernel size is as large as 31x31, in contrast to commonly used 3x3. RepLKNet greatly closes the performance gap between CNNs and ViTs, e.g., achieving comparable or superior results than Swin Transformer on ImageNet and a few typical downstream tasks, with lower latency. RepLKNet also shows nice scalability to big data and large models, obtaining 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K, which is very competitive among the state-of-the-arts with similar model sizes. Our study further reveals that, in contrast to small-kernel CNNs, large-kernel CNNs have much larger effective receptive fields and higher shape bias rather than texture bias. Code & models at https://github.com/megvii-research/RepLKNet.

1. Introduction

The paper asks whether a few large kernels can replace stacks of small convolutions for building receptive fields in CNNs. It develops RepLKNet around large depth-wise convolutions and reports strong classification and downstream performance against transformer and CNN baselines.

  • ViTs commonly gather information from large regions through global or local large-kernel attention, whereas CNNs typically stack many 3×3 convolutions to enlarge receptive fields.
  • The authors systematically explore kernels from 3×3 to 31×31 and summarize five guidelines for efficient large-convolution design.The guidelines include large depth-wise kernels, identity shortcuts, and re-parameterization with small kernels.
  • RepLKNet is a pure CNN that replaces multi-head self-attention with re-parameterized large depth-wise convolutions and uses kernels up to 31×31.Its macro-architecture generally follows Swin Transformer with modifications.
  • 84.8% top-1 accuracy on ImageNet-1K makes the similarly sized RepLKNet baseline 0.3% better than Swin-B with lower latency.
  • 4.4% on COCO detection and 6.1% on ADE20K segmentation are the reported improvements over comparable ResNeXt-101 or ResNet-101 backbones.
  • 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K demonstrate competitive scaling with more pretraining data and computation.
  • The authors associate RepLKNet’s performance with larger effective receptive fields and greater use of shape information than conventional CNNs.

2. Related Work

Related work has explored large convolutions, attention alternatives, and scaling strategies, but prior approaches generally used smaller kernels or did not establish strong large-scale CNN comparisons. Structural re-parameterization provides a mechanism for training with auxiliary structures while producing an equivalent inference model.

  • Global Convolution Networks use factorized 1×K and K×1 convolutions for semantic segmentation, while large kernels were also reported to harm ImageNet performance.
  • Swin Transformer uses shifted windows of size 7 to 12, and follow-up convolutional models replace attention with static or dynamic 7×7 depth-wise convolutions.
  • Earlier large-kernel and attention-alternative works did not answer how large-kernel CNNs scale relative to strong transformer baselines such as models larger than Swin-L.
  • ConvMixer uses convolutions up to 9×9, while ConvNeXt uses 7×7 depth-wise convolutions, without demonstrating benefits from kernels such as 31×31.
  • CNN scaling commonly varies depth, width, resolution, bottleneck ratio, and group width, while kernel size is often neglected despite its importance for downstream tasks.
  • Structural re-parameterization equivalently converts model structures through parameter transformations, allowing small kernels or shortcuts used during training to be absorbed into inference kernels.

3. Guidelines of Applying Large Convolutions

The section presents five guidelines for making large-kernel convolutions effective, emphasizing depth-wise computation, shortcuts, re-parameterization, and downstream-task benefits. Experiments show that these choices improve efficiency, optimization, and performance as kernels grow.

  • Guideline 1: Large depth-wise convolutions increase RepLKNet FLOPs by 18.6% and parameters by 10.4% when stage kernels grow from [3] to [29] [27] [13].Large kernels improve computational density, while optimized implementations reduce their latency contribution from 49.5% to 12.3%.
  • Guideline 2: Identity shortcuts improve large-kernel MobileNet V2 accuracy by 0.77%, whereas removing shortcuts causes large kernels to reduce accuracy.The comparison replaces depth-wise 3×3 layers with 13×13 layers under identical ImageNet training settings.
  • Guideline 3: 3×3 re-parameterization adds a parallel small-kernel branch during training, then merges it and batch-normalization parameters into the large kernel for inference.The resulting inference model is equivalent to the training model while containing no small kernels.
  • Guideline 4: After re-parameterization, increasing MobileNet V2 kernels from 3×3 to 9×9 improves ImageNet accuracy by 1.33% and Cityscapes mIoU by 3.99%.For the larger RepLKNet configuration, increasing kernels to [29] [27] [13] improves ImageNet accuracy by 0.96% and ADE20K mIoU by 3.12%.
  • Interpretation: Large kernels increase shape bias and effective receptive fields, providing a design rationale for their stronger downstream-task gains.The supplied passage attributes the downstream advantage partly to larger effective receptive fields and increased shape bias.
  • Guideline 5: A 13×13 kernel remains useful on a 7×7 feature map, improving performance even when the kernel is on par with or larger than the feature-map size.The experiment enlarges only the last-stage depth-wise convolutions while retaining 3×3 kernels in earlier stages.

4. RepLKNet: a Large-Kernel Architecture

RepLKNet is a pure CNN that uses re-parameterized large depth-wise convolutions within a staged architecture. Across classification, segmentation, and detection, it shows strong accuracy-efficiency trade-offs and scalability.

  • Architecture: RepLKNet is a pure CNN architecture designed around large-kernel convolutions and complexity comparable to or exceeding ResNet-152 or Swin-B.The architecture targets whether large kernels can reduce the performance gap between CNNs and vision transformers.
  • Architecture: Each RepLK Block combines shortcuts, depth-wise large kernels, and 1×1 convolutions, with 5×5 re-parameterization used during training.Stages are separated by transition blocks that increase channels with 1×1 convolution and downsample with depth-wise 3×3 convolution.
  • Kernel-size scaling: Kernel-size experiments vary K while fixing B=[2, 2, 18, 2] and C=[128, 256, 512, 1024], comparing RepLKNet-3/7/13/25/31.The models use progressively larger kernel configurations, including [29] [27] [13] for RepLKNet-31.
  • Kernel-size scaling: 0.82 higher mIoU on ADE20K results from scaling kernels from [13] to [29] [27] [13], with only 5.3% more parameters and 3.5% higher FLOPs.On ImageNet, increasing kernels from 3 to 13 improves accuracy, while further enlargement brings no additional improvement.
  • Benchmark results: 84.8% ImageNet-1K accuracy makes RepLKNet-31B 0.3% higher than Swin-B while running 43% faster.The models also outperform Swin-B on Cityscapes and ADE20K, while RepLKNet-XL reaches 56.0 mIoU on ADE20K and 55.5 mAP on COCO.

5. Discussions

The discussion attributes large-kernel CNN advantages to more effective receptive-field expansion than deep stacks of small kernels. It also connects large kernels with higher shape bias and identifies kernel size as a general CNN scaling dimension.

  • Motivation: A 7×7 convolution can be decomposed into three 3×3 convolutions, yet the paper asks why conventional deep small-kernel CNNs remain inferior to large-kernel networks.The comparison motivates examining receptive-field construction rather than only representational equivalence.
  • Effective receptive field: ERF grows linearly with kernel size but sub-linearly with depth, so large kernels obtain broad receptive fields with fewer layers.The paper also notes that increasing depth introduces optimization difficulty and that ResNets may behave like ensembles of shallow networks.
  • Effective receptive field: RepLKNet-31 distributes high-contribution pixels more uniformly than ResNet-101/152 and RepLKNet-13, indicating a larger effective receptive field.ResNet-152 shows a pattern similar to ResNet-101, suggesting additional 3×3 layers do not substantially enlarge the ERF.
  • Shape bias: RepLKNet-31B has higher shape bias than Swin Transformer and small-kernel CNNs.The analysis relates this pattern to effective receptive fields, while RepLKNet-3 is close to ResNet-152 because both use 3×3 convolutions.
  • Generic design element: Replacing ConvNeXt’s 7×7 kernels with kernels up to 31×31 improves models across size counterparts, showing kernel size is an important scaling dimension.The reported comparisons include ConvNeXt-Tiny with large kernels outperforming ConvNeXt-Small and ConvNeXt-Small with large kernels outperforming ConvNeXt-Base.

6. Limitations

At larger data and model scales, RepLKNet begins to trail Swin Transformers on ImageNet, although downstream scores remain comparable. The authors leave open whether this reflects hyperparameter tuning or a fundamental CNN limitation.

  • 0.7% lower ImageNet top-1 accuracy is reported for RepLKNet-31L than Swin-L with ImageNet-22K pretraining.

7. Conclusion

The paper argues that replacing many small kernels with a few large kernels improves CNN receptive fields and performance, especially on downstream tasks. It positions large convolutions as an alternative lens for understanding self-attention.

  • A few large kernels produce larger effective receptive fields more efficiently than many small kernels.
  • Large-kernel CNNs substantially improve downstream performance and narrow the gap between CNNs and ViTs as data and models scale.
  • Large convolutions may help explain the intrinsic mechanism of multi-head self-attention because they exhibit similar behaviors.

Appendix A: Training Configurations

The appendix documents distinct training configurations for MobileNet V2, RepLKNet variants, and the scaled RepLKNet-XL model, including changes in hardware, optimization, augmentation, resolution, and pretraining data.

  • MobileNet V2 training uses 8 GPUs, SGD with momentum 0.9, batch size 32 per GPU, 224×224 inputs, and 100 epochs.
  • RepLKNet training uses 32 GPUs, AdamW, batch size 64 per GPU, 120 epochs, cosine annealing, and 10-epoch warm-up.
  • RepLKNet-31B fine-tuning at 384×384 uses 32 GPUs, batch size 32 per GPU, 30 epochs, and no CutMix or mixup.
  • ImageNet-22K pretraining of RepLKNet-31B/L uses 128 GPUs, batch size 32 per GPU, 90 epochs, and a 10% drop-path rate.
  • RepLKNet-XL uses B = [2, 2, 18, 2], C = [256, 512, 1024, 2048], K = [27, 27, 27, 13], and inverted bottlenecks with expansion ratio 1.5.
  • RepLKNet-XL pretraining uses MegData73M, containing 38 million labeled and 35 million unlabeled images with soft pseudo labels from task-specific ViT-Ls.
  • RepLKNet-XL is fine-tuned on ImageNet-1K at 320×320 for 30 epochs, while downstream training uses a 50% drop-path rate and stage-wise learning-rate decay.

Appendix B: Visualizing the ERF

The ERF visualization procedure uses contribution scores derived from input-to-feature relationships, aggregates them across examples and channels, and rescales each model for comparison.

  • Each model’s aggregated contribution matrix is rescaled to [0, 1] by dividing by its maximum entry for cross-model comparability.

Appendix C: Dense Convolutions vs. Dilated Convolutions

Depth-wise dilated convolutions can match dense convolutions in maximum receptive field but have substantially lower representational capacity. The authors attribute this limitation to their equivalence with sparse large convolutions and note a possible gridding problem.

  • Depth-wise dilated convolutions have the same maximum receptive field as depth-wise dense convolutions but much lower representational capacity.The text attributes this to their mathematical equivalence with sparse large convolutions.
  • Dilated convolutions may suffer from a gridding problem, according to cited literature.
  • Mixtures of convolutions with different dilations are proposed as a future way to address these drawbacks.

Appendix D: Visualizing the Kernel Weights with Small-Kernel Re-parameterization

The appendix visualizes aggregated 13×13 MobileNet V2 kernels with and without 3×3 re-parameterization. Kernels generally emphasize central and criss-cross weights, while 3×3 re-parameterization further strengthens the center and improves performance.

  • 73.24% and 72.53% are the ImageNet scores for MobileNet V2 models with and without 3×3 re-parameterization, respectively.
  • Aggregated 13×13 kernels generally have the largest magnitude at the center, with values decreasing toward the edges.
  • The 13×1 and 1×13 criss-cross “skeleton” parameters are relatively larger than surrounding kernel parameters.
  • 3×3 re-parameterization further enhances the central 3×3 region, which improves performance.
  • Figure 6 aggregates MobileNet V2 13×13 kernel parameters into 13×13 matrices.
Loading 2203.06717v4…