PaperlayerView exact revision on arXiv

Source-linked AI summary

More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkäinen, Mykola Pechenizkiy, Decebal Mocanu, Zhangyang Wang

arXiv:2207.03620v3cs.CV

TL;DR

Existing large-kernel ConvNets saturate beyond 31×31, leaving unclear whether CNNs can scale further without losing performance or locality. The paper introduces a sparsity-based recipe using rectangular decomposition and sparse width expansion, yielding SLaK with 51×51 kernels; across ImageNet and downstream tasks, it performs better than or on par with strong ConvNets and Transformers.

  • Problem

    Existing large-kernel CNN methods saturate beyond 31×31, leaving unresolved whether extreme kernels can surpass Transformer-based models while retaining useful locality.

  • Method

    The paper combines rectangular kernel decomposition with sparse groups and increased width, using dynamic sparsity to train sparse kernels from scratch.

  • Results

    SLaK uses sparse factorized 51×51 kernels and performs better than or on par with major ConvNets and attention-based models across ImageNet and stated downstream tasks.

  • Takeaways & Limitations

    Sparsity provides a recipe for scaling pure CNN kernels beyond 31×31, reaching kernels up to 61×61 with improved performance and reduced FLOPs.

  • Takeaways & Limitations

    The kernel-decomposition hypothesis assumes that M×N and N×M kernels preserve large effective receptive fields while retaining fine-grained local feature capture.

Abstract

from arXiv · show

Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local-window attention mechanism, showing appealing performance and efficiency. While one of them, i.e. RepLKNet, impressively manages to scale the kernel size to 31x31 with improved performance, the performance starts to saturate as the kernel size continues growing, compared to the scaling trend of advanced ViTs such as Swin Transformer. In this paper, we explore the possibility of training extreme convolutions larger than 31x31 and test whether the performance gap can be eliminated by strategically enlarging convolutions. This study ends up with a recipe for applying extremely large kernels from the perspective of sparsity, which can smoothly scale up kernels to 61x61 with better performance. Built on this recipe, we propose Sparse Large Kernel Network (SLaK), a pure CNN architecture equipped with sparse factorized 51x51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architectures like ConvNeXt and RepLKNet, on ImageNet classification as well as a wide range of downstream tasks including semantic segmentation on ADE20K, object detection on PASCAL VOC 2007, and object detection/segmentation on MS COCO.

1 INTRODUCTION

Large receptive fields motivate recent large-kernel ConvNets, but existing approaches saturate beyond 31×31. The paper addresses this gap with sparsity-based kernels that scale further while retaining strong performance.

  • 1 INTRODUCTION: ViTs use global or larger local attention windows to capture larger receptive fields than conventional small-window CNN convolutions.Recent CNNs respond by incorporating large kernels, including RepLKNet’s 31×31 convolutions.
  • 1 INTRODUCTION: RepLKNet reaches 31×31 kernels with results comparable to Swin Transformer, but performance saturates as kernels grow further.This leaves open whether CNNs can exceed Transformer-based models by scaling beyond 31×31.
  • 1 INTRODUCTION: The paper studies extreme kernels through sparsity and reports three observations about trainability, decomposition, and width expansion.Naive enlargement and structural re-parameterization fail beyond 31×31, while rectangular decomposition reaches 61×61 and sparse groups with increased width improve performance.

2 RELATED WORK

Related work connects large receptive fields to both attention and convolution, while dynamic sparsity offers a way to train intrinsically sparse networks from scratch.

  • Large Kernel in Attention: Self-attention acts like a global depth-wise kernel, while Swin Transformer uses shifted local windows to improve efficiency with large receptive fields.Attention windows of at least 7 are presented as an alternative class of large kernels.
  • Large Kernel in Convolution: Large convolutional kernels have expanded from 7×7 and 11×11 to factorized forms such as 1×M plus M×1, though earlier factorization degraded ImageNet performance.The related work frames factorization as an established but imperfect route to larger receptive fields.
  • Dynamic Sparsity: Dynamic sparsity adjusts sparse weights during training rather than relying only on post-training pruning.Figure 2 describes pruning less important weights and adding new weights to optimize sparse kernels.
  • Dynamic Sparsity: Dynamic sparsity trains intrinsically sparse neural networks from scratch using only a small proportion of parameters and FLOPs.This reduces training and inference computation and memory relative to dense models.

3 FAILURES OF EXISTING APPROACHES TO GO BEYOND 31×31 KERNELS

The paper tests whether existing large-kernel ConvNet techniques can scale beyond 31×31 and identifies loss of locality as a likely explanation for their failure.

  • 3 FAILURES OF EXISTING APPROACHES TO GO BEYOND 31×31 KERNELS: The study benchmarks ConvNeXt and RepLKNet techniques on ImageNet-1K for extreme kernels larger than 31×31.ConvNeXt serves as the benchmark, using an efficient large-kernel implementation.
  • 3 FAILURES OF EXISTING APPROACHES TO GO BEYOND 31×31 KERNELS: The experiments use 120-epoch training to sketch large-kernel scaling trends, reserving 300 epochs for later state-of-the-art comparisons.The training setup applies standard augmentation, regularization, and AdamW optimization.
  • 3 FAILURES OF EXISTING APPROACHES TO GO BEYOND 31×31 KERNELS: Existing large-kernel methods lose performance as kernel sizes increase, despite RepLKNet’s success at 31×31.Table 1 compares direct enlargement with structural re-parameterization against the original 7×7 ConvNeXt.
  • 3 FAILURES OF EXISTING APPROACHES TO GO BEYOND 31×31 KERNELS: At 51×51, an extreme kernel is roughly global after the standard 4× downsampling, suggesting that excessive enlargement may weaken locality.The paper therefore seeks to preserve global relations while reintroducing local structure.

4 A RECIPE FOR EXTREMELY LARGE KERNELS BEYOND 31×31

The proposed recipe combines rectangular kernel decomposition with sparse groups and increased width. This enables larger kernels, dynamic sparse adaptation, and improved efficiency-performance trade-offs.

  • A RECIPE FOR EXTREMELY LARGE KERNELS BEYOND 31×31: The recipe has two steps: decompose large kernels into parallel rectangular kernels, then use sparse groups while expanding width.These steps target both trainability and model capacity.
  • Decomposing a large kernel: An M×M kernel is approximated by parallel M×N and N×M kernels, balancing long-range dependencies with local detail.The shorter edge N provides locality while the long edge preserves broad spatial coverage.
  • Decomposing a large kernel: The decomposition’s computational and memory overhead grows linearly with kernel size rather than quadratically.It initially sacrifices some accuracy at 31×31 because it reduces parameters and FLOPs, then improves performance at larger sizes up to 61×61.
  • Using sparse groups, expand more width: Sparse groups extend ConvNeXt’s increased-width principle by replacing dense convolutions with sparse convolutions and dynamically adapting weights.Weights are initialized using layer-wise SNIP sparsity and updated by pruning low-magnitude weights while randomly growing replacements.
  • Using sparse groups, expand more width: At 40% sparsity, 1.3× width keeps parameters and FLOPs roughly comparable to the dense model while increasing performance from 81.3% to 81.6% with 51×51 kernels.With 61×61 kernels, the method outperforms prior state of the arts while saving 55% FLOPs.
  • Large Kernels Generalize Better than Small Kernels with Our Recipe: Applying the recipe to 7×7 kernels yields no gain or only marginal gains relative to 51×51 kernels, supporting large kernels as the main source of improvement.The breakdown experiment reports consistently increasing performance with kernel size up to 51×51.
  • SLaK: SLaK builds on ConvNeXt with 51×51 factorized kernels, 1.3× wider stages, and 40% sparsity.The implementation replaces each M×M kernel with M×5 and 5×M kernels and inserts BatchNorm before summing outputs.

5 EVALUATION OF SLAK

SLaK is evaluated across ImageNet classification, semantic segmentation, and object detection, where larger sparse kernels consistently match or exceed established CNN and Transformer baselines.

  • ImageNet-1K: SLaK outperforms or matches RepLKNet, ConvNeXt, Swin Transformer, and other state-of-the-art models on ImageNet-1K with similar model sizes and FLOPs.The model achieves these results without complex attention modules or patch embedding.
  • ImageNet-1K: 0.7% higher accuracy over ConvNeXt-S results from directly replacing its 7×7 kernels with 51×51 kernels.SLaK-B gains relatively more over ConvNeXt-B at 384×384 than at 224×224 input resolution.
  • ADE20K: 1.2% mIoU improvement over ConvNeXt-T (RepLKNet) is achieved by SLaK-T with 51×51 kernels on ADE20K semantic segmentation.The improvement also surpasses ConvNeXt-S, and the advantage persists under longer training procedures.
  • PASCAL VOC 2007: 1.4% mAP improvement over 7×7 kernels is achieved by SLaK-T with 51 kernel sizes on PASCAL VOC 2007 object detection.ConvNeXt-T with 31×31 kernels already improves mAP by 0.7% over its 7×7 version.
  • MS COCO: SLaK with 51×51 kernels consistently outperforms smaller-kernel models on MS COCO using Cascade Mask R-CNN backbones.The COCO experiments use multi-scale training and AdamW.

6 ANALYSIS OF SLAK

SLaK combines factorized large kernels with sparse grouping to scale receptive fields efficiently while retaining local detail and improving resource requirements.

  • Kernel decomposition: The decomposed M×N and N×M kernels are designed to preserve large-kernel receptive fields while using the shorter edge N to focus on local features.The hypothesis links factorization to simultaneous long-range and fine-grained feature capture.
  • Effective receptive field: SLaK captures a larger effective receptive field than RepLKNet while retaining high-contribution pixels in non-center regions.This supports balancing long-range dependencies with local details.
  • Kernel scaling efficiency: Beyond 31×31, kernel decomposition creates a large scaling-efficiency gap over full-kernel scaling in GFLOPs and parameter count.Even 151×151 kernels with the proposed methods require fewer FLOPs and parameters than full-kernel 51×51 scaling.
  • Inference latency: Without sparsity-friendly hardware, vanilla sparse large kernels have runtime similar to dense large kernels of the same size.The latency measurements use one-layer depth-wise convolutions on a single A100 GPU in FP32 without special acceleration.

7 CONCLUSION

The paper concludes that sparsity enables ConvNets to scale beyond 31×31 kernels while achieving better performance than Swin Transformer and ConvNeXt.

  • Conclusion: SLaK provides a sparsity-inspired recipe for extremely large kernels and builds a pure ConvNet that scales kernel size beyond 51×51.The reported results suggest sparsity can support network scaling.

Contents of the Appendices:

The appendices provide implementation details, additional sparsity and kernel analyses, further comparisons, and a limitations discussion.

  • Experimental details: Appendix A covers experimental settings, hyperparameters, and configurations used throughout the paper.These materials support reproduction of the reported experiments.
  • Additional analyses: Appendices B–I examine dynamic sparsity, comparisons, sparsity-width trade-offs, kernel decomposition, receptive fields, learning curves, throughput, and related design choices.The appendices include comparisons with CSwin Transformers and analyses of the shorter decomposed-kernel edge.
  • Limitations and broader discussion: Appendix J investigates the position of large kernels, while Appendix K addresses limitations and broad impact.These sections extend the main analysis beyond core benchmark results.

A EXPERIMENTAL SETTINGS

The experiments use dynamic sparsity with sparse initialization, masked AdamW optimization, and periodic prune-and-grow adaptation. SLaK is evaluated across ImageNet and downstream tasks using standardized training setups and comparisons with CSwin Transformer.

  • A EXPERIMENTAL SETTINGS: SLaK experiments use AdamW, batch size 4096, weight decay 0.05, and ImageNet pre-training for either 120 or 300 epochs.The learning rate is 4e-3 with 20-epoch warmup followed by cosine decay.
  • A EXPERIMENTAL SETTINGS: Downstream evaluations cover ADE20K segmentation, PASCAL VOC detection, and COCO detection/segmentation, with SLaK also performing on par with CSwin Transformer.The CSwin comparison uses AdamW for 300 epochs.
  • B DYNAMIC SPARSITY: Dynamic sparsity initializes binary masks, optimizes masked weights with AdamW, and periodically prunes and randomly regrows parameters.The adaptation rate follows a cosine decay schedule.
  • B.1 SPARSE MODEL INITIALIZATION: SNIP-based layer-wise sparsity ratios are used to balance trainability, capacity, computational FLOPs, and final performance.SNIP scores are computed from weight-gradient products and globally thresholded across layers.
  • B.3 SPARSE WEIGHT ADAPTATION: Sparse weight adaptation removes low-magnitude weights and randomly grows the same number of new weights, keeping parameter count fixed while exploring new connections.SLaK-T uses adaptation frequency f = 100 and adaptation rate p = 0.3.

D TRADE-OFF BETWEEN SPARSITY AND WIDTH

The study examines how sparsity, width, kernel size, and effective receptive field interact. Increasing width helps until a moderate expansion, while larger kernels improve loss, accuracy, and receptive-field coverage.

  • D TRADE-OFF BETWEEN SPARSITY AND WIDTH: At roughly 5.0M FLOPs, performance increases with width up to a 1.5× width factor, then declines as extreme sparsity makes training difficult.The tested settings range from (0.20, 1.1×) to (0.82, 2.5×) for (sparsity, width factor).
  • F ERF QUANTITATION OF MODELS WITH DIFFERENT KERNEL SIZES: SLaK’s global kernels consider a larger range of input pixels than ConvNeXt and RepLKNet, as measured by the high-contribution area ratio r.A larger r indicates a smoother distribution of high-contribution pixels and a larger effective receptive field.
  • D TRADE-OFF BETWEEN SPARSITY AND WIDTH: Increasing kernel size from 31×31 to 51×51 further decreases ImageNet training loss and improves test accuracy while retaining promising convergence speed.The comparison is shown through learning curves for multiple architectures.

H INFERENCE THROUGHPUT MEASUREMENT

The throughput analysis separates kernel decomposition from sparsity and compares their practical costs on common GPUs. Decomposition substantially improves scaling efficiency, but unstructured sparsity alone does not provide hardware speedups.

  • H INFERENCE THROUGHPUT MEASUREMENT: Swin-T and ConvNeXt-T achieve throughput around 1.8× and 2.5× larger than SLaK-T on an A100 GPU without sparsity-aware acceleration.The measurement uses PyTorch 1.10.0, cuDNN 8.2.0, FP32 precision, and ImageNet-1K.
  • H INFERENCE THROUGHPUT MEASUREMENT: 3× speedup is achieved by decomposing large kernels compared with naive kernel scaling, while the 1.3× wider model remains 2× faster than a vanilla large-kernel model.Sparse large kernels have comparable runtimes to dense large kernels without sparsity-aware accelerators.
  • APPENDIX LARGE-KERNEL DECOMPOSITION: Parallel and sequential kernel decomposition outperform stacked small kernels and dilated kernels with similar receptive fields, with sequential decomposition 0.1% below parallel decomposition.The comparison controls for dynamic sparsity and adds a 5×5 kernel to all settings.
  • J EFFECT OF LARGE KERNEL POSITION: Large kernels are more useful in later stages: scaling only the third and fourth stages matches scaling all stages, while early-stage scaling provides no benefit.The ablation evaluates kernel placement across four model stages.

K LIMITATIONS AND DISCUSSION OF BROADER IMPACT

The paper’s main limitation is practical hardware support: its binary-mask sparse architecture does not translate theoretical efficiency into real speedups on common GPUs and TPUs. The authors also discuss possible negative uses and resource trade-offs.

  • K LIMITATIONS: Binary-mask unstructured sparsity limits real speedups because common GPUs and TPUs provide limited support for sparse neural networks.Consequently, the reported inference FLOPs are theoretical values.
  • K BROADER IMPACT: Using the technique could enable software with negative impacts such as privacy leakage and fairness issues, despite the paper’s scientific focus.The passage presents these as possible broader impacts rather than demonstrated outcomes.
Loading 2207.03620v3…