PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN

Kin Wai Lau, Lai-Man Po, Yasar Abbas Ur Rehman

arXiv:2309.01439v3cs.CV

TL;DR

Large-kernel depth-wise convolutions in LKA can make VAN increasingly expensive as kernel size grows. The paper proposes LSKA, which replaces 2D kernels with cascaded horizontal and vertical 1D kernels, and reports comparable or better performance with lower complexity across vision and robustness evaluations.

  • Problem

    Increasing convolutional kernel size in LKA causes quadratic growth in computational and memory footprints, limiting efficient use of extremely large kernels.

  • Method

    LSKA decomposes the 2D kernels in LKA's depth-wise and dilated depth-wise convolutions into cascaded horizontal and vertical 1D separable kernels.

  • Results

    LSKA-based VAN achieves comparable or better performance than LKA while reducing computational complexity and outperforming ViTs and ConvNeXt across vision tasks and robustness tests.

  • Takeaways & Limitations

    LSKA enables VAN to use larger kernels with lower computational and memory footprints while retaining strong task performance and robustness.

Abstract

from arXiv · show

Visual Attention Networks (VAN) with Large Kernel Attention (LKA) modules have been shown to provide remarkable performance, that surpasses Vision Transformers (ViTs), on a range of vision-based tasks. However, the depth-wise convolutional layer in these LKA modules incurs a quadratic increase in the computational and memory footprints with increasing convolutional kernel size. To mitigate these problems and to enable the use of extremely large convolutional kernels in the attention modules of VAN, we propose a family of Large Separable Kernel Attention modules, termed LSKA. LSKA decomposes the 2D convolutional kernel of the depth-wise convolutional layer into cascaded horizontal and vertical 1-D kernels. In contrast to the standard LKA design, the proposed decomposition enables the direct use of the depth-wise convolutional layer with large kernels in the attention module, without requiring any extra blocks. We demonstrate that the proposed LSKA module in VAN can achieve comparable performance with the standard LKA module and incur lower computational complexity and memory footprints. We also find that the proposed LSKA design biases the VAN more toward the shape of the object than the texture with increasing kernel size. Additionally, we benchmark the robustness of the LKA and LSKA in VAN, ViTs, and the recent ConvNeXt on the five corrupted versions of the ImageNet dataset that are largely unexplored in the previous works. Our extensive experimental results show that the proposed LSKA module in VAN provides a significant reduction in computational complexity and memory footprints with increasing kernel size while outperforming ViTs, ConvNeXt, and providing similar performance compared to the LKA module in VAN on object recognition, object detection, semantic segmentation, and robustness tests.

I. INTRODUCTION

The paper revisits large-kernel attention in CNNs because increasing kernel sizes create costly computation and memory demands. It proposes LSKA, a separable-kernel alternative intended to retain performance while improving efficiency and supporting larger kernels.

  • ViTs improve vision performance through long-range dependency modeling but incur quadratic computational and memory costs.
  • Large receptive fields can improve CNN attention, yet directly increasing convolutional kernel size is computationally and memory intensive.
  • LSKA replaces a k × k depth-wise kernel with cascaded 1 × k and k × 1 separable kernels.
  • The authors evaluate LSKA against LKA, ViTs, ConvNeXt, and other baselines across recognition, detection, segmentation, and robustness tasks.
  • The proposed design reduces parameter growth with kernel size, maintains performance, and supports large-kernel benefits at the same inference-time cost as small-kernel LKA.

B. Attention Mechanisms with Large Kernel

Large Kernel Attention combines CNN locality with long-range and spatially adaptive processing, but large depth-wise kernels remain expensive. LSKA restructures these kernels into cascaded one-dimensional operations to reduce complexity while preserving performance.

  • Attention mechanisms: VAN combines CNN locality with LKA-based long-range dependence and spatial adaptability, achieving strong results across major vision tasks.
  • LKA formulation: The trivial LKA design uses a large 2D depth-wise convolution, whose computational cost grows quadratically with kernel size.
  • LSKA formulation: LSKA splits the 2D kernels of depth-wise and dilated depth-wise convolutions into cascaded horizontal and vertical 1D kernels.
  • LKA formulation: Original LKA decomposes the large kernel into a small depth-wise convolution followed by a dilated depth-wise convolution and a 1 × 1 convolution.
  • Complexity: LKA still incurs high computational complexity and memory footprints beyond 23 × 23 kernels, whereas LSKA-trivial and LSKA reduce complexity relative to their LKA counterparts.

B. Large Separable Kernel with Attention

LSKA replaces the 2D kernels in LKA with cascaded 1D separable kernels, preserving comparable performance while reducing computational complexity.

  • B. Large Separable Kernel with Attention: LSKA splits LKA's 2D depthwise kernels into two cascaded 1D separable kernels.The modified configuration applies this decomposition to depthwise and depthwise-dilated convolution.
  • B. Large Separable Kernel with Attention: LSKA and LSKA-trivial significantly reduce VAN's computational complexity compared with their LKA counterparts.
  • B. Large Separable Kernel with Attention: Table I compares convolution, self-attention, LKA, and LSKA across their properties, with n denoting the number of pixels.

C. Properties of LSKA

LSKA preserves adaptive attention while improving long-range dependence and scalability to extremely large kernels. Increasing kernel size enlarges the effective receptive field and supports continued accuracy growth where LKA saturates.

  • Long-range dependence: LSKA captures long-range dependence as its kernel size increases from 7 to 65, with darker ERF regions spreading across the heatmaps.The ERF analysis uses darker heatmap regions as an indicator of larger receptive fields.
  • Spatial and channel adaptability: LSKA inherits spatial and channel adaptability from LKA while using cascaded horizontal and vertical kernels to reduce memory and computation.
  • Scalability to the extreme large kernel: LSKA reduces computational cost while maintaining a comparatively constant parameter count as kernel size grows, unlike LKA and SLaK.
  • Scalability to the extreme large kernel: LSKA-Base accuracy continues growing from kernel sizes 23 to 53, whereas LKA-Base saturates beyond kernel size 23.

D. Complexity Analysis of LSKA

The complexity analysis derives FLOPs and parameter counts for LSKA and related attention modules under equal feature-map dimensions. Kernel decomposition reduces depthwise parameters and FLOPs relative to LKA.

  • D. Complexity Analysis of LSKA: The analysis computes FLOPs and parameters assuming identical input and output feature-map dimensions H × W × C.Bias terms are ignored to simplify the calculations.
  • D. Complexity Analysis of LSKA: LSKA saves (2d−1)/2 parameters in the original LKA depthwise convolutional layer.
  • D. Complexity Analysis of LSKA: The FLOP saving matches the parameter saving, and LSKA is computationally more effective than LSKA-trivial.

E. Model Architecture

VAN-LSKA follows the VAN architecture while replacing LKA with LSKA in convolution blocks. The model uses four stages with specified stride, kernel, channel, expansion, and block-count hyperparameters.

  • Model Architecture: The model hyperparameters specify stage-wise stride, kernel size, output channels, CFFN expansion ratio, and block count.These are denoted by Si, Ki, Ci, Ei, and Li, respectively.
  • Model Architecture: VAN-LSKA follows VAN and replaces its LKA module with LSKA in the convolution block.Each block contains batch normalization, LSKA, and a convolutional feed-forward network.
  • Model Architecture: The architecture begins with a 7 × 7 stride-4 input convolution, then uses four stages with stride-2 downsampling before stages 2–4.The stem reduces input resolution by four and produces 32 or 64 channels depending on model capacity.
  • Model Architecture: VAN-LSKA-Tiny, Small, and Base use the same numbers of convolution blocks, channels, and CFFN expansion ratios as corresponding VAN-LKA models.This supports a fair comparison between the two attention modules.

IV. EXPERIMENT

The experiments compare VAN with LSKA against VAN with LKA across classification and downstream vision tasks using matched evaluation settings. LSKA reduces parameters and computation while retaining comparable performance.

  • Experimental Setup: The study compares VAN-LSKA and VAN-LKA on image classification, object detection, semantic segmentation, and robustness evaluation.The robustness benchmarking additionally considers perturbation datasets not evaluated in the seminal VAN work.
  • Image Classification: ImageNet-1K classification reports validation top-1 accuracy, parameter count, and inference speed under matched training settings.Testing uses 224×224 center crops, batch size 64, and a 1080Ti GPU for speed measurement.
  • Image Classification: VAN-LSKA-Tiny saves 2.4% of parameters and 11.1% of computational FLOPs while achieving slightly better performance than VAN-LKA-Tiny.For Small, savings are 1.4% in parameters and 4% in FLOPs; for Base, 1.5% and 2%, respectively, with unchanged accuracy.
  • Object Detection: On COCO object detection and instance segmentation, VAN-LSKA backbones use fewer parameters and FLOPs with comparatively similar performance to LKA.With Mask R-CNN Base, LSKA saves 0.3 million parameters and 2 GFLOPs with a 0.3% decrease, achieving 46.3 versus 46.6.

C. Semantic Segmentation

Semantic segmentation and robustness experiments show that LSKA preserves LKA-like performance while reducing computational cost. Larger kernels can improve receptive-field benefits, although performance eventually saturates in smaller models.

  • C. Semantic Segmentation: On ADE20K with Semantic FPN, VAN-LSKA-Base saves 0.2M parameters and reduces GFLOPs by 1.05% while slightly improving performance over VAN-LKA-Base.Tiny and Small also save 0.1M parameters with comparable performance and GFLOPs reductions of 0.49% and 0.36%.
  • Robustness Evaluation: Robustness evaluation covers ImageNet-C, ImageNet-¯C, ImageNet-A, ImageNet-R, and ImageNet-SK using top-1 accuracy, mCE, corruption error, and retention rate.Lower values are better for mCE and corruption error, while higher values are better for retention rate.
  • Robustness Evaluation: Across parameter regimes, VAN-LSKA differs from VAN-LKA by only 0.1%–0.6% while saving 2.5%, 1.4%, and 1.5% parameters for Tiny, Small, and Base.Computational time is reduced by 11.1%, 4%, and 2% for the same capacities.
  • Kernel Size Effects: For Base models, LSKA performance improves from kernel size 23 to 53, whereas LKA saturates beyond kernel size 23.In Tiny models, LSKA performance slightly decreases and begins saturating from kernel sizes 23 to 65.

B. Varying the kernel size of LSKA and LKA on various downstream tasks

Across downstream tasks, increasing kernel size generally improves performance through 35 × 35, while larger kernels can improve robustness and shift representations toward shape rather than texture.

  • Downstream Task Evaluation: Increasing VAN-LSKA-Tiny’s kernel from 7 × 7 to 35 × 35 improves Mask R-CNN mean average precision by 1.6%.Semantic segmentation and common-corruption classification show similar trends.
  • Shape–Texture Analysis: Larger-kernel VAN models provide better performance on corrupted data, and the authors conjecture that increasing kernel size biases representations toward object shape over texture.The paper links this interpretation to prior observations that shape-based representations benefit downstream tasks.
  • Shape–Texture Analysis: The shape–texture analysis estimates retained semantic information through mutual information in latent representations using texture-shape cue conflict images.The dataset contains 1,280 style-transfer-generated images sharing semantic properties such as shape or texture.
  • Shape–Texture Analysis: Texture classification uses a KNN classifier on VAN-LKA-Tiny and VAN-LSKA-Tiny backbones with K = 1 and reports validation top-1 accuracy on DTD.DTD contains 5,640 images across 47 categories.

VI. COMPARSION WITH STATE-OF-THE-ART METHODS

VAN-LSKA is compared with state-of-the-art CNNs and ViTs across classification, detection, segmentation, and robustness. It matches VAN-LKA while using fewer parameters and GFLOPs, and reports stronger results than several competing architectures.

  • Comparison setup: VAN-LSKA is evaluated against CNNs and ViTs on ImageNet classification, ImageNet-C robustness, COCO detection, and ADE20K segmentation.The comparison includes ConvNeXt, RepLKNet, SLaK, VAN-LKA, DeiT, PVT, PVTv2, and Swin Transformers.
  • Results: VAN-LSKA outperformed ViTs across image classification, object detection, semantic segmentation, and robustness evaluation.This result is reported for the Tiny, Small, and Base network versions.
  • Results: VAN-LSKA surpassed Swin-T by 1.5% classification accuracy while saving 7.4% parameters.The reported gaps also include 6.4% on robustness evaluation and 2.2% on object detection.
  • Results: VAN-LSKA surpassed ConvNeXt by 0.7% on ImageNet classification and 2.3% on ImageNet-C while saving 8.4% parameters.It also exceeded RepLKNet and SLaK by 0.3% on ImageNet classification and improved ImageNet-C by 2.2% and 1.4%, respectively.
  • Interpretation: The reported advantage is associated with LSKA enabling a large receptive field and a higher shape-to-texture representation bias.For kernel size 35×35, VAN-LSKA is reported to have higher shape and lower texture dimensionality than the compared models.
  • Results: VAN-LSKA achieved comparable performance to VAN-LKA on classification, robustness, detection, and segmentation with lower parameter size and GFLOPs.The VAN-LSKA and VAN-LKA kernel size is set to k = 35 for the reported benchmark tasks.

VII. BENCHMARKING ROBUSTNESS EVALUATION ON VITS AND CNNS

The paper benchmarks VAN-LSKA against ViTs and CNNs on corruption and out-of-distribution datasets. VAN-LSKA generally outperforms these baselines while matching VAN-LKA with fewer parameters and lower computational cost.

  • Evaluation setup: Robustness is evaluated on ImageNet-C, ImageNet-A, ImageNet-R, and ImageNet-Sketch using pretrained ImageNet1k models.The comparison includes transformer-based and CNN-based baselines.
  • Results: VAN-LSKA outperformed all compared transformer-based models across the evaluated robustness and out-of-distribution datasets.The reported datasets are ImageNet-C, ImageNet-A, ImageNet-R, and ImageNet-SKetch.
  • Results: VAN-LSKA performed better than ConvNeXt, RepLKNet, and SLaK on most datasets while saving 8.4%, 11.7%, and 12.6% parameters, respectively.The parameter savings are reported relative to those three CNN baselines.
  • Results: Compared with VAN-LKA, VAN-LSKA achieved comparable robustness performance with lower parameter sizes and GFLOPs.This preserves the reported performance level while reducing model and computational footprints.
  • Interpretation: The conclusion characterizes LSKA as a robust learner compared with previous large-kernel CNNs and ViTs.The paper links increased shape information and reduced texture information to robustness through a reported correlation.

APPENDIX

The appendix describes procedures for estimating shape and texture dimensionality and comparing effective receptive fields and kernel-size variants. These analyses relate representation content and receptive-field behavior to LSKA design choices.

  • Shape and texture dimensionality: The shape–texture analysis samples image pairs sharing shape or texture, then estimates mutual information between corresponding model representations.The representations are produced by an ImageNet1k-pretrained model E.
  • Shape and texture dimensionality: Under a bivariate-normal assumption, mutual information is estimated from the correlation coefficient between neurons in paired latent representations.The procedure follows the methodology cited in the paper.
  • Shape and texture dimensionality: Shape and texture scores are formed by summing correlation coefficients, then softmax-normalized across shape, texture, and residual factors.The normalized factor dimensionalities sum to the latent representation dimension N.
  • Kernel-size comparisons: The appendix compares LKA-trivial, LKA, LSKA-trivial, and LSKA across different kernel sizes on Tiny ImageNet classification models.This comparison is presented as a kernel-size study.
  • Effective receptive field: LSKA and LKA show effective receptive-field saturation beyond kernel sizes 35, 53, and 65.The appendix reports similar saturation behavior for the two modules.
Loading 2309.01439v3…