PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Attention Network

Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, Shi-Min Hu

arXiv:2202.09741v5cs.CV

TL;DR

Self-attention poses challenges for vision because it neglects image structure, has quadratic high-resolution costs, and lacks channel adaptability. The paper introduces linear large kernel attention (LKA) and the LKA-based Visual Attention Network (VAN), which outperform similar-sized ViTs and CNNs across varied vision tasks. The authors present VAN as a simple baseline and report continued opportunities for structural improvement and applications beyond vision.

  • Problem

    Self-attention for vision neglects 2D image structure, has quadratic complexity for high-resolution images, and ignores channel adaptability.

  • Method

    The paper proposes linear large kernel attention (LKA) and builds the Visual Attention Network (VAN) on it, combining convolutional and self-attention advantages.

  • Results

    VANs outperform similar-level ViTs and CNNs across image classification, detection, segmentation, and pose estimation tasks.

  • Takeaways & Limitations

    VAN provides a simple vision backbone that challenges treating self-attention as irreplaceable for visual tasks.

  • Takeaways & Limitations

    The paper evaluates VAN only on visual tasks and leaves its performance in other areas unexplored.

Abstract

from arXiv · show

While originally designed for natural language processing tasks, the self-attention mechanism has recently taken various computer vision areas by storm. However, the 2D nature of images brings three challenges for applying self-attention in computer vision. (1) Treating images as 1D sequences neglects their 2D structures. (2) The quadratic complexity is too expensive for high-resolution images. (3) It only captures spatial adaptability but ignores channel adaptability. In this paper, we propose a novel linear attention named large kernel attention (LKA) to enable self-adaptive and long-range correlations in self-attention while avoiding its shortcomings. Furthermore, we present a neural network based on LKA, namely Visual Attention Network (VAN). While extremely simple, VAN surpasses similar size vision transformers(ViTs) and convolutional neural networks(CNNs) in various tasks, including image classification, object detection, semantic segmentation, panoptic segmentation, pose estimation, etc. For example, VAN-B6 achieves 87.8% accuracy on ImageNet benchmark and set new state-of-the-art performance (58.2 PQ) for panoptic segmentation. Besides, VAN-B2 surpasses Swin-T 4% mIoU (50.1 vs. 46.1) for semantic segmentation on ADE20K benchmark, 2.6% AP (48.8 vs. 46.2) for object detection on COCO dataset. It provides a novel method and a simple yet strong baseline for the community. Code is available at https://github.com/Visual-Attention-Network.

1 INTRODUCTION

The paper identifies structural, computational, and channel-adaptability limitations in self-attention for vision, then proposes LKA and VAN to address them. VAN is reported to outperform similar-sized CNNs and ViTs across multiple vision tasks.

  • Self-attention treats images as 1D sequences, incurs quadratic cost on high-resolution images, and adapts spatially but not across channels.
  • Large kernel attention (LKA) is introduced as a linear attention mechanism tailored for visual tasks.
  • VAN is a vision backbone built on LKA and designed to combine convolution’s local structure with self-attention’s long-range dependence and adaptability.
  • VANs outperform similar-level vision transformers and CNNs in experiments spanning classification, detection, segmentation, and pose estimation.
  • ImageNet-1K results compare VAN with DeiT, PVT, Swin Transformer, ConvNeXt, and Focal Transformer using accuracy-parameters and accuracy-FLOPs trade-offs.

2 RELATED WORK

Computer vision research has progressed from CNNs and attention mechanisms toward self-attention networks that capture long-range dependencies and adapt to visual inputs. Related work also explores decomposing MLPs into spatial and channel components to reduce computational cost and parameters.

  • Convolutional Neural Networks: CNNs use local contextual information and translation invariance, becoming the mainstream computer-vision framework after AlexNet.
  • Attention Mechanisms: Attention mechanisms adaptively select information from input features and benefit classification, detection, and semantic-segmentation tasks.Computer-vision attention includes channel, spatial, temporal, and branch attention, as well as their combinations.
  • Self-Attention: Self-attention captures long-range dependence and input adaptability, leading vision transformers to achieve significantly better performance than traditional approaches.It originated in NLP and has become increasingly important in computer vision.
  • MLPs: MLP decomposition into spatial and channel MLPs reduces computational cost and parameters, helping unlock stronger computer-vision performance.

3 METHOD

The method decomposes large-kernel convolution into efficient spatial-local, spatial-long-range, and channel operations, then uses the resulting LKA module in a hierarchical VAN backbone.

  • Large Kernel Attention: Attention maps adaptively select discriminative features and suppress noisy responses based on the input feature.
  • Large Kernel Attention: Large-kernel convolution is decomposed into depth-wise local convolution, depth-wise dilated convolution, and 1×1 channel convolution to capture long-range relationships efficiently.The decomposition uses a spatial local convolution, a spatial long-range convolution, and a channel convolution.
  • Large Kernel Attention: LKA combines local context, a large receptive field, linear complexity, and spatial and channel adaptability without requiring sigmoid or softmax normalization.The attention map is produced from the decomposed convolution and applied through element-wise multiplication.
  • Complexity Analysis: For C = 32, the proposed decomposition uses fewer parameters than standard convolution and MobileNet decomposition for a 21×21 convolution.The reported reductions are 133× and 4.5× relative to the respective alternatives.
  • Visual Attention Network (VAN): VAN uses four hierarchical stages with decreasing spatial resolution and increasing channels, stacking normalization, convolutions, GELU, LKA, and feed-forward layers.Each stage downsamples first, then keeps spatial resolution and channel count fixed within the stage.
  • Large Kernel Attention: For a 21×21 kernel, the decomposition minimizes its parameter expression at dilation rate 3, using 5×5 depth-wise and 7×7 dilated depth-wise convolutions.

4 EXPERIMENTS

Experiments evaluate VAN across classification and broader vision tasks, using ablations and comparisons with CNNs, ViTs, and MLPs. VAN combines strong accuracy with a favorable throughput trade-off and competitive results across benchmarks.

  • Experimental settings: VAN experiments cover ImageNet classification, COCO detection and segmentation, ADE20K semantic segmentation, and additional visual tasks.The evaluation also includes throughput measurements and Grad-CAM visualizations.
  • Ablation study: Removing depth-wise convolution lowers ImageNet accuracy by 0.5% (74.9% vs. 75.4%), highlighting local structural information.
  • Ablation study: Removing depth-wise dilation convolution lowers ImageNet accuracy by 1.3% (74.1% vs. 75.4%), supporting the importance of long-range dependence.
  • Ablation study: Adding attention improves VAN-B0 by about 1.1% (74.3% vs. 75.4%), while channel modeling through 1 × 1 convolution improves it by 0.8% (74.6% vs. 75.4%).
  • Kernel-size ablation: A 21×21 decomposed convolution works better than a 7×7 one, while increasing it to 28×28 provides no obvious additional gain.
  • Comparison with existing methods: VAN outperforms similarly sized CNNs, ViTs, and MLPs, including ConvNeXt-T by 0.7% (82.8% vs. 82.1%), Swin-T by 1.5% (82.8% vs. 81.3%), and gMLP-S by 3.2% (82.8% vs. 79.6%).VAN also achieves 87.8% Top-1 accuracy with 200M parameters on ImageNet-22K-pretrained evaluation.
  • Efficiency: VAN achieves a better accuracy-throughput trade-off than Swin Transformer on RTX 3090 measurements.The comparison is presented through throughput results and an accuracy-throughput diagram.
  • Visualization: VAN-B2 clearly focuses on target objects in Grad-CAM visualizations and is compared with Swin-T and ConvNeXt-T.

4.2 Object Detection

VAN is evaluated for object detection and instance segmentation on COCO using standard detection frameworks and comparable training strategies. It surpasses CNN- and transformer-based backbones and reaches state-of-the-art performance across detection methods.

  • Settings: Experiments use COCO 2017 with MMDetection and matching training and validation strategies for fair comparison with Swin Transformer and PoolFormer.The evaluation includes RetinaNet, Mask R-CNN, Cascade Mask R-CNN, and Sparse R-CNN.
  • Results: VAN surpasses ResNet and PVT by a large margin under RetinaNet 1x and Mask R-CNN 1x settings.
  • Results: VAN achieves state-of-the-art performance with Mask R-CNN and Cascade Mask R-CNN while comparing favorably with Swin Transformer and ConvNeXt.

4.3 Semantic Segmentation

On ADE20K semantic segmentation, VAN-based backbones outperform CNN- and transformer-based alternatives at comparable parameter and FLOP budgets. The gains extend across multiple VAN variants and pretrained settings.

  • Results: VAN-based methods outperform ResNet, ResNeXt, PVT, PoolFormer, and PVTv2 using FPN at comparable parameters and FLOPs.
  • Results: VAN surpasses four PVTv2 variants by +1.3, +0.4, +1.5, and +0.8 mIoU for B0, B1, B2, and B3, respectively.
  • Results: With UperNet, VAN-B2 is +5.2 and +4.0 mIoU higher than ResNet-101 and Swin-T, respectively.
  • Results: ImageNet-22K-pretrained VAN models outperform Swin Transformer and ConvNeXt with less computational overhead.

4.4 Panoptic Segmentation

VAN is evaluated for COCO panoptic segmentation with Mask2Former and outperforms Swin Transformer across model sizes. VAN-B6 reaches a reported state-of-the-art panoptic quality score.

  • Settings: Panoptic segmentation experiments use the COCO panoptic dataset, Mask2Former, and ImageNet-1K or ImageNet-22K-pretrained backbones.
  • Results: VAN outperforms Swin Transformer for both large and small panoptic segmentation models.
  • Results: VAN-B2 exceeds Swin-T by +1.7 PQ on COCO panoptic segmentation.
  • Results: 58.2 PQ: VAN-B6 sets new state-of-the-art performance for panoptic segmentation.

4.5 Pose Estimation

The supplied results show VAN outperforming established backbones in pose estimation, fine-grain classification, and saliency detection under several benchmark settings.

  • Pose Estimation: Pose estimation experiments use COCO keypoints with SimpleBaseline as the decoder and MMPose as the implementation framework.The dataset contains 200K images and 17 keypoints, with training on COCO train 2017 and testing on COCO val 2017.
  • Pose Estimation: 74.9 AP lets VAN-B2 outperform Swin-T by 2.5 AP and PVT-S by 3.5 AP for 256 × 192 pose estimation.VAN-B2 also exceeds Swin-B by 2 AP at 256 × 192 and by 1.8 AP at 384 × 288, with less computation and parameters.
  • Fine-grain Classification: VAN-B4 achieves 91.3% Top-1 accuracy on CUB-200 fine-grain classification without a task-specific algorithm.This result exceeds DeiT and ViT-B on the stated benchmark.
  • Saliency Detection: VAN clearly surpasses ResNet and PVT backbones across the listed saliency detection datasets.The experiments replace the backbone in an EDN-based saliency detector and evaluate on common benchmarks including DUTS, DUT-O, and PASCAL-S.

5 DISCUSSION

The discussion argues that self-attention should not be treated as the default attention mechanism for vision. It presents LKA and VAN as evidence that a CNN-based alternative can surpass transformer-based methods on vision tasks.

  • 5 DISCUSSION: VAN surpasses state-of-the-art transformer-based methods for vision tasks, motivating renewed consideration of which attention mechanism best suits visual data.The discussion specifically frames self-attention as one special attention mechanism rather than an irreplaceable default.

6 FUTURE WORK

The authors identify structural refinement, large-scale self-supervised and transfer learning, and broader application areas as future directions for VAN.

  • Structure: The current VAN structure is described as intuitive, leaving potential improvements through kernel sizes, multi-scale designs, and multi-branch structures.The authors explicitly identify these as directions for continued structural improvement.
  • Self-supervised and Transfer Learning: Large-scale self-supervised learning and transfer learning are proposed as future applications for VAN.The authors connect VAN's use of image 2D structure and input-dependent output adjustment to these directions.
  • More Application Areas: The paper evaluates VAN in visual tasks but leaves performance in other areas, including NLP, for future exploration.The authors state that limited resources restricted the demonstrated application areas and express interest in VAN becoming a general model.

7 CONCLUSION

The paper introduces LKA and builds VAN upon it, reporting state-of-the-art performance in several visual tasks and directing future work toward further framework improvements.

  • 7 CONCLUSION: LKA combines convolution and self-attention advantages, while VAN achieves state-of-the-art performance in some visual tasks.The conclusion lists image classification, object detection, and semantic segmentation among the demonstrated tasks.
Loading 2202.09741v5…