PaperlayerView exact revision on arXiv

Source-linked AI summary

ACSAN: All-Convolutional Self-Attention Network Integrating Convolutional Operations with Vision Transformer

Zimian Wei, Tianqi Zhang, Hengyue Pan

arXiv:2610.35375v1cs.CVcs.AI

TL;DR

ViTs perform well in vision but rely on large-scale training data and have substantial parameter and computation demands. ACSAN replaces ViT’s fully connected matrix multiplications with convolution-based self-attention, and the paper reports strong classification performance against baseline models when training data are insufficient.

  • Problem

    ViTs’ large numbers of trainable parameters contribute to reliance on large-scale training data and high training costs, limiting their fit for resource-constrained settings.

    Source: §1 · p. 3, §2.2 · p. 6

  • Method

    ACSAN uses DCSA, which generates attention inputs with multi-scale dilated convolutions, and replaces ViT MLP layers with convolution layers.

    Source: §1 · p. 4, §3.1 · p. 7, §3.1.1 · p. 7

  • Results

    ACSAN achieves excellent performance compared with baseline models when training data are insufficient.

    Source: §5 · p. 15

  • Takeaways & Limitations

    The reported results support ACSAN’s use for image classification under insufficient-training-data conditions.

    Source: §1 · p. 4, §5 · p. 15

    MDCU output channel count is set as a predefined multiple of its input channel count (c_o = r * c_i).

    Source: §3.1.1 · p. 8

Abstract

from arXiv · show

In the past few years, Vision Transformer (ViT) has achieved excellent performance in the field of computer vision. However, there still some problems remain. For instance, ViTs heavily rely on large-scale training data since they simply contain a large number of trainable parameters, which slows down their learning efficiency and limits their applicability in resource-constrained scenarios. In this paper, we propose a novel deep learning model, namely All-Convolutional Self-Attention Network (ACSAN), which applies convolution operations to replace fully-connected matrices multiplications in the ViT framework. To implement ACSAN, we firstly design Dynamic Convolutional Self-Attention (DCSA) as the basic module, which employs multi-scale dilated convolutions to generate Query, Key, and Value. We defined spatial self-attention and channel self-attention to generate attention features respectively. The two attention features are finally merged as the output of the DCSA module. Based on DCSA, we then construct the basic ACSAN block. Comparing with ViT, we use DCSA to replace the regular self-attention operations, and use convolution layers to replace MLPs. To evaluate the performance of the proposed method, we train ACSAN on three widely-used image classification databases, i.e. CIFAR-10, CIFAR-100 and ImageNet-1k, from scratch. Experimental results show that ACSAN achieves excellent classification accuracy comparing with standard ViT with comparable or fewer parameters, as well as other state-of-the-art methods. Ablation studies verify that each component contributes positively to the final performance.

1. Introduction

The introduction motivates convolutional self-attention as a way to combine local image features with attention-based dependencies while reducing ViTs’ reliance on training data and computationally costly operations. It presents DCSA and ACSAN and reports image-classification results across three databases, including settings with insufficient training samples.

1. Introduction

  • ViTs’ data demands, parameter-heavy self-attention and MLPs, limited local-feature capture, and matrix-multiplication costs motivate alternatives to standard ViT.The introduction notes that ViT requires large-scale pre-training for good ImageNet performance.Source: §1 · p. 3
  • Prior work combines convolution and attention or adds local structure to ViTs, motivating architectures that balance local feature extraction with attention.Examples include shifted-window and local-attention approaches, as well as ACmix, UniFormer, and DMFormer.Source: §1 · pp. 3–4
  • DCSA integrates convolution into self-attention, using multi-scale dilated convolutions to capture image features at multiple scales while retaining long-range dependency modeling.The proposed network also replaces ViT MLP layers with convolution layers.Source: §1 · p. 4
  • ACSAN builds on DCSA to extract local and global image features, with a convolution stem followed by staged DCSA blocks and downsampling.The architecture is shown in Figure 1.Source: §1 · p. 4, Figure 1 · p. 5
  • ACSAN is evaluated on CIFAR-10, CIFAR-100, and ImageNet-1k, with results indicating effectiveness when training samples are insufficient and higher training efficiency.The reported results are from experiments described in the introduction; no quantitative comparison is given here.Source: §1 · p. 4

2. Related Works

The related-work section contrasts CNNs’ local feature extraction with ViTs’ global patch modeling, data needs, and training costs. It then reviews approaches that combine convolution and attention to capture both local and global information.

2.1. Convolutional Neural Network

  • CNNs use local receptive fields, parameter sharing, and hierarchical feature extraction to capture local image features while reducing computation and storage costs.The section situates CNNs as a widely used computer-vision architecture.Source: §2.1 · p. 5

2.2. Vision Transformer

  • ViTs treat images as sequences and use self-attention to model global dependencies between image patches.With sufficient training data or large-scale pre-training, ViT-based models can outperform CNNs on many image tasks.Source: §2.2 · p. 5
  • With limited training data, ViT performance may decrease; its many trainable parameters also raise training costs and complicate use on resource-constrained platforms.Prior work seeks to reduce ViTs’ data requirements.Source: §2.2 · pp. 5–6
  • Convolution-attention hybrids combine local and global feature modeling through approaches including attention branches, MHRA variants, and multi-scale dilated convolutions.The reviewed examples include BAM, CBAM, SENet, ECA-Net, attention-augmented convolution, ACMix, UniFormer, and DMFormer.Source: §2.2 · p. 6

3. Methodology

The methodology builds ACSAN around DCSA, which combines convolution operations with self-attention. DCSA generates Q, K, and V with multi-scale dilated convolutions, computes spatial and channel attention, and fuses the resulting features.

3.1. DCSA Module

  • DCSA combines convolution with self-attention through a multi-scale dilated convolution unit, convolutional self-attention branches, and a feature-fusion unit.It is the fundamental computational module of ACSAN.Source: §3.1 · p. 7

3.1.1. Multi-scale Dilated Convolution Unit

  • A 1×1 convolution adjusts input channels, then multi-scale dilated convolutions generate Q, K, and V for self-attention.The paper uses kernel sizes 3, 5, and 7; the output channel count is set by the predefined expansion/compression rate.Source: §3.1.1 · pp. 7–8

3.1.2. Convolution Self-Attentions

  • Spatial attention captures dependencies between spatial locations, while channel attention captures dependencies between feature channels.Each branch produces an attention weight matrix that is applied to V and reshaped to the feature-map dimensions.Source: §3.1.2 · pp. 8–9

3.1.3. Feature Fusion Unit

  • The feature-fusion unit projects the channel- and spatial-attention outputs to a predefined channel count, then sums them for the DCSA output.The projections include convolution, batch normalization, and ReLU.Source: §3.1.3 · p. 9

3.1.4. Multi-Head DCSA

  • Multi-head DCSA splits the input by channels, processes each sub-feature independently, and concatenates the resulting CP and SP outputs along the channel dimension.The combined outputs are used to obtain the final multi-head DCSA output.Source: §3.1.4 · pp. 9–10

3.2. ACSAN Architecture

  • Each DCSA Block contains a DCSA module, an EqConvFC convolution layer, and batch-normalization layers, with residual connections.Residual connections are adopted to alleviate the gradient vanishing problem.Source: §3.2 · p. 10
  • ACSAN concatenates DCSA Blocks within stages without changing feature-map sizes and uses stride-2 3×3 convolutions between stages for downsampling and channel changes.A fully connected classification layer converts DCSA module outputs to class feature vectors.Source: §3.2 · p. 10

4. Experimental Results

Across CIFAR-10, CIFAR-100, and ImageNet, the experiments compare ACSAN with ViT and other baselines, with all models trained without pre-training. The reported results favor ACSAN’s classification performance and its combination of spatial and channel attention.

4.1. Databases and Implementation Details

  • The proposed ACSAN is evaluated on CIFAR-10, CIFAR-100, and ImageNet, and neither ACSAN nor baseline models use pre-training during training across all three databases.For CIFAR-10, baseline ViT models have similar numbers of trainable parameters to their ACSAN counterparts.Source: §4.1 · p. 12, §4.2 · p. 12

4.2. CIFAR-10 Experiments

  • On CIFAR-10, ACSAN models report better performance and lower computation complexity than ViT counterparts with similar parameter counts.Table 3 compares models trained from scratch without pre-training.Source: Table 3 · p. 12, §4.2 · p. 13

4.3. CIFAR-100 Experiments

  • On CIFAR-100, ACSAN is reported to handle very small per-class training samples without pre-training.Table 4 compares ACSAN and ViT models trained from scratch.Source: §4.3 · p. 13, Table 4 · p. 13

4.4. ImageNet Experiments

  • On ImageNet, ACSAN reports better classification accuracy than corresponding ViT models with fewer parameters and FLOPs, and a balance of computation and storage costs with performance against other baselines.The comparison uses models trained from scratch.Source: §4.4 · p. 13, Table 5 · p. 14

4.5. Ablation Studies

  • Ablations on CIFAR-10 indicate that combining channel and spatial self-attention benefits performance, while DCSA and the fully convolutional network improve classification when training data is limited.The compared variants include single-attention branches, removal of self-attention, depthwise separable convolutions, and DMFormer.Source: §4.5 · pp. 14–15, Table 6 · p. 14

5. Conclusion

The paper proposes ACSAN, an all-convolutional self-attention network built from DCSA modules and convolutional replacements for ViT MLP layers. Experiments report strong performance against baseline models when training data is limited.

  • ACSAN combines ViT self-attention with convolution operations through DCSA modules and replaces ViT MLP layers with convolution layers; experiments report strong baseline-comparison performance with limited training data.Source: §5 · p. 15

Related papers

Loading 2610.35375v1…