Source-linked AI summary
On the Integration of Self-Attention and Convolution
Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, Gao Huang
TL;DR
The paper addresses the apparent separation between convolution and self-attention by showing that both rely heavily on shared 1×1 projection computations. ACmix reuses these projections across convolution and self-attention paths, and experiments report improved performance with a practical efficiency caveat.
Problem
Convolution and self-attention are usually treated as distinct paradigms despite their potentially complementary properties and different design formulations.
Method
ACmix projects input feature maps once with 1×1 convolutions, then reuses the intermediate features for convolution-style and self-attention-style aggregation.
Results
The hybrid model consistently outperforms pure convolution or self-attention counterparts and demonstrates effectiveness and efficiency across image classification and object detection benchmarks.
Takeaways & Limitations
Sharing expensive projections provides an integration of convolution and self-attention with minimum computational overhead compared with pure counterparts.
Takeaways & Limitations
Tensor shifts in the convolution path can break data locality, hinder vectorized implementation, and impair practical inference efficiency.
Abstract
from arXiv · showhide
Convolution and self-attention are two powerful techniques for representation learning, and they are usually considered as two peer approaches that are distinct from each other. In this paper, we show that there exists a strong underlying relation between them, in the sense that the bulk of computations of these two paradigms are in fact done with the same operation. Specifically, we first show that a traditional convolution with kernel size k x k can be decomposed into k^2 individual 1x1 convolutions, followed by shift and summation operations. Then, we interpret the projections of queries, keys, and values in self-attention module as multiple 1x1 convolutions, followed by the computation of attention weights and aggregation of the values. Therefore, the first stage of both two modules comprises the similar operation. More importantly, the first stage contributes a dominant computation complexity (square of the channel size) comparing to the second stage. This observation naturally leads to an elegant integration of these two seemingly distinct paradigms, i.e., a mixed model that enjoys the benefit of both self-Attention and Convolution (ACmix), while having minimum computational overhead compared to the pure convolution or self-attention counterpart. Extensive experiments show that our model achieves consistently improved results over competitive baselines on image recognition and downstream tasks. Code and pre-trained models will be released at https://github.com/LeapLabTHU/ACmix and https://gitee.com/mindspore/models.
1. Introduction
Convolution and self-attention use different aggregation paradigms, yet their expensive projection computations share a common 1×1 convolution operation. ACmix exploits this relationship to combine both paradigms with minimal computational overhead and consistently improved empirical performance.
- Convolution and self-attention provide complementary properties, motivating approaches that integrate both modules.
- Both paradigms heavily rely on the same 1×1 convolution operations for projecting input feature maps.
- ACmix reuses projected intermediate features for self-attention and convolution aggregation paths, avoiding duplicate expensive projections.
- The paper reveals a shared computational structure between self-attention and convolution as a perspective for designing new learning paradigms.
- ACmix consistently outperforms pure convolution or self-attention counterparts across the reported empirical comparisons.
2. Related Work
Related work has explored self-attention as a convolutional augmentation, replacement, or complementary component, while transformer variants add convolution for inductive bias or efficiency. ACmix differs by sharing the projection computation between the two paths.
- Other approaches combine self-attention and convolution within a single block as complementary components.
- Earlier methods used self-attention to augment convolution modules or incorporated it as standalone blocks replacing traditional convolutions.
- Transformer-based variants add convolution for tokenization, reduced attention computation, convolutional stems, or additional inductive biases.
- ACmix combines the paradigms by sharing their 1×1 projection operations and reusing intermediate features for different aggregation paths.
3. Revisiting Convolution and Self-Attention
Convolution and self-attention can both be decomposed into an expensive projection stage followed by lightweight spatial aggregation. This shared structure explains how their operations and computational costs can be compared directly.
- 3.1. Convolution: Convolution is reformulated as separate stages: position-specific projections followed by feature shifts and summation.Each kernel position contributes a projected feature map that is shifted according to its spatial offset before aggregation.
- 3.1. Convolution: The first convolution stage is equivalent to multiple 1 × 1 convolutions, while shifting and aggregation in the second stage are lightweight.The projection uses the kernel weights associated with a particular position, whereas the subsequent operations combine spatially displaced maps.
- 3.2. Self-Attention: Self-attention similarly uses 1 × 1 convolutions to project inputs into queries, keys, and values before computing attention weights and aggregating values.The attention operation gathers features from a local region and concatenates outputs across N heads.
- 3.3. Computational Cost: Both modules concentrate their computational cost in Stage I, whose theoretical complexity is quadratic in channel size, whereas Stage II is linear or lightweight.For self-attention, Stage I preserves the training parameters; for convolution, Stage II requires no additional training parameters.
- 3.3. Computational Cost: In a ResNet50 analysis, Stage I accounts for 99% of convolution computation and 83% of self-attention computation.These measured proportions agree with the theoretical cost analysis across the two modules.
4. Method
ACmix exposes shared 1×1-convolution feature learning in convolution and self-attention, then reuses projected features for separate aggregation paths. Its shift-based convolution path is optimized with group or depthwise convolutions to reduce practical overhead.
- Shared decomposition: Both modules use 1×1 convolutions for feature projection, while their second stages perform different aggregation procedures.Stage I is feature learning; Stage II aggregates features according to convolutional or attention-based paradigms.
- Computational analysis: The shared projection stage has quadratic FLOPs and parameter complexity in channel size C, whereas aggregation is lightweight or nearly computation-free.This computational imbalance motivates sharing the projection operations.
- ACmix design: ACmix performs the shared projection once and reuses intermediate feature maps for self-attention and convolution aggregation.The mixed module combines both paths while avoiding duplicate projection work.
- Aggregation paths: The attention path groups projected features into query, key, and value triples, while the convolution path generates k^2 maps and shifts and aggregates them locally.Attention follows multi-head self-attention; convolution uses a light fully connected layer before shift-based aggregation.
- Efficient implementation: Tensor shifts can impair inference efficiency because they break data locality and are difficult to vectorize.The paper replaces inefficient shifts with fixed-kernel depthwise convolution and also describes group-convolution implementations.
- Efficient implementation: Learnable shift-initialized kernels improve capacity, while multiple convolution groups match the output-channel dimensions of the two paths.ACmix’s Stage II overhead remains comparatively minor because its added fully connected and group-convolution costs scale linearly with C.
5. Experiments
ACmix is evaluated across ImageNet classification, ADE20K segmentation, COCO detection, inference speed, and component ablations, showing improved accuracy or efficiency with modest overhead.
- ImageNet Classification: ACmix models consistently improve ImageNet classification results across ResNet, SAN, PVT, and Swin-Transformer backbones.ResNet-ACmix 26 matches SASA-ResNet 50 with 80% FLOPs; SAN-ACmix 15 outperforms SAN 19 with 80% FLOPs, while PVT-ACmix-T and Swin-ACmix-S use 40% and 60% of comparator FLOPs, respectively.
- Downstream Tasks: ACmix improves semantic segmentation under all evaluated ADE20K settings and consistently outperforms detection baselines with similar parameters or FLOPs on COCO.The downstream evaluations use Semantic-FPN, UperNet, RetinaNet, Mask R-CNN, and Cascade Mask R-CNN settings.
- Practical Inference Speed: ACmix achieves 1.3x fps with comparable mAP to PVT-S and 1.8x fps with 1.9mAP improvement over PVT-L on Ascend 910.FPS is evaluated on a single Ascend 910 with input image size (3, 576, 576).
- Ablation Study: The convolution and self-attention paths outperform either single path, while learned combination weights outperform fixed ratios by adapting path strengths across network positions.ACmix combines outputs as Fout = α · Fatt + β · Fconv.
- Path Analysis: Learned α and β values show relatively stable path preferences across parallel experiments, especially in deeper layers.The parameters reflect model bias toward convolution or self-attention at different depths.
6. Conclusion
The paper identifies shared projection computation between convolution and self-attention and uses it to construct a hybrid operator with shared heavy operations.
- Conclusion: ACmix reveals that convolution and self-attention share computation overhead when projecting input feature maps.This shared operation motivates integrating both paradigms within one operator.
- Conclusion: The proposed hybrid operator integrates self-attention and convolution by sharing their heavy operations.The paper reports effectiveness and efficiency on image classification and object detection benchmarks.
A. Model Architectures
The modified architectures replace original 3×3 convolution or self-attention modules with ACmix while preserving the listed baseline model families.
- Model Architectures: ACmix versions are constructed for ResNet, SAN, PVT, and Swin-Transformer architectures by substituting the original module.The listed baseline families include ResNet 26/38/50, SAN 10/15/19, PVT-T/S, and Swin-T/S.
B. Dataset and Training Setup
Experiments use ImageNet-1K pretraining or classification data, COCO detection data, and stated optimizer, schedule, hardware, and architecture-specific settings.
- Datasets and Training: ImageNet experiments use 1.28 million training images, 50,000 validation images, 1,000 classes, and architecture-specific training configurations.ResNet models train for 100 epochs with SGD on 8 GPUs; Transformer models follow their original configurations.
- Datasets and Training: COCO experiments use 80k training samples and 35k validation samples, with ImageNet-pretrained backbones and a 12-epoch 1x schedule.ResNet and SAN use SGD, while PVT and Swin-Transformer use AdamW.
- Datasets and Training: Segmentation models use AdamW with architecture-specific learning rates and schedules, including 80k or 160K total iterations and specified resizing procedures.Swin-Transformer uses a 6e-5 initial learning rate, 1,500 warmup iterations, and 512 × 512 training crops.
- Implementation Settings: Implementation settings include N = 4 for ResNet-ACmix, grouped values for SAN-ACmix, original configurations for PVT-ACmix and Swin-ACmix, and ka =7 and kc =3 by default.The SAN implementation divides values into four groups with channel dimension C/4.
D. Positional Encoding
The models use relative positional encoding when computing self-attention weights in ResNet-ACmix and Swin-ACmix, but omit it from the computation-complexity analysis.
- Relative positional encoding is adopted when computing attention weights in ResNet-ACmix and Swin-ACmix.
- The attention weights use query–key similarity, relative positional encoding, scaling by √d, and softmax over the neighborhood.
- Positional encoding is excluded from the complexity analysis because its cost is linear in channel dimension C and minor compared with feature projections.
E. Practical Costs for Other Models
Practical comparisons report FLOPs and parameters for convolution, self-attention, and ACmix across multiple model structures, with ACmix adding minimal cost.
- More than 60% of computation occurs in Stage I of self-attention for SAN and Swin models, similarly to ResNet 50.
- Across the reported model structures, ACmix introduces minimum computational cost while integrating convolution and self-attention.
- Table 8 compares practical FLOPs and parameters for different modules across models, reporting bracketed fractions of whole-module totals.
- Architecture tables compare models with and without ACmix modules for ResNet, SAN, PVT, and Swin backbones.