Source-linked AI summary
Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition
Qibin Hou, Cheng-Ze Lu, Ming-Ming Cheng, Jiashi Feng
TL;DR
Visual recognition needs spatial representations that retain the global modeling advantages of self-attention without its high-resolution cost. Conv2Former replaces self-attention with convolutional modulation in a hierarchical ConvNet, and experiments report better performance than popular ConvNets and most Transformer-based models across classification, detection, and segmentation.
Problem
Self-attention models global pairwise dependencies but has quadratic computational complexity as image resolution increases, motivating more efficient convolution-based spatial encoding.
Method
Conv2Former replaces self-attention with convolutional modulation, using convolutional features as weights to modulate value representations through a Hadamard product.
Results
Conv2Former performs better than previous CNN-based models and most Transformer-based models on ImageNet classification, COCO object detection, and ADE20k semantic segmentation.
Takeaways & Limitations
Convolutional modulation provides a more efficient way to use large-kernel convolutions while retaining strong visual-recognition performance across multiple tasks.
Takeaways & Limitations
The study only designs CNN-based models; combining convolutional modulation with Transformers remains future work.
Abstract
from arXiv · showhide
This paper does not attempt to design a state-of-the-art method for visual recognition but investigates a more efficient way to make use of convolutions to encode spatial features. By comparing the design principles of the recent convolutional neural networks ConvNets) and Vision Transformers, we propose to simplify the self-attention by leveraging a convolutional modulation operation. We show that such a simple approach can better take advantage of the large kernels (>=7x7) nested in convolutional layers. We build a family of hierarchical ConvNets using the proposed convolutional modulation, termed Conv2Former. Our network is simple and easy to follow. Experiments show that our Conv2Former outperforms existent popular ConvNets and vision Transformers, like Swin Transformer and ConvNeXt in all ImageNet classification, COCO object detection and ADE20k semantic segmentation.
1. Introduction
Visual recognition progressed from convolutional models toward Vision Transformers, whose self-attention captures global dependencies but is costly at high resolution. Conv2Former simplifies this mechanism with convolutional modulation, producing a fully convolutional architecture that benefits from large kernels and performs strongly across major vision tasks.
- Motivation: Vision Transformers improved visual recognition by modeling global pairwise dependencies, but self-attention becomes computationally expensive for high-resolution images.Convolutions provide local connectivity, whereas self-attention encodes spatial information globally.
- Proposed approach: Convolutional modulation replaces self-attention's adaptive attention matrix with convolution-generated weights that reweigh value representations through a Hadamard product.The convolutional kernels are static, unlike the input-adaptive attention matrix.
- Proposed approach: Conv2Former is obtained by replacing Vision Transformer self-attention with convolutional modulation in a fully convolutional, Transformer-style network.Its computation increases linearly rather than quadratically with image resolution, making it suitable for downstream tasks.
- Large-kernel design: Conv2Former benefits from larger convolution kernels, including 11×11 and 21×21, unlike earlier findings that kernels beyond 9×9 added little performance.The paper contrasts its design with ConvNeXt and other large-kernel approaches.
- Evaluation: Conv2Former performs better than popular ConvNets such as ConvNeXt and EfficientNetV2 across ImageNet classification, COCO detection and instance segmentation, and ADE20k segmentation.The models were also pretrained on ImageNet-22k for downstream evaluation.
2. Related Work
Related work spans the evolution of ConvNets, Vision Transformers, large-kernel convolutions, and hybrid architectures. Conv2Former differs by reducing self-attention to convolutional modulation rather than decomposing large kernels or mixing separate Transformer and convolution modules.
- Convolutional networks: ConvNets evolved from early shallow models to deep residual and multi-path architectures that improved trainability and design flexibility.ResNet shortcut connections made training very deep models possible.
- Large-kernel convolutions: Large-kernel methods explore depthwise, dilated, sparse, or other convolutional designs to expand receptive fields and improve visual recognition.VAN decomposes large-kernel convolutions using depthwise and dilated convolutions.
- Conv2Former: Conv2Former differs from VAN by reducing self-attention to convolutional modulation rather than decomposing large-kernel convolutions.Its architecture is organized as a four-stage pyramid with multiple model variants.
- Vision Transformers: Vision Transformers, including ViT and DeiT, established Transformer architectures for image recognition and reduced reliance on large-scale data through augmentation and distillation.Pyramid designs such as Swin Transformer also connect this line of work to hierarchical visual models.
- Hybrid architectures: Hybrid models combine convolutions and Transformers or introduce attention mechanisms into ConvNets, especially for efficient network design.Examples include MobileViT, EfficientViT, EdgeNeXt, and MobileFormer.
3. Model Design
Conv2Former replaces self-attention with convolutional modulation in a hierarchical, Transformer-style ConvNet. Its design uses depthwise convolutions and large kernels to encode spatial information efficiently while adapting to input content.
- Overall architecture: Conv2Former uses a four-stage pyramid architecture with patch embedding blocks between stages to reduce feature-map resolution.The network includes five variants: Conv2Former-N, -T, -S, -B, and -L.
- Convolutional modulation block: Each Conv2Former block retains Transformer-like spatial encoding and channel mixing but replaces self-attention with a convolutional modulation layer.The block combines a spatial encoding component with an FFN for channel mixing.
- Self-attention: Self-attention forms an N×N similarity matrix, so its computational complexity grows quadratically with sequence length as image resolution increases.Here N = H × W, linking sequence length directly to spatial resolution.
- Convolutional modulation: Convolutional modulation applies depthwise-convolution features to value representations with a Hadamard product, producing content-adaptive local spatial aggregation.Each location interacts with pixels in a k × k region, while linear layers provide channel interaction.
- Advantages: Compared with self-attention, convolutional modulation is more memory-efficient for high-resolution images while remaining adaptive to input content unlike classic residual blocks.The method uses convolutions to build spatial relationships and modulation to adapt the block output.
- Micro design: 82.8 → 83.4 Top-1 accuracy for Conv2Former-T and 84.1 → 84.5 for Conv2Former-B as the kernel grows from 5 × 5 to 21 × 21.Because of model-efficiency considerations, the default kernel size is set to 11 × 11.
4. Experiments
Experiments evaluate Conv2Former across ImageNet classification, COCO detection and segmentation, ADE20k segmentation, and method ablations. The model consistently outperforms comparable ConvNets and Transformers, while convolutional modulation benefits from large kernels and Hadamard fusion.
- Experiment Setup: Conv2Former evaluates ImageNet-1k and ImageNet-22k classification, COCO detection and instance segmentation, and ADE20k semantic segmentation against numerous architectures.Compared methods include Swin Transformer, ConvNeXt, EfficientNetV2, DeiT, RegNet, FocalNet, CoAtNet, RepLKNet, and MOAT.
- ImageNet Classification: 1.1% and 1.7% gains over ConvNeXt-T and SwinT-T, respectively, are reported for tiny Conv2Former models on ImageNet-1k.Conv2Former-N matches SwinT-T with 15M parameters and 2.2G FLOPs versus 28M parameters and 4.5G FLOPs.
- ImageNet Classification: 84.4% top-1 accuracy is achieved by Conv2Former-B versus 84.3% for EfficientNet-B7, using 15G versus 37G computations.The paper reports that Conv2Former-B performs better despite EfficientNet-B7 using approximately twice the computations.
- ImageNet Classification: 87.7% is achieved by Conv2Former-L after ImageNet-22k pretraining and 384 × 384 finetuning, exceeding the reported hybrid-model results.Across similar model sizes, Conv2Former variants also outperform ConvNeXt; Conv2Former-B exceeds ConvNeXt-B and MOAT-2.
- Method Analysis: Increasing the kernel size from 5×5 to 21×21 consistently improves Conv2Former-T and Conv2Former-B performance, unlike ConvNeXt's reported saturation beyond 7×7.The performance gain appears to saturate only at 21×21 in this analysis.
- Method Analysis: Hadamard-product fusion performs better than element-wise summation and the other tested weighting strategies in the convolutional modulation block.Table 6 reports the simple Hadamard product as the best fusion strategy for Conv2Former-T.
- COCO Results: About 2% AP improvement over SwinT-T and ConvNeXt-T is reported for tiny Conv2Former-T with Mask R-CNN object detection.Instance segmentation gains exceed 1%, and Cascade Mask R-CNN gains exceed 1% over the same baselines.
- ADE20k Results: 1.3% mIoU improvement at tiny scale and 1.1% at base scale are reported over ConvNeXt on ADE20k, while Conv2Former-L reaches 54.3% mIoU.At all model sizes, the reported Conv2Former results outperform Swin Transformer and ConvNeXt.
5. Conclusions and Discussions
Conv2Former uses convolutional modulation to simplify self-attention with convolutions and Hadamard products, targeting more efficient use of large-kernel convolutions. Across classification, detection, and segmentation, it outperforms previous CNN-based models and most Transformer-based models, while remaining limited to CNN-based designs.
- Conclusions: Conv2Former simplifies self-attention through convolutional modulation using only convolutions and a Hadamard product.The operation is designed to use large-kernel convolutions more efficiently.
- Conclusions: Experiments show Conv2Former performs better than previous CNN-based models and most Transformer-based models across ImageNet classification, object detection, and semantic segmentation.
- Discussions: The discussion identifies large-kernel convolutions, fixed-sized kernels for large receptive fields, and lightweight attention mechanisms as open CNN design opportunities.
- Limitations: The study focuses on CNN-based models, leaving combinations of convolutional modulation blocks with Transformers for future work.
A.1. ImageNet-1k/22k
The ImageNet experiments use cosine learning-rate decay and specify stochastic-depth rates for different model variants and training stages. Additional settings describe differences from ConvNeXt and separate rates for pretraining and fine-tuning.
- Training settings: All experiments use cosine learning-rate decay for training.
- Training settings: Conv2Former training omits layerwise learning-rate decay and exponential moving average because they did not help during training.
- Training settings: Stochastic-depth rates for model variants and pretraining are reported in Table 10.
- Training settings: Stochastic-depth rates for ImageNet-22k pretraining and ImageNet-1k fine-tuning are reported in Table 11.
A.2. COCO Detection
COCO experiments follow the experimental settings used in prior work, with stochastic-depth rates provided separately for fine-tuning.
- COCO fine-tuning: COCO training follows the experimental settings of references [2], except for the stochastic-depth rates.The rates used for COCO fine-tuning are listed in Table 12.
A.3. ADE20k Semantic Segmentation
ADE20k semantic-segmentation experiments follow the settings used in reference [51], with stochastic-depth rates specified separately.
- ADE20k segmentation: ADE20k semantic-segmentation experiments follow the settings used in reference [51], except for the stochastic-depth rates summarized in Table 13.