Source-linked AI summary
CMT: Convolutional Neural Networks Meet Vision Transformers
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, Chang Xu
TL;DR
Vision transformers capture long-range dependencies but remained behind CNNs in performance and computational cost. CMT addresses this gap with a hybrid architecture that combines transformer global modeling with CNN local-feature extraction and scales it into a model family. The resulting CMT models achieve stronger accuracy and efficiency across ImageNet and downstream vision tasks, including 83.5% ImageNet top-1 accuracy for CMT-S.
Problem
Vision transformers still lagged similar CNNs in performance and computational cost, despite their ability to capture long-range dependencies.
Method
CMT combines a convolution stem and convolution-enhanced transformer blocks, uses staged multi-scale processing, and applies scaling to create variants for different resource constraints.
Results
CMT achieves better accuracy with fewer parameters and FLOPs than convolutional and transformer counterparts, including 83.5% top-1 accuracy on ImageNet for CMT-S.
Takeaways & Limitations
CMT provides a family of visual recognition models that combines local and global information and generalizes to object detection and instance segmentation.
Abstract
from arXiv · showhide
Vision transformers have been successfully applied to image recognition tasks due to their ability to capture long-range dependencies within an image. However, there are still gaps in both performance and computational cost between transformers and existing convolutional neural networks (CNNs). In this paper, we aim to address this issue and develop a network that can outperform not only the canonical transformers, but also the high-performance convolutional models. We propose a new transformer based hybrid network by taking advantage of transformers to capture long-range dependencies, and of CNNs to model local features. Furthermore, we scale it to obtain a family of models, called CMTs, obtaining much better accuracy and efficiency than previous convolution and transformer based models. In particular, our CMT-S achieves 83.5% top-1 accuracy on ImageNet, while being 14x and 2x smaller on FLOPs than the existing DeiT and EfficientNet, respectively. The proposed CMT-S also generalizes well on CIFAR10 (99.2%), CIFAR100 (91.7%), Flowers (98.7%), and other challenging vision datasets such as COCO (44.3% mAP), with considerably less computational cost.
1. Introduction
CMT combines convolutional and transformer components to capture local and global visual information while addressing limitations of pure vision transformers. Its staged architecture preserves higher-resolution features, reduces computational burden, and scales into variants for different resource constraints.
- Pure vision transformers can underperform similar-sized CNNs because patch tokenization overlooks image-specific 2D structure and spatial locality.Transformers capture long-range dependencies between patches, but the patch sequence can ignore local information important for vision.
- CMT combines a convolution stem with transformer-based CMT blocks to extract fine-grained local features and learn global representations.The CMT block enhances local information with depth-wise convolution while retaining transformer-based representation learning.
- Figure 1 compares CMT with other models using ImageNet top-1 accuracy and COCO val2017 RetinaNet object-detection results under single-scale 1x training.The figure separates classification performance from downstream detection performance across different backbones.
- CMT maintains higher-resolution first-stage features at H/4×W/4 versus H/16×W/16 in ViT, supporting dense prediction tasks.Its stage-wise design then reduces resolution and sequence length while increasing feature dimension, helping alleviate computation from high resolution.
- CMT uses four stride-2 convolutional layers to build multi-scale features and flexibly increase feature dimensions across stages.The architecture also includes local perception and inverted residual feed-forward components to capture local and global structure.
2. Related Work
CNNs established strong visual recognition through convolutional feature extraction, while transformers brought long-range dependency modeling from NLP into vision. Related work therefore motivates combining transformer global modeling with CNN locality because vision tasks require both.
- Convolutional neural networks: Small convolutional kernels have limited receptive fields, making global information difficult to obtain for classification, detection, and segmentation.This limitation motivated investigation of self-attention-based transformers for long-range information.
- Convolutional neural networks: CNNs achieved major success in computer vision by stacking convolutional and pooling layers to extract discriminative visual features.Subsequent architectures explored multi-path blocks, stronger generalization, and other architectural improvements.
- Convolutional neural networks: Efficient CNN research traded accuracy and efficiency through architectures such as MobileNets and EfficientNets, which used neural architecture search for mobile-size networks.These models were reported as new state-of-the-art efficient networks.
- Vision transformers: ViT introduced a pure transformer for image classification by using image patches as input, while DeiT enabled data-efficient training directly on ImageNet-1K.ViT initially required the large private JFT-300M dataset for promising results.
- Vision transformers: Vision transformers model long-range dependencies effectively but had not matched conventional CNNs, while visual perception also requires locality.CMT follows CNN-inspired architecture while combining transformer-based modeling with convolutional layers.
3. Approach
CMT combines convolutional local feature extraction with transformer-based global modeling in a hierarchical architecture. Its block design and compound scaling strategy target stronger representation ability and more efficient processing than standard transformer blocks.
- Architecture: CMT-S begins with a convolution stem and uses four stages to generate hierarchical feature maps at different scales.The stem uses three 3 × 3 convolutions, while stage transitions reduce resolution progressively for dense prediction tasks.
- CMT Block: The CMT block combines a local perception unit, lightweight multi-head self-attention, and an inverted residual feed-forward network.These components respectively support local information extraction, efficient global interaction, and convolution-enhanced feed-forward processing.
- CMT Block: Lightweight attention reduces the spatial size of keys and values with stride-k depth-wise convolution before attention computation.A relative position bias is also added, and its learned values can be transferred to different spatial sizes through bicubic interpolation.
- Complexity Analysis: CMT blocks are more computationally efficient than standard transformer blocks and better suited to processing higher-resolution feature maps.The comparison is made under the stated reduction ratio k ≥1 in lightweight multi-head self-attention.
- Scaling Strategy: The compound scaling strategy jointly scales depth, feature dimensions, and input resolution using a coefficient φ and constants α, β, and γ.The constraint α·β^1.5·γ^2 ≈2.5 is intended to balance computational-cost growth with performance gains; four CMT variants are introduced.
4. Experiments
Experiments evaluate CMT across ImageNet classification, COCO detection and segmentation, and transfer-learning datasets, including ablations of architecture, normalization, and scaling. Results show that CMT variants achieve strong accuracy and computational efficiency across these settings.
- ImageNet Classification: CMT variants consistently outperform comparable CNNs and transformers on ImageNet with lower computational cost.Table 2 groups models with similar accuracy and reports the proposed CMTs as consistently more efficient.
- CMT Block Ablation: Replacing all batch-normalization layers with layer normalization reduces CMT-S performance to 83.0%, while replacing all layer normalization prevents convergence.The ablation indicates that normalization functions must be applied selectively within the CMT block.
- Scaling Strategy: 83.4% versus 83.8% shows that depth-only scaling underperforms the proposed compound scaling strategy on ImageNet.The comparison is reported for the scaled network against the original CMT-S.
- Instance Segmentation: For Mask R-CNN instance segmentation, CMT-S surpasses Twins-PCPVT-S by 1.7% AP and Twins-SVT-S by 1.9% AP.CMT-S based RetinaNet and Mask R-CNN reach 14.8 FPS and 11.2 FPS, respectively, on COCO val2017.
- Transfer Learning: CMT-S outperforms other transformer-based models on five transfer-learning datasets with less FLOPs and matches EfficientNet-B7 performance using 9x fewer FLOPs.The evaluated datasets are CIFAR10, CIFAR100, Stanford Cars, Flowers, and Oxford-IIIT Pets.
5. Conclusion
CMT is a hybrid architecture for visual recognition and downstream vision tasks that combines CNNs and transformers to capture local and global information. The paper also introduces scaled CMT variants for different resource constraints and reports strong experimental results.
- CMT combines CNNs and transformers to capture local and global information for visual recognition and downstream computer vision tasks.The paper specifically mentions object detection and instance segmentation.
- A scaling strategy generates a family of CMT variants for different resource constraints.
- Extensive experiments on ImageNet and other downstream vision tasks demonstrate the effectiveness and superiority of CMT architectures.