Source-linked AI summary
ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions
Chunlong Xia, Xinliang Wang, Feng Lv, Xin Hao, Yifeng Shi
TL;DR
Dense prediction exposes ViT’s limited local interaction and feature-scale diversity, while vision-specific redesigns can require additional pre-training. ViT-CoMer adds multi-scale CNN features and bidirectional CNN-Transformer fusion to a plain ViT, achieving 64.3% AP on COCO val2017 without extra detection data and 62.1% mIoU on ADE20K val.
Problem
ViT lacks inner-patch information interaction and diverse feature scales for dense prediction, while vision-specific redesigns introduce additional pre-training costs.
Method
ViT-CoMer combines a plain ViT with an MRFP CNN branch and CTI bidirectional fusion to integrate multi-scale spatial and hierarchical features.
Results
64.3% AP is achieved on COCO val2017 without extra detection data, while 62.1% mIoU is achieved on ADE20K val.
Takeaways & Limitations
ViT-CoMer provides a feature-enhanced dense prediction backbone that can use advanced ViT pre-trained weights and achieves comparable or superior performance to state-of-the-art backbones.
Abstract
from arXiv · showhide
Although Vision Transformer (ViT) has achieved significant success in computer vision, it does not perform well in dense prediction tasks due to the lack of inner-patch information interaction and the limited diversity of feature scale. Most existing studies are devoted to designing vision-specific transformers to solve the above problems, which introduce additional pre-training costs. Therefore, we present a plain, pre-training-free, and feature-enhanced ViT backbone with Convolutional Multi-scale feature interaction, named ViT-CoMer, which facilitates bidirectional interaction between CNN and transformer. Compared to the state-of-the-art, ViT-CoMer has the following advantages: (1) We inject spatial pyramid multi-receptive field convolutional features into the ViT architecture, which effectively alleviates the problems of limited local information interaction and single-feature representation in ViT. (2) We propose a simple and efficient CNN-Transformer bidirectional fusion interaction module that performs multi-scale fusion across hierarchical features, which is beneficial for handling dense prediction tasks. (3) We evaluate the performance of ViT-CoMer across various dense prediction tasks, different frameworks, and multiple advanced pre-training. Notably, our ViT-CoMer-L achieves 64.3% AP on COCO val2017 without extra training data, and 62.1% mIoU on ADE20K val, both of which are comparable to state-of-the-art methods. We hope ViT-CoMer can serve as a new backbone for dense prediction tasks to facilitate future research. The code will be released at https://github.com/Traffic-X/ViT-CoMer.
1. Introduction
ViT-CoMer combines a plain ViT with CNN features to address limited local interaction and single-scale representation while retaining access to open-source pre-trained ViT weights. It is evaluated across dense prediction tasks, achieving strong detection and segmentation results.
- Backbone paradigms: The adapted backbone paradigm combines CNN and Transformer features to support multi-scale dense prediction while retaining advanced pre-training.This contrasts with vision-specific backbones that require retraining after structural modifications.
- Motivation and approach: ViT-CoMer combines a plain ViT with CNN features and directly loads open-source advanced pre-trained weights.The ViT module uses open-source pre-training, while the remaining components use random initialization.
- Core contributions: MRFP supplies multi-receptive-field spatial features, while CTI fuses CNN and Transformer features across hierarchical scales.Together, the modules address limited local interaction and single-scale representation in plain ViT features.
- Experimental results: 64.3% AP is achieved by ViT-CoMer-L on COCO val2017 without extra detection training data.The result uses advanced open-source pre-training and is reported as the best record under that condition.
- Experimental results: 62.1% mIoU is achieved by ViT-CoMer-L on ADE20K val, comparable to state-of-the-art methods.The evaluation covers object detection, instance segmentation, and semantic segmentation benchmarks.
2. Related Work
Related work addresses ViT’s limited feature hierarchy and local interaction through vision-specific backbones, adapters, and feature-interaction designs. ViT-CoMer instead adds CNN-based multi-scale features and bidirectional fusion while preserving the plain ViT backbone.
- Plain backbones: Plain ViT-based dense prediction can suffer from insufficiently rich feature representations and requires strong multi-scale perception.ViTDet uses a simple feature pyramid, but its performance remains below state-of-the-art methods according to the cited discussion.
- Vision-specific backbones: Vision-specific backbones use mechanisms such as shifted windows and feature pyramids to address local interaction and single-scale limitations.Examples include Swin-Transformer and PVT, while MixFormer uses bidirectional convolution and self-attention interaction.
- Adapted backbones: ViT-Adapter integrates spatial priors and leverages ViT pre-trained weights but requires full fine-tuning and lacks feature interaction among spatial priors.VPT reduces the number of trainable parameters but is reported as less effective than full fine-tuning.
- Adapted backbones: ViT-CoMer enhances spatial hierarchical features by fusing CNN and Transformer representations while using full fine-tuning.The approach is intended to improve dense prediction performance without redesigning the ViT architecture.
3. The ViT-CoMer Method
ViT-CoMer combines a plain ViT with multi-scale convolutional features through MRFP and bidirectional CNN-Transformer fusion. These modules enrich receptive fields, align multi-scale representations, and repeatedly exchange information between branches.
- 3.1. Overall Architecture: ViT-CoMer uses a plain ViT branch alongside CNN features, preserving compatibility with open-source ViT pre-trained weights.
- 3.2. Multi-Receptive Field Feature Pyramid: MRFP processes multi-scale C3, C4, and C5 features with grouped depth-wise convolutions using different kernel sizes before restoring their dimensions.The feature pyramid supplies multi-scale information, while varied kernels expand receptive fields.
- 3.3. CNN-Transformer Bidirectional Fusion Interaction: CTI adds the ViT feature X to the middle-scale CNN feature F4, then applies multi-scale self-attention to unify the two modalities.The fused features span 1/8, 1/16, and 1/32 resolutions.
- 3.3. CNN-Transformer Bidirectional Fusion Interaction: The updated ViT feature is formed by adding a learnably scaled CNN output, with α initialized to zero to limit early influence from the randomly initialized CNN branch.
- 3.3. CNN-Transformer Bidirectional Fusion Interaction: CTI repeats the fusion process to inject updated features into the CNN branch, enabling bidirectional exchange across multi-scale and multi-level representations.The number of interaction stages depends on the ViT depth.
4. Experiment
ViT-CoMer is evaluated across dense prediction tasks, frameworks, model sizes, and pre-training settings. It consistently improves plain ViT and remains competitive with vision-specific backbones, with ablations supporting its multi-scale convolutional and bidirectional fusion components.
- Object Detection and Instance Segmentation: ViT-CoMer-S improves plain ViT-S by +5.6% box mAP and +3.4% mask mAP under the 1× schedule.Under the 3× schedule, the gains are +4.8% box mAP and +3.1% mask mAP.
- Object Detection and Instance Segmentation: ViT-CoMer consistently outperforms other backbones across COCO detection frameworks, model sizes, and configurations.
- Pre-training: ViT-CoMer leverages diverse pre-training, with self-supervised pre-training improving ViT-CoMer-L over ImageNet-22K pre-training by +3.0% APb and +2.7% APm.ViT-CoMer-B with multi-modal pre-training also gains +1.7% APb and +1.7% APm over ImageNet-1K pre-training.
- Object Detection and Instance Segmentation: ViT-CoMer with BEiTv2 initialization outperforms existing state-of-the-art algorithms on COCO val2017 without extra training data.
- Semantic Segmentation: ViT-CoMer-S achieves 47.7% MS mIoU, exceeding Swin-T by 1.9% and ViT-Adapter-S by 0.6%.ViT-CoMer-L achieves 55.6% MS mIoU, exceeding Swin-L by 2.1% and ViT-Adapter-L by 1.2%.
- Ablation Study: Adding MRFP and CTI progressively improves plain ViT, yielding overall gains of 5.6% APb and 3.4% APm.MRFP contributes 1.3% APb and 1.1% APm, while bidirectional CTI produces the final incremental gains reported in the ablation.
- Ablation Study: Increasing bidirectional fusion interactions reaches a performance plateau, so the model uses N=4 by default.
- Ablation Study: MRFP performs best with kernel sizes 3 and 5, while larger kernels increase the number of parameters.
5. Conclusion
ViT-CoMer combines a plain, non-hierarchical ViT with multi-scale convolutional feature interaction to leverage CNN and Transformer strengths. Experiments across dense prediction tasks show superior performance to plain and adapted backbones while using advanced ViT pre-trained weights.
- ViT-CoMer integrates a multi-scale convolutional feature interaction module into a plain ViT without altering its architecture.
- The method reconstructs fine-grained hierarchical semantic features and leverages the complementary strengths of CNNs and Transformers.
- Experiments on object detection, instance segmentation, and semantic segmentation show superior performance over plain and adapted backbones.
- ViT-CoMer can use advanced ViT pre-trained weights and achieves comparable or better performance than state-of-the-art backbones.