Source-linked AI summary
Inception Transformer
Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, Shuicheng Yan
TL;DR
Transformers capture long-range and low-frequency visual information but are less effective at high-frequency local details. iFormer combines channel-split convolution/max-pooling and self-attention mixers, with a frequency ramp across layers, and outperforms representative backbones across classification, detection, and segmentation. The paper also notes manually defined frequency-ramp channel ratios and the absence of large-scale ImageNet-21K training as limitations.
Problem
ViTs model low-frequency global shapes and structures well but are less powerful at high-frequency local edges and textures.
Method
iFormer splits channels between parallel convolution/max-pooling and self-attention mixers, then shifts more channels toward low-frequency processing in higher layers.
Results
iFormer outperforms representative vision Transformers across image classification, object detection, and semantic segmentation.
Takeaways & Limitations
The results demonstrate iFormer’s potential as a general-purpose vision backbone across classification, detection, and semantic segmentation.
Takeaways & Limitations
The frequency-ramp channel ratios are manually defined, and iFormer was not trained on large-scale datasets such as ImageNet-21K due to computational constraints.
Abstract
from arXiv · showhide
Recent studies show that Transformer has strong capability of building long-range dependencies, yet is incompetent in capturing high frequencies that predominantly convey local information. To tackle this issue, we present a novel and general-purpose Inception Transformer, or iFormer for short, that effectively learns comprehensive features with both high- and low-frequency information in visual data. Specifically, we design an Inception mixer to explicitly graft the advantages of convolution and max-pooling for capturing the high-frequency information to Transformers. Different from recent hybrid frameworks, the Inception mixer brings greater efficiency through a channel splitting mechanism to adopt parallel convolution/max-pooling path and self-attention path as high- and low-frequency mixers, while having the flexibility to model discriminative information scattered within a wide frequency range. Considering that bottom layers play more roles in capturing high-frequency details while top layers more in modeling low-frequency global information, we further introduce a frequency ramp structure, i.e. gradually decreasing the dimensions fed to the high-frequency mixer and increasing those to the low-frequency mixer, which can effectively trade-off high- and low-frequency components across different layers. We benchmark the iFormer on a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection and ADE20K segmentation. For example, our iFormer-S hits the top-1 accuracy of 83.4% on ImageNet-1K, much higher than DeiT-S by 3.6%, and even slightly better than much bigger model Swin-B (83.3%) with only 1/4 parameters and 1/3 FLOPs. Code and models will be released at https://github.com/sail-sg/iFormer.
1 Introduction
ViTs model global low-frequency information well but struggle with local high-frequency details. iFormer addresses this gap by combining frequency-specific mixers and progressively shifting channel capacity toward low-frequency processing in higher layers, achieving strong results across vision tasks.
- Motivation: ViTs capture global shapes and structures effectively but are less powerful at learning local edges and textures.Self-attention is primarily a global operation, favoring low-frequency information.
- Inception Transformer: iFormer splits channels between a high-frequency mixer using max-pooling and parallel convolution and a low-frequency mixer using self-attention.This design targets frequency-specific information while retaining self-attention for global modeling.
- Frequency ramp structure: The frequency ramp feeds more channels to the low-frequency mixer and fewer to the high-frequency mixer from lower to higher layers.The structure trades off high- and low-frequency components across layers.
- Results: iFormer surpasses state-of-the-art ViTs and CNNs across image classification, object detection, and semantic segmentation.The reported benchmarks include ImageNet-1K, COCO detection, and ADE20K segmentation.
2 Related work
Related Transformer and CNN backbones offer complementary strengths: attention captures global dependencies, while convolutions model local dependencies and high-frequency information. Existing hybrid designs combine them serially or in parallel, but parallel approaches may process all channels redundantly despite differing local and global channel roles.
- Transformers: Transformer attention captures global dependencies but mainly models low-frequency information and can produce similar representations across layers.These properties motivate efforts to complement attention with local modeling.
- CNNs and hybrids: CNNs model local dependencies and extract high-frequency information, motivating their integration into Transformer architectures.Hybrid methods introduce convolution into Transformers in serial or parallel configurations.
- Hybrid designs: Serial hybrids inject local information into global context but model only one dependency type in each layer.Parallel methods instead combine attention and convolution branches, commonly merging their outputs by elementwise sum.
3 Method
iFormer augments Vision Transformers with an Inception token mixer that separates high- and low-frequency processing, then uses a frequency ramp to rebalance these components across layers.
- Inception token mixer: iFormer splits input channels between high-frequency convolution/max-pooling paths and a low-frequency self-attention path.This parallel design combines local-detail extraction with global information exchange.
- Inception token mixer: The high-frequency mixer uses parallel max-pooling and depthwise-convolution branches to capture detail-sensitive representations.The two branch outputs are later concatenated with the low-frequency output.
- Inception token mixer: A fusion module adds depthwise convolution after upsampling while retaining a per-location cross-channel linear layer.This exchanges information between patches and addresses excessive smoothness between adjacent tokens.
- Inception Transformer block: Each iFormer block applies LayerNorm before the Inception token mixer and feed-forward network, following the Transformer block design.The block combines the Inception token mixer with a conventional FFN.
- Inception token mixer: The low-frequency mixer applies average pooling before multi-head self-attention and upsampling afterward to reduce computation at large resolutions.Pooling and upsampling use kernel size and stride 2 only in the first two stages.
- Frequency ramp structure: The frequency ramp gradually decreases high-frequency channel dimensions and increases low-frequency channel dimensions from shallow to deep layers.This structure balances high- and low-frequency components across the four-stage backbone.
4 Experiments
Experiments evaluate iFormer across image classification, detection, segmentation, ablations, frequency behavior, and visual localization. The results report strong benchmark performance and support both the Inception mixer and frequency ramp structure.
- Benchmark scope: iFormer is evaluated on ImageNet classification, COCO detection and instance segmentation, and ADE20K semantic segmentation, with ablations and visualizations.The benchmark compares iFormer with representative ViTs, CNNs, and hybrid backbones.
- Image classification: 84.6% top-1 accuracy is achieved by iFormer-B on ImageNet at approximately 50M parameters, improving over similarly sized state-of-the-art ViTs by 1.0%.It also improves over similarly sized hybrid ViTs by 0.7% and ConvNeXt-S by 1.5%.
- Image classification: iFormer consistently outperforms counterparts when fine-tuned at 384×384 resolution on ImageNet across different computation settings.Table 2 reports the larger-resolution fine-tuning comparison.
- Object detection and instance segmentation: iFormer-S improves over ResNet50 by 8.2 APb and 7.5 APm on COCO, while iFormer-B exceeds UniFormer-B by 0.9 APb and Swin-S by 3.5 APb.The comparisons use Mask R-CNN backbones under similar computation configurations.
- Semantic segmentation: 48.6 mIoU is achieved by iFormer-S on ADE20K, surpassing UniFormer-S by 2.0 mIoU with less computation complexity.Compared with UniFormer-B, iFormer-S improves by 0.6 mIoU with only 1/2 the parameters and nearly 1/3 the FLOPs.
- Ablation study and visualization: Ablations show that combining attention with convolution and max-pooling improves accuracy while reducing computation, while the frequency ramp configuration with increasing low-frequency and decreasing high-frequency channels performs best.Fourier analysis associates attention with low frequencies and convolution/max-pooling with high frequencies; Grad-CAM shows more complete object localization for iFormer than Swin.
5 Conclusion
iFormer combines convolution/max-pooling with self-attention through channel splitting and uses a frequency ramp to trade off high- and low-frequency components across layers. The authors identify manual channel-ratio selection and limited training scale as limitations.
- iFormer couples convolution/max-pooling and self-attention through channel splitting to expand Transformer frequency coverage efficiently.The architecture uses an Inception mixer for high- and low-frequency processing.
- The frequency ramp gradually shifts channel capacity from high-frequency to low-frequency processing across layers.This is intended to support hierarchical representations with more local detail in lower layers and more global information in higher layers.
- Manually defining the channel ratio for each iFormer block requires substantial experience to adapt the structure across tasks.The paper suggests neural architecture search as a straightforward potential solution.
- iFormer was not trained on large-scale datasets such as ImageNet-21K because of computational constraints.The authors identify larger-scale training as future work.
A Appendix
The paper presents iFormer as a general-purpose backbone for image classification, object detection, and semantic segmentation, while noting potential misuse in harmful applications.
- iFormer is designed for multiple vision tasks, including image classification, object detection, and semantic segmentation.
- As a general-purpose backbone, iFormer could be used in harmful applications such as illegal face recognition.
A.1 Results on semantic segmentation
On ADE20K semantic segmentation with UperNet, iFormer achieves higher reported mIoU scores than the compared backbones, including Swin-T and UniFormer-S.
- 48.4 mIoU and 48.8 MS mIoU are reported for iFormer on ADE20K semantic segmentation.The evaluation uses the UperNet framework.
- 3.9 mIoU is the reported improvement of iFormer-S over Swin-T while using fewer parameters.
- 0.8 mIoU is the reported improvement of iFormer-S over UniFormer-S.
A.2.1 Fourier spectrum of different layers
Fourier-spectrum visualizations indicate that iFormer emphasizes high-frequency components in lower layers and low-frequency information in higher layers, with the balance changing progressively across depth.
- Figure 6 presents Fourier spectra for the 6-th, 12-th, and 18-th iFormer-S layers.
- The 6-th layer captures more high-frequency components, whereas the 18-th layer captures more low-frequency information.
- High-frequency information decreases from the 6-th to the 18-th layer while low-frequency information increases.The authors interpret this progression as evidence that iFormer trades off frequency components across layers.
A.2.2 CAM
Grad-CAM examples compare how iFormer-S and Swin-T attend to image content. iFormer more accurately and completely focuses on key objects in the shown examples.
- iFormer-S and Swin-T are compared using Grad-CAM activation maps on ImageNet-1K-trained models.
- iFormer attends to key objects more accurately and completely than Swin in the displayed examples.
- In the hog example, iFormer locates the hog accurately, whereas Swin also attends to an irrelevant part.
A.3 Configurations of iFormers
The iFormer variants use a four-stage architecture with progressively changing spatial resolutions and frequency allocations. Their configurations specify high- and low-frequency components, mixer operations, pooling details, and FLOPs measurement settings.
- iFormer provides three variants—iFormer-S, iFormer-B, and iFormer-L—for fair comparisons under different computation configurations.Table 7 gives their detailed configurations.
- The high-frequency channel ratio Ch/C decreases from shallow to deep layers, while the low-frequency ratio Cl/C increases.This frequency ramp trades off high- and low-frequency components across layers.
- Each iFormer block uses depthwise convolution and max-pooling as high-frequency mixers, with both kernel sizes set to 3 × 3.
- Table 7 defines pool stride for pooling and upsample layers in the attention branch, while FLOPs are measured at 224 × 224 resolution.