PaperlayerView exact revision on arXiv

Source-linked AI summary

MixFormer: Mixing Features across Windows and Dimensions

Qiang Chen, Qiman Wu, Jian Wang, Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, Jingdong Wang

arXiv:2204.02557v2cs.CV

TL;DR

Local-window self-attention is efficient but has limited receptive fields and weak channel modeling. MixFormer addresses these issues with parallel attention and depth-wise convolution plus bi-directional cross-branch interactions, achieving competitive classification and stronger downstream results at lower cost. Its main scope boundary is that gains from applying the design to global attention remain unclear.

  • Problem

    Local-window self-attention has limited receptive fields and weak channel modeling because it uses non-overlapped windows and shares weights across channels.

  • Method

    MixFormer parallelizes local-window self-attention with depth-wise convolution and adds bi-directional interactions that exchange complementary channel and spatial information.

  • Results

    MixFormer achieves competitive ImageNet-1K results, surpasses RegNet and Swin Transformer, and outperforms alternatives across five dense prediction tasks with lower computational costs.

  • Takeaways & Limitations

    MixFormer provides an efficient general-purpose vision transformer with improved performance across image classification and diverse downstream vision tasks.

  • Takeaways & Limitations

    The reported design is mainly limited to window-based vision transformers, while applying it to global attention produced slightly worse results and requires further study.

Abstract

from arXiv · show

While local-window self-attention performs notably in vision tasks, it suffers from limited receptive field and weak modeling capability issues. This is mainly because it performs self-attention within non-overlapped windows and shares weights on the channel dimension. We propose MixFormer to find a solution. First, we combine local-window self-attention with depth-wise convolution in a parallel design, modeling cross-window connections to enlarge the receptive fields. Second, we propose bi-directional interactions across branches to provide complementary clues in the channel and spatial dimensions. These two designs are integrated to achieve efficient feature mixing among windows and dimensions. Our MixFormer provides competitive results on image classification with EfficientNet and shows better results than RegNet and Swin Transformer. Performance in downstream tasks outperforms its alternatives by significant margins with less computational costs in 5 dense prediction tasks on MS COCO, ADE20k, and LVIS. Code is available at \url{https://github.com/PaddlePaddle/PaddleClas}.

1. Introduction

MixFormer addresses the limited receptive field and weak channel modeling of local-window self-attention by combining parallel convolutional and attention paths with bi-directional interactions. It achieves competitive classification results and stronger downstream performance at lower computational cost.

  • 1. Introduction: Local-window self-attention improves efficiency but limits receptive fields and channel modeling because it operates in non-overlapped windows and shares channel weights.Prior combinations with convolution capture intra-window and cross-window relations successively, leaving them less interweaved.
  • 1. Introduction: MixFormer combines local-window self-attention and depth-wise convolution in parallel, while bi-directional interactions provide complementary channel and spatial clues across branches.The parallel design models intra-window and cross-window relations simultaneously, and the interactions enhance modeling ability in both dimensions.
  • 1. Introduction: MixFormer performs on par with EfficientNet and surpasses RegNet and Swin Transformer across image classification and dense prediction tasks.The model series spans 0.7G to 3.6G computational complexity and is evaluated across multiple vision tasks.
  • 1. Introduction: MixFormer-B4 gains 2.9 box mAP and 2.1 mask mAP over Swin-T on MS COCO with Mask R-CNN while using less computational cost.The reported comparison uses the 1× schedule.

2. Related Works

Related work develops efficient vision transformers, receptive-field expansion, and dynamic mechanisms for modeling channel or spatial relations. MixFormer combines these directions through lightweight interactions between complementary attention and convolution branches.

  • 2. Related Works: Local-window self-attention and depth-wise convolution share weights on different dimensions, motivating their use as complementary components.The former shares weights across channels, whereas the latter shares weights across spatial positions.
  • 2. Related Works: Window-based vision transformers reduce the quadratic cost of global attention for high-resolution tasks but retain limitations in local relation modeling and receptive field.Local-window attention computes within non-overlapping windows, motivating operations that connect nearby windows.
  • 2. Related Works: Researchers expand receptive fields through shifting, expanding, shuffling, or convolution, but these approaches can capture intra-window and cross-window relations in successive rather than interwoven steps.MixFormer builds on convolution as an efficient way to model local and cross-window relations.
  • 2. Related Works: Dynamic networks use data-dependent weights or paths to flexibly model relations across channel and spatial dimensions.MixFormer applies lightweight dynamic interactions across two branches based on their complementary weight-sharing dimensions.

3. Method

MixFormer builds a hybrid pyramid transformer from a Mixing Block that parallelizes local-window attention and depth-wise convolution, then exchanges channel and spatial information between them. Its architecture uses staged downsampling, projection, and manually configured variants to balance accuracy and efficiency.

  • 3.1. The Mixing Block: The Mixing Block parallelizes local-window self-attention and depth-wise convolution, concatenates their normalized outputs, and sends the merged features to an FFN.The parallel paths model intra-window and cross-window relations simultaneously, enabling feature interweaving across branches.
  • 3.1. The Mixing Block: Bi-directional interactions transfer depth-wise convolution’s channel context and local-window attention’s spatial context to the opposite branches.Channel interaction strengthens channel modeling, while spatial interaction transfers spatial relations between paths.
  • 3.2. MixFormer: MixFormer is a four-stage hybrid transformer with convolutional stem and downsampling layers, pyramid feature maps, and a projection layer expanding channels to 1280.The stages downsample at rates {4, 8, 16, 32}; the projection layer precedes the classification head.

4. Experiments

MixFormer is evaluated across classification, detection, segmentation, and additional dense prediction tasks, where it generally improves accuracy while maintaining computational efficiency. Ablations attribute these gains to parallel feature mixing, bi-directional interactions, and stage/block design choices.

  • 4.1. Image Classification: 83.0% Top-1 accuracy is achieved by MixFormer-B4 with 3.6G FLOPs, surpassing Swin-T by 1.7% while saving 20% computational costs.MixFormer performs on par with EfficientNet and outperforms RegNet across computational budgets.
  • 4.2. Object Detection and Instance Segmentation: +2.9 box mAP and +2.1 mask mAP over Swin-T are achieved by MixFormer-B4 with Mask R-CNN under the 1× schedule.MixFormer-B1 also exceeds ResNet50 despite using 0.7G versus 4.1G FLOPs.
  • 4.3. Semantic Segmentation: MixFormer-B4 outperforms Swin-T by 2.2 mIoU on ADE20K semantic segmentation.MixFormer-B3 reaches 45.5 mIoU with fewer FLOPs than Swin-T, matching the reported performance of OCRNet with HRNet-W48.
  • 4.4. Ablation Study: Combining channel and spatial interactions improves results by 0.3% Top-1 accuracy, 0.9/0.7 box/mask mAP, and 1.1 mIoU.The gains are reported across ImageNet-1K, COCO, and ADE20K with lightweight interaction designs.
  • 4.4. Ablation Study: Reducing depth-wise convolution from 3 × 3 to 1 × 1 lowers performance by 1.3 Top-1 accuracy, 4.0/3.0 box/mask mAP, and 3.3 mIoU.The ablation supports using at least a 3 × 3 convolution window to connect across windows.
  • 4.5. Generalization: MixFormer outperforms alternatives in 5 dense prediction tasks, including keypoint detection and long-tail instance segmentation.It exceeds Swin-T by 1.0 APmask on LVIS and outperforms baseline models across all reported keypoint-detection metrics.

5. Limitations

MixFormer is primarily designed for window-based vision transformers, and its extension to global attention remains unresolved. Manual construction also limits the current model series, leaving room for NAS-based improvements.

  • MixFormer may be limited to window-based vision transformers because its designs target local-window self-attention.Applying the mixing block to global attention produced a slightly worse result on DeiT-Tiny.
  • The gains from applying the parallel design and bi-directional interactions to global self-attention remain unclear.The authors report that more effort is needed to adapt the mixing block to global attention.
  • Manually building the MixFormer series restricts it to existing instances, while NAS could further improve results.

6. Conclusion

MixFormer is an efficient general-purpose vision transformer that addresses limited receptive fields and weak channel modeling in window-based transformers. Its parallel mixing and bi-directional interactions support strong results across image classification and downstream vision tasks.

  • MixFormer addresses limited receptive fields and weak modeling capability on the channel dimension in window-based vision transformers.
  • MixFormer combines local-window attention with depth-wise convolution in parallel and adds bi-directional branch interactions.The design enlarges receptive fields while improving modeling in channel and spatial dimensions.
  • Extensive experiments show that MixFormer outperforms alternatives on image classification and various downstream vision tasks.

A. More Variants of MixFormer

Additional MixFormer variants demonstrate scalability from mobile-level models to larger architectures. The reported results show competitive ImageNet accuracy at low computation and strong performance for larger models.

  • The MixFormer series scales from smaller to larger models, with B0 and B5 presented as additional examples alongside B1–B4.The authors state that further variants can be obtained from the design.
  • 76.5% Top-1 accuracy on ImageNet-1K is achieved by MixFormer-B0 with 0.4G FLOPs.This places MixFormer-B0 at a mobile-level computational cost.
  • 83.8% top-1 accuracy on ImageNet-1K is achieved by MixFormer-B6, with performance superior to Swin-B at 15.4G FLOPs.MixFormer-B6 is also comparable to other models with fewer FLOPs.

B. Additional Experiments

Additional experiments examine window size and the applicability of the mixing block to global attention. Larger windows preserve ImageNet performance in one test, whereas the DeiT-Tiny adaptation performs below baseline.

  • Window Sizes in Local-Window Self-Attention: 78.4% ImageNet-1K accuracy is achieved with both ws=12 and ws=7 in the MixFormer-B1 window-size study.The authors retain ws=7 in all variants following the conventional Swin Transformer design.
  • Window Sizes in Local-Window Self-Attention: The window-size experiment supports using ws=7 because ws=12 achieves on-par performance rather than a reported improvement.
  • Apply Mixing Block to DeiT: The mixing block lowers DeiT-Tiny ImageNet-1K accuracy to 71.3% versus the 72.2% baseline.The authors suggest global attention may not share the same problem and leave its detailed design for future work.

C. Detailed Experimental Settings

The experiments use standardized training protocols across ImageNet-1K and five dense-prediction tasks, with task-specific optimizers, schedules, resolutions, and testing procedures. Comparative designs are matched for computational complexity, and detailed settings are provided in accompanying tables.

  • Successive and Parallel Designs: The successive and parallel combinations of local-window self-attention and depth-wise convolution are compared at matched computational complexity.Channels are adjusted between blocks to ensure a fair comparison; Figure 4 omits secondary block details for clarity.
  • Image Classification: ImageNet-1K classification uses 300 training epochs at 224 × 224 resolution, with settings adjusted for model size.Detailed classification settings are listed in Table 17.
  • Object Detection and Instance Segmentation: MS COCO detection and instance segmentation use Mask R-CNN and Cascade Mask R-CNN with AdamW, an initial learning rate of 0.0002, and batch size 16.MixFormer4 uses trainable normalization layers for fair comparison, with detailed hyperparameters in Table 18.
  • Semantic Segmentation: ADE20K semantic segmentation uses AdamW, learning rate 0.00006, weight decay 0.01, batch size 16, and 160K training iterations.Main comparisons report single-scale and multi-scale testing, while ablations report single-scale results.
  • Keypoint and Long-tail Instance Segmentation: Keypoint detection follows a 210-epoch AdamW protocol at 256 × 192 resolution, while LVIS long-tail segmentation uses a 1× schedule with balanced sampling.The keypoint setup largely follows HRFormer, and LVIS uses multi-scale training with common augmentations across models.

D. Discussion with Related Works

MixFormer exchanges information across dimensions and windows to address limitations associated with weight sharing and non-overlapped local windows. Unlike related approaches, it concatenates non-overlapped-window features with local-connected depth-wise-convolution features.

  • Information Exchange Across Dimensions: MixFormer addresses weak modeling ability from channel and spatial weight sharing through channel and spatial interactions across dimensions.Its cross-dimension exchange differs from Conformer’s coupling of local and global features across convolutional and transformer branches.
  • Comparison with Related Works: The discussion distinguishes MixFormer from Conformer, Twins, Shuffle Transformer, and MSG Transformer by comparing the type of branch or spatial connection each method uses.Conformer exchanges information between transformer and convolution branches, whereas the other cited methods construct global, neighboring-window, shuffled, or token-based connections.
  • Information Exchange Across Windows: MixFormer achieves cross-window information exchange by concatenating non-overlapped-window features with local-connected depth-wise-convolution features.This design is contrasted with global or shuffled connections used by Twins, Shuffle Transformer, and MSG Transformer.
Loading 2204.02557v2…