PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaFormer Baselines for Vision

Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, Xinchao Wang

arXiv:2210.13452v4cs.CVcs.AIcs.LG

TL;DR

The paper asks how much competitive vision performance depends on sophisticated token mixers within MetaFormer. It evaluates basic and conventional mixers, finding strong lower-bound, universal, and state-of-the-art results, while StarReLU reduces activation cost and improves accuracy over GELU.

  • Problem

    The paper investigates MetaFormer’s capacity without focusing on token-mixer design, addressing whether basic or conventional mixers can support competitive performance.

  • Method

    The paper instantiates MetaFormer with identity mapping, random mixing, separable convolutions, and vanilla self-attention, and evaluates the resulting models on ImageNet-1K.

  • Results

    MetaFormer provides strong lower-bound and arbitrary-mixer performance, ConvFormer outperforms ConvNeXt, CAFormer reaches 85.5% top-1 accuracy, and StarReLU reduces activation FLOPs by up to 71% versus GELU.

  • Takeaways & Limitations

    Basic or conventional token mixers can yield dependable and competitive MetaFormer baselines, while StarReLU is promising for reducing activation cost alongside neural-network performance.

  • Takeaways & Limitations

    Random mixing introduces frozen parameters and computation quadratic in the token number, making it unsuitable for large token numbers.

Abstract

from arXiv · show

MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, without focusing on token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and summarize our observations as follows: (1) MetaFormer ensures solid lower bound of performance. By merely adopting identity mapping as the token mixer, the MetaFormer model, termed IdentityFormer, achieves >80% accuracy on ImageNet-1K. (2) MetaFormer works well with arbitrary token mixers. When specifying the token mixer as even a random matrix to mix tokens, the resulting model RandFormer yields an accuracy of >81%, outperforming IdentityFormer. Rest assured of MetaFormer's results when new token mixers are adopted. (3) MetaFormer effortlessly offers state-of-the-art results. With just conventional token mixers dated back five years ago, the models instantiated from MetaFormer already beat state of the art. (a) ConvFormer outperforms ConvNeXt. Taking the common depthwise separable convolutions as the token mixer, the model termed ConvFormer, which can be regarded as pure CNNs, outperforms the strong CNN model ConvNeXt. (b) CAFormer sets new record on ImageNet-1K. By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1K: it achieves an accuracy of 85.5% at 224x224 resolution, under normal supervised training without external data or distillation. In our expedition to probe MetaFormer, we also find that a new activation, StarReLU, reduces 71% FLOPs of activation compared with GELU yet achieves better performance. We expect StarReLU to find great potential in MetaFormer-like models alongside other neural networks.

1 INTRODUCTION

The paper probes MetaFormer’s capacity independently of token-mixer novelty, finding dependable performance with identity or random mixing and state-of-the-art results with conventional convolutions and self-attention. It also introduces StarReLU, which reduces activation cost while improving accuracy over GELU.

  • Exploration scope: MetaFormer is evaluated with identity mapping, random mixing, separable convolution, and vanilla self-attention on ImageNet-1K classification.The study deliberately uses basic or established token mixers to explore MetaFormer’s capacity rather than designing new mixers.
  • Lower bound: 80.4% top-1 accuracy: IdentityFormer uses identity mapping with 73M parameters and 11.5G MACs, establishing a solid lower bound.IdentityFormer demonstrates favorable performance even with the lowest degree of token mixing.
  • Arbitrary mixers: 81.4% accuracy: RandFormer improves IdentityFormer by 1.0% using random mixing in the top two stages and identity mapping in the bottom two.The random mixer is used with frozen parameters and avoids excessive computation by limiting random mixing to the top stages.
  • State-of-the-art performance: ConvFormer consistently outperforms ConvNeXt while using separable depthwise convolutions as a token mixer in a pure-CNN formulation.This result uses a conventional token mixer rather than a novel operator.
  • StarReLU: 71% activation FLOPs reduction: StarReLU exceeds GELU by 0.3%/0.2% accuracy on ConvFormer-S18/CAFormer-S18, respectively.StarReLU is designed as a Squared ReLU variant to alleviate distribution shifts.

2 METHOD

MetaFormer abstracts Transformer blocks without specifying the token mixer, enabling models built with identity mapping, random mixing, convolutions, or attention. The paper uses these instantiations to probe MetaFormer's capacity and introduces StarReLU as a lower-cost activation.

  • MetaFormer architecture: MetaFormer embeds input images as token sequences and repeats blocks that mix normalized tokens before applying a channel MLP with residual connections.TokenMixer propagates information among tokens, while the channel MLP uses learnable parameters W1 and W2.
  • Basic token mixers: Global random mixing computes W_RX with a frozen N×N matrix, making its computation and frozen-parameter cost quadratic in token number.Here X has sequence length N and channel dimension C, and WR is randomly initialized before being frozen.
  • Basic token mixers: IdentityFormer uses identity mapping in all four stages, while RandFormer retains identity mapping in the bottom two stages and uses global random mixing in the top two.The staged design limits the extra frozen parameters and computation caused by random mixing at long token lengths.
  • Convolutional and attention mixers: ConvFormer instantiates MetaFormer with inverted depthwise separable convolutions, using pointwise convolutions around a depthwise convolution and a nonlinear activation.The implementation uses kernel size 7 and expansion ratio 2 within a four-stage framework.
  • Convolutional and attention mixers: CAFormer uses convolutions in the first two stages and vanilla self-attention in the last two, balancing linear-cost local processing against quadratic-cost global attention.The design avoids applying vanilla self-attention to early stages with many tokens.
  • StarReLU: StarReLU is introduced as a new activation with learnable scale and bias, costing 4 FLOPs, or 3 FLOPs when only one is used.The paper contrasts this with GELU's approximately 14 FLOPs and reports better performance for StarReLU.

3 EXPERIMENTS

Experiments on ImageNet-1K show that MetaFormer maintains strong performance with identity or random token mixers and reaches state-of-the-art results with conventional convolutional and attention mixers. These models also scale with larger pre-training data and show competitive robustness.

  • Basic token mixers: 80.4% top-1 accuracy is achieved by IdentityFormer-M48 on ImageNet-1K with approximately 73M parameters and 12G MACs.IdentityFormer uses identity mapping as its token mixer, establishing the reported lower-bound performance.
  • Basic token mixers: IdentityFormer-S12/S24 outperform RSB-ResNet-18/34 by 4.0%/2.7%, respectively, while isotropic variants also perform well with convolutional stems.IdentityFormer-S (iso.) reaches 68.2%, increasing to 75.4% with a four-layer convolutional stem.
  • Basic token mixers: 81.4% accuracy is achieved by RandFormer-M48, improving IdentityFormer by 1.0% through random mixing in the top two stages.RandFormer preserves identity mapping in the bottom two stages to limit computation and frozen parameters.
  • Commonly-used token mixers: ConvFormer-B36 surpasses ConvNeXt-B by 0.5% top-1 accuracy while using 51% of its parameters and 66% of its MACs at 224x224 resolution.ConvFormer is a pure-CNN model using separable depthwise convolutions as the token mixer.
  • Commonly-used token mixers: 85.5% top-1 accuracy is reached by CAFormer on ImageNet-1K at 224x224 resolution under normal supervised training without external data or distillation.CAFormer uses separable convolutions in the bottom stages and vanilla self-attention in the top stages.
  • Scaling and robustness: ImageNet-21K pre-training raises ConvFormer-B36 and CAFormer-B36 to 87.0% and 87.4%, respectively, while both remain superior to cited competing models.The reported improvements over ImageNet-1K-only training are 2.2% and 1.9%.
  • Scaling and robustness: ConvFormer and CAFormer show better or competitive robustness than Swin and ConvNeXt without additional fine-tuning or specialized modules.For ImageNet-1K-trained models, ConvFormer-S18 reaches 25.3% on ImageNet-A and 48.7% on ImageNet-R.

3.2 Object detection and instance segmentation

On COCO, ConvFormer and CAFormer backbones consistently outperform Swin and ConvNeXt for object detection and instance segmentation.

  • Results: 52.3 box AP and 45.2 mask AP are obtained by Cascade Mask R-CNN with CAFormer-S18, versus 50.4/50.4 and 43.7/43.7 for Swin-T/ConvNeXt-T.CAFormer is used with sliding-window attention to address out-of-memory issues at the larger image resolution.

3.3 Semantic segmentation

For ADE20K semantic segmentation, ConvFormer and CAFormer backbones outperform Swin and ConvNeXt in UperNet.

  • Results: 48.9 mIoU is achieved with CAFormer-S18, exceeding Swin-T and ConvNeXt-T by 3.1 and 2.2, respectively.The models are evaluated as UperNet backbones on ADE20K.

3.4 Ablation

Ablations show that StarReLU improves ImageNet-1K performance over ReLU, while ResScale and disabling biases are retained as effective design choices. The experiments also compare model efficiency and scaling contexts.

  • Activation: Replacing StarReLU with ReLU lowers ConvFormer-S18/CAFormer-S18 accuracy from 83.0%/83.6% to 82.1%/82.9%.Squared ReLU also provides satisfactory performance in the reported ablation.
  • Block design: ResScale performs best among the evaluated branch-output scaling techniques, and disabling biases improves CAFormer-S18 by 0.1%.Disabling biases does not affect ConvFormer-S18 performance in the cited ablation.

3.5 Benchmark speed

StarReLU is faster than GELU on an NVIDIA A100, while ConvFormer and CAFormer offer better accuracy–throughput trade-offs than several compared models.

  • Experimental setup: ConvFormer and CAFormer are benchmarked against Swin and ConvNeXt after replacing StarReLU with GELU for fair comparison.The comparison is reported in Table 12.
  • Model throughput: ConvNeXt has the highest throughput among similarly sized models with similar MACs.The passage attributes this to ConvNeXt having one residual connection versus two in MetaFormer blocks.
  • Model throughput: ConvFormer and CAFormer achieve higher accuracy and relatively higher throughput than Swin and MaxViT.Together, these properties produce a better accuracy–throughput trade-off, as shown in Figure 4.

4 RELATED WORK

Related work motivates MetaFormer by showing that competitive vision models can use token mixers simpler than attention. The paper positions its contribution as benchmarking basic and established mixers rather than introducing new ones.

  • Transformer alternatives: MLP-Mixer and FNet show that spatial MLPs and Fourier transforms can replace attention while retaining competitive results.This supports the broader view that attention is not the only viable token mixer in Transformer-like vision models.
  • MetaFormer: MetaFormer abstracts Transformer into a general architecture and proposes that its structure, rather than attention alone, is critical for promising performance.PoolFormer tests this hypothesis with pooling and surpasses well-tuned ResNet, ViT, and MLP-like baselines.
  • Downstream evaluation: The related-work evaluation spans semantic segmentation on ADE20K and object detection and instance segmentation on COCO.ADE20K uses UperNet, while COCO uses Mask R-CNN and Cascade Mask R-CNN with specified MAC and FPS measurement settings.
  • MetaFormer extensions: Prior MetaFormer-based models use attention-based, MLP-based, convolution-based, hybrid, and other token mixers.Unlike those efforts, this paper uses basic or commonly used operators instead of introducing novel token mixers.
  • Evaluation context: Inference throughput is measured on an NVIDIA A100 GPU with batch size 128 and TF32 at 224^2 resolution.Figure 4 presents the accuracy–throughput trade-off, while Table 12 reports model throughputs.

5 CONCLUSION

The paper explores MetaFormer with simple and conventional token mixers, finding strong lower-bound, universality, and record-setting results while identifying StarReLU as a more efficient activation than GELU.

  • Conclusion: IdentityFormer and RandFormer demonstrate MetaFormer’s solid lower bound and universality to token mixers.They use identity mapping and randomized mixing, respectively.
  • Conclusion: ConvFormer and CAFormer obtain record-setting results using conventional separable convolutions and vanilla self-attention.These models deliberately avoid novel token-mixer designs.
  • Conclusion: StarReLU achieves better performance while greatly reducing activation FLOPs compared with GELU.The paper expects StarReLU to have broader applications in MetaFormer-like models and other neural networks.

APPENDIX A EXPECTATION AND VARIANCE OF SQUARED RELU

The appendix derives expectation and variance quantities for Squared ReLU under a standard normal input assumption, using Gaussian moments and the definitions of expectation and variance.

  • Assumption: x ∼ N(0, 1) is assumed for the input to Squared ReLU.The derivation relies on the input having mean 0 and variance 1.
  • Expectation: E((ReLU(x))^2) is derived as 0.5 E(x^2), yielding 0.5 under the stated assumption.The displayed derivation evaluates the second moment of the rectified input.
  • Variance: The derivation evaluates E(x^4) through a standard-normal integral before using it in the variance calculation.The passage identifies E(·) and Var(·) as expectation and variance, respectively.
  • Variance: Variance is computed as E((ReLU(x))^4) − E((ReLU(x))^2)^2.This follows the standard variance identity stated in the displayed derivation.

APPENDIX B CODE OF SEPARABLE CONVOLUTION

This appendix presents Algorithm 2 as PyTorch-like code for the inverted separable convolution used in MobileNetV2.

  • Algorithm 2 shows the implementation code for inverted separable convolution.
  • The implementation is presented in a PyTorch-like coding style.
  • The convolution operation is associated with MobileNetV2.

APPENDIX C HYPER-PARAMETERS

The appendix lists hyper-parameters for the paper’s MetaFormer models across ImageNet-1K training, fine-tuning, and pre-training settings.

  • Table 13 reports hyper-parameters for IdentityFormer, RandFormer, and PoolFormerV2 trained on ImageNet-1K.
  • Table 15 reports hyper-parameters for ConvFormer and CAFormer pre-trained on ImageNet-1K and fine-tuned on ImageNet-1K.
  • Table 14 reports hyper-parameters for ConvFormer and CAFormer trained on ImageNet-1K and finetuned at larger resolution of 3842.
Loading 2210.13452v4…