PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientFormer: Vision Transformers at MobileNet Speed

Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, Jian Ren

arXiv:2206.01191v5cs.CV

TL;DR

Vision Transformers are promising but often too slow for real-time use on resource-constrained devices. EfficientFormer addresses this by combining a dimension-consistent pure-transformer design with latency-driven slimming, achieving strong accuracy at mobile inference speeds. The resulting models outperform existing transformer designs while remaining faster than many competitive CNNs.

  • Problem

    Vision Transformers are often slower than lightweight CNNs, making real-time deployment difficult on resource-constrained hardware.

  • Method

    EfficientFormer uses a dimension-consistent pure-transformer design derived from latency analysis and applies latency-driven slimming to optimize mobile inference speed.

  • Results

    EfficientFormer models outperform existing transformer models while being faster than most competitive CNNs across image classification, detection, and segmentation tasks.

  • Takeaways & Limitations

    Properly designed vision transformers can reach extremely low mobile latency while maintaining high performance.

  • Takeaways & Limitations

    Actual EfficientFormer speed may vary across hardware and compilers, and more exhaustive search could yield better results when search cost is acceptable.

Abstract

from arXiv · show

Vision Transformers (ViT) have shown rapid progress in computer vision tasks, achieving promising results on various benchmarks. However, due to the massive number of parameters and model design, \textit{e.g.}, attention mechanism, ViT-based models are generally times slower than lightweight convolutional networks. Therefore, the deployment of ViT for real-time applications is particularly challenging, especially on resource-constrained hardware such as mobile devices. Recent efforts try to reduce the computation complexity of ViT through network architecture search or hybrid design with MobileNet block, yet the inference speed is still unsatisfactory. This leads to an important question: can transformers run as fast as MobileNet while obtaining high performance? To answer this, we first revisit the network architecture and operators used in ViT-based models and identify inefficient designs. Then we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm. Finally, we perform latency-driven slimming to get a series of final models dubbed EfficientFormer. Extensive experiments show the superiority of EfficientFormer in performance and speed on mobile devices. Our fastest model, EfficientFormer-L1, achieves $79.2\%$ top-1 accuracy on ImageNet-1K with only $1.6$ ms inference latency on iPhone 12 (compiled with CoreML), which runs as fast as MobileNetV2$\times 1.4$ ($1.6$ ms, $74.7\%$ top-1), and our largest model, EfficientFormer-L7, obtains $83.3\%$ accuracy with only $7.0$ ms latency. Our work proves that properly designed transformers can reach extremely low latency on mobile devices while maintaining high performance.

1 Introduction

Vision Transformers offer strong vision performance but remain slower than lightweight CNNs on resource-constrained devices. EfficientFormer addresses this gap with a dimension-consistent pure-transformer design and latency-driven slimming, achieving mobile-level speed with high accuracy.

  • Motivation: ViT models are typically slower than lightweight CNNs, limiting real-time deployment on mobile devices and wearables.Factors include large parameter counts, quadratic token-length computation, non-foldable normalization, and limited compiler optimizations.
  • Motivation: Existing approaches reduce transformer cost through convolutions, MobileNet blocks, sparse attention, architecture search, or pruning, but the central latency-performance question remains.These methods improve the computation-performance trade-off without resolving whether pure transformers can match MobileNet speed.
  • Approach: EfficientFormer introduces a dimension-consistent vision-transformer design based on latency analysis and identifies inefficient designs and operators.The design is evaluated using iPhone 12 and CoreML as a reproducible mobile testbed.
  • Approach: Latency-driven slimming converts a supernet into a family of EfficientFormer models by optimizing inference speed rather than MACs or parameter count.This directly targets on-device latency as the optimization objective.
  • Results: 79.2% top-1 accuracy and 1.6 ms latency are achieved by EfficientFormer-L1 on ImageNet-1K, matching MobileNetV2×1.4 speed with 4.5% higher accuracy.EfficientFormer-L7 reaches 83.3% accuracy at 7.0 ms latency and outperforms MobileViT-XS.
  • Conclusion: EfficientFormer provides evidence that vision transformers can combine ultra-fast mobile inference with strong performance.The models are presented as a baseline for edge deployment of vision transformers.

2 Related Work

Prior work accelerates ViTs through new architectures, attention mechanisms, hybrid modules, and search, but lightweight CNNs remain stronger when edge inference speed is central. EfficientFormer instead targets the latency-performance boundary of pure vision transformers and directly optimizes mobile latency.

  • Prior approaches: Existing acceleration methods include new architectures, sparse or redesigned attention, hybrid MobileNet modules, and search algorithms for smaller and faster ViTs.MobileViT combines MobileNet blocks for early feature extraction with MHSA blocks in later stages.
  • EfficientFormer: EfficientFormer differs by pursuing the latency-performance boundary of pure vision transformers rather than relying on hybrid designs.Its design is derived from latency analysis and can be further improved through architecture search.

3 On-Device Latency Analysis of Vision Transformers

On-device profiling identifies patch embedding, reshaping, feature-dimension changes, normalization, and nonlinearities as important latency considerations. EfficientFormer responds with hardware-friendly 4D and 3D blocks that preserve dimensions and avoid unnecessary reshaping while retaining MHSA where beneficial.

  • Patch embedding: Large-kernel, large-stride patch embedding is a mobile speed bottleneck because compilers poorly support these convolutions.EfficientFormer replaces it with a convolution stem using several hardware-efficient 3 × 3 convolutions and fast downsampling.
  • Token mixers: Consistent feature dimensions make MHSA compatible with mobile latency, whereas frequent Reshape operations can become the bottleneck.DeiT-Small achieves comparable speed to LeViT-256 despite greater computation when reshaping is avoided.
  • Token mixers: EfficientFormer compares pooling and MHSA as token mixers, using pooling for simplicity and efficiency and MHSA for better performance.Shifted-window attention is outside scope because public mobile compilers generally do not support it.
  • Operators: CONV-BN is more latency-favorable than LN or GN-Linear because batch normalization can be folded into convolution during inference.The authors report only a slight performance downgrade relative to GN and comparable results to channel-wise LN.
  • EfficientFormer design: The architecture uses a convolution stem followed by MetaBlocks with 4D and 3D token-mixer configurations arranged dimension-consistently.The overview combines local pooling in MB4D with global multi-head self-attention in MB3D.
  • Operators: Nonlinearity latency depends on hardware and compiler: on iPhone 12, GeLU is close to ReLU, while HardSwish is substantially slower.LeViT-256 latency is 44.5 ms with HardSwish versus 11.9 ms with GeLU in the reported experiment.

4 Design of EfficientFormer

EfficientFormer combines a dimension-consistent transformer design with latency-driven architecture search and slimming. Its 4D and 3D partitions preserve efficient convolution-style processing while enabling later-stage global attention.

  • Architecture: EfficientFormer is a fully transformer-based model that does not integrate MobileNet structures.The network contains a patch embedding, meta transformer blocks, four stages, and embedding operations between stages.
  • Dimension-Consistent Design: The dimension-consistent design separates convolution-style 4D processing from 3D linear projections and attention.The network begins with a 4D partition and applies the 3D partition in later stages.
  • Dimension-Consistent Design: The model starts with a two-convolution stem and uses Pool mixers in MB4D blocks to extract low-level features.The stem uses two 3 × 3 convolutions with stride 2 for patch embedding; the 4D partition uses CONV-BN-based processing without Group or Layer Normalization before Pool mixers.
  • Latency-Driven Slimming: The supernet represents candidate blocks with MetaPaths and searches stage widths, depths, and the placement of MB3D blocks.The search space includes Cj, Nj, and the last N blocks that apply MB3D.
  • Search Space: MB3D is restricted to later stages because MHSA computation grows quadratically with token length and early stages capture low-level features.The supernet allows MB4D or identity paths in the first two stages, and MB3D, MB4D, or identity paths in the last two.
  • Latency-Driven Slimming: The search trains the supernet once, uses Gumbel Softmax importance scores and a latency lookup table, then iteratively slims the network by latency-aware actions.Actions include selecting identity for a least-important path, removing the first MB3D, or reducing the width of the least-important stage; the process continues until the target latency is reached.

5 Experiments and Discussion

Experiments evaluate EfficientFormer on classification, detection, instance segmentation, and semantic segmentation, comparing accuracy and latency with CNN, ViT, and hybrid models. Results show strong accuracy–latency trade-offs, while the authors note platform-dependent speed and search-cost limitations.

  • Image Classification: EfficientFormer-L1 matches MobileNetV2×1.4 speed while achieving 4.5% higher top-1 accuracy on iPhone Neural Engine.EfficientFormer-L7 also exceeds 83% top-1 accuracy while running more than 3× faster than EfficientNet-B5.
  • Image Classification: EfficientFormer-L3 reaches higher accuracy than DeiT-Small while running 4× faster.The reported comparison is 82.4% versus 81.2% top-1 accuracy.
  • Image Classification: EfficientFormer-L1 outperforms MobileViT-XS by 4.4% top-1 accuracy and runs faster across Nvidia A100, iPhone CPU, and iPhone NPU.At similar inference time, EfficientFormer-L7 exceeds MobileViT-XS by 8.5% top-1 accuracy.
  • Detection and Instance Segmentation: With similar computation cost, EfficientFormer-L3 exceeds ResNet50 by 3.4 box AP and 3.7 mask AP, and PoolFormer-S24 by 1.3 box AP and 1.1 mask AP.These results use EfficientFormer as a backbone for COCO object detection and instance segmentation.
  • Semantic Segmentation: EfficientFormer-L3 exceeds PoolFormer-S24 by 3.2 mIoU on semantic segmentation under a similar computation budget.The passage attributes the result to global attention supporting better long-term dependencies in high-resolution dense prediction.
  • Discussion: EfficientFormer speed may vary across hardware and compilers, and enumeration-based brute-force search could improve slimming results when search cost is acceptable.The limitation specifically mentions GeLU and HardSwish implementation differences across platforms.

6 Conclusion

The conclusion argues that EfficientFormer combines hardware-friendly 4D MetaBlocks with 3D MHSA and latency-driven slimming to produce fast, capable vision-transformer models. Across classification, detection, and segmentation, the models outperform existing transformer models while being faster than most competitive CNNs.

  • 6 Conclusion: EfficientFormer uses dimension-consistent 4D MetaBlocks and 3D MHSA blocks to combine hardware friendliness with transformer capacity.The design is derived from latency analysis of inefficient operators in ViT-based architectures.
  • 6 Conclusion: A fast latency-driven slimming method derives optimized EfficientFormer configurations from the proposed design space.The conclusion presents slimming as the final model-optimization step.
  • 6 Conclusion: Across image classification, object detection, and segmentation, EfficientFormer models outperform existing transformer models while being faster than most competitive CNNs.The conclusion frames these experiments as validation that vision transformers can achieve ultra-fast edge inference.
  • 6 Conclusion: The paper concludes that vision transformers can operate at MobileNet speed on mobile devices.Future work will explore EfficientFormer on additional resource-constrained devices.

Checklist

The checklist records affirmative responses for contribution and limitation disclosure, ethics-related discussion, reproducibility materials, training details, error bars, compute reporting, and asset citations or licensing. Items involving theory, human subjects, or new assets are marked not applicable where indicated.

  • Checklist: The authors report that they described their contributions, limitations, and potential negative societal impacts.They also state that the ethics review guidelines were read and followed.
  • Checklist: The paper reports providing code, data, reproduction instructions, training details, error bars, and compute-resource information for experiments.These checklist items are all marked affirmatively.
  • Checklist: Existing assets are cited and their licenses are mentioned, while inclusion of new assets is marked not applicable.The checklist also marks theoretical assumptions and complete proofs as not applicable.
  • Checklist: Human-subject consent, personally identifiable information, offensive content, participant instructions, risks, and compensation are marked not applicable.The checklist contains no crowdsourcing or human-subject study requirements for this work.

A Latency-Driven Slimming Algorithm

The slimming algorithm uses a pretrained super-network and latency lookup table to select a sub-network under a target latency budget. It estimates architectural-choice importance and reduces depth, width, or 3D MetaBlocks before training the selected architecture from scratch.

  • Importance Estimation: Importance scores for architectural choices are estimated from the trainable architecture parameter α.The speed-oriented method avoids retraining each sub-network during importance estimation.
  • Super-net Pretraining: The algorithm begins with super-net pretraining and a latency lookup table describing candidate 4D and 3D MetaBlocks.The lookup table is used to approximate the final latency budget.
  • Latency-driven Slimming: The search action space includes depth reduction, width reduction, and 3D MetaBlock reduction.These actions are applied while searching for a sub-network with the target latency.
  • Final Model Training: The selected sub-network is trained from scratch after slimming to obtain the final model.The algorithm terminates after finding a sub-network satisfying the target latency budget.

B Ablation Analysis

Ablations show that EfficientFormer’s operator choices and dimension-consistent design improve accuracy and latency, while latency-driven slimming selects configurations beyond MACs alone.

  • Patch Embedding: 48% lower inference latency and 0.7% higher accuracy result from replacing non-overlapping large-kernel patch embedding with a convolution stem.The comparison uses EfficientFormer variants V5 and V1.
  • Normalization: 0.3% top-1 accuracy improvement from Group Normalization incurs latency overhead because it cannot be folded during inference.Layer Normalization also increases latency with negligible performance improvement, motivating CONV-BN throughout the 4D partition.
  • Activation Functions: ReLU provides little speedup over GeLU on iPhone 12 with CoreMLTools, while HardSwish is significantly slower than both.The authors therefore select activation functions case by case.
  • Dimension Consistent Design: Dimension-consistent design achieves faster inference than non-dimension-consistent design for EfficientFormer-L7 and DummyNet.The comparison is reported for both models.
  • Latency Driven Slimming: Latency-driven slimming compares networks at equal computation, using 1.3 GMACs as the reference for EfficientFormer-L1.Random networks vary depth and width relative to EfficientFormer-L1 and are trained on ImageNet-1K with the same strategy.

C Analysis of Hardware Utilization

Hardware-utilization analysis evaluates throughput relative to computation cost and shows that EfficientFormer maintains a favorable latency–accuracy balance across model scales.

  • Hardware Utilization: Throughput in TFLOPS measures hardware utilization by dividing model FLOPs by execution time; higher throughput indicates better use of hardware computation.The analysis benchmarks EfficientFormer variants against CNN and ViT baselines.
  • Hardware Utilization: EfficientFormer-LS variants span 1.1M–31.3M parameters and 0.09G–3.9G MACs for latency and utilization benchmarking on the iPhone 12 NPU.The series is created by linearly scaling EfficientFormer-L1 depth and width.
  • Hardware Utilization: 1.3 GMACs EfficientFormer-L1 reaches 1.6 ms latency while maintaining high accuracy, whereas super-tiny variants remain near 1 ms but underutilize hardware.For the smallest models, data processing and transfer become bottlenecks.
  • Hardware Utilization: At 4 GMACs, EfficientFormer-LS-14 achieves 3.3× higher TFLOPS than DeiT-S and 2.2× higher TFLOPS than PoolFormer, with throughput comparable to ResNet50.In the lightweight domain, EfficientFormer-LS-4 achieves 2.2× higher TFLOPS than EfficientNet-B0.

D Architecture of EfficientFormers

EfficientFormer architecture details specify stage resolutions, block counts, embedding dimensions, attention dimensions, head count, and MLP expansion.

  • Architecture Details: EfficientFormer-L1, L3, and L7 are specified by stage resolution, block count, and embedding dimension.Attention blocks use eight heads, with Query/Key dimensions reported for each model.
  • Architecture Details: The default MLP expansion ratio is 4, matching the ratio used in most ViT architectures.The architecture table also reports the dimension of Queries and Keys for MHSA blocks.
Loading 2206.01191v5…