PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Vision Transformers for MobileNet Size and Speed

Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, Jian Ren

arXiv:2212.08059v2cs.CVcs.AIcs.LG

TL;DR

Efficient vision transformers still face a mobile-deployment gap because they can be slower or larger than lightweight CNNs, despite efforts to improve their efficiency. This paper revisits transformer design, introduces a low-latency supernet and fine-grained joint search over size and speed, and develops EfficientFormerV2. At equal size and latency, EfficientFormerV2-S0 achieves 3.5% higher ImageNet-1K top-1 accuracy than MobileNetV2, while EfficientFormerV2-S1 matches EfficientFormer-L1 with substantially lower size and latency.

  • Problem

    Vision Transformers remain higher-latency or larger than lightweight CNNs, motivating the question of whether transformers can achieve comparable mobile size and speed while preserving high performance.

  • Method

    The paper revisits transformer design and combines mobile-friendly architectural changes with a low-latency supernet and fine-grained joint search over model size and latency.

  • Results

    3.5% higher top-1 accuracy on ImageNet-1K is achieved by EfficientFormerV2-S0 than MobileNetV2 at exactly the same size and iPhone 12 latency; EfficientFormerV2-S1 matches EfficientFormer-L1 while being 2× smaller and 1.3× faster.

  • Takeaways & Limitations

    Properly designed and jointly optimized vision transformers can reach MobileNetV2-level size and speed while maintaining higher performance.

  • Takeaways & Limitations

    Efficient-ViT approaches remain constrained by mobile hardware support for sophisticated operations, and prior methods often optimize only one efficiency metric.

Abstract

from arXiv · show

With the success of Vision Transformers (ViTs) in computer vision tasks, recent arts try to optimize the performance and complexity of ViTs to enable efficient deployment on mobile devices. Multiple approaches are proposed to accelerate attention mechanism, improve inefficient designs, or incorporate mobile-friendly lightweight convolutions to form hybrid architectures. However, ViT and its variants still have higher latency or considerably more parameters than lightweight CNNs, even true for the years-old MobileNet. In practice, latency and size are both crucial for efficient deployment on resource-constraint hardware. In this work, we investigate a central question, can transformer models run as fast as MobileNet and maintain a similar size? We revisit the design choices of ViTs and propose a novel supernet with low latency and high parameter efficiency. We further introduce a novel fine-grained joint search strategy for transformer models that can find efficient architectures by optimizing latency and number of parameters simultaneously. The proposed models, EfficientFormerV2, achieve 3.5% higher top-1 accuracy than MobileNetV2 on ImageNet-1K with similar latency and parameters. This work demonstrate that properly designed and optimized vision transformers can achieve high performance even with MobileNet-level size and speed.

1. Introduction

Vision Transformers model global spatial dependencies effectively but remain costly for mobile deployment. EfficientFormerV2 addresses this gap through mobile-friendly architecture and joint size–latency search, achieving MobileNet-level efficiency with higher ImageNet-1K accuracy.

  • Motivation: MHSA captures global spatial dependencies and dynamic second-order information, but its computation grows quadratically with image resolution.This cost contributes to higher computation and latency than lightweight CNNs.
  • Motivation: Window-based attention reduces attention complexity to linear in resolution, but sophisticated attention patterns can be difficult to accelerate on mobile hardware.The approach constrains the receptive field to a predefined window or span.
  • Approach: EfficientFormerV2 jointly optimizes parameters, latency, and performance using mobile-friendly architectural improvements and fine-grained architecture search.The search produces models under varied size and speed constraints.
  • Results: 3.5% higher top-1 accuracy on ImageNet-1K is achieved by EfficientFormerV2-S0 than MobileNetV2 at exactly the same size and iPhone 12 latency.Latency is profiled on iPhone 12 running iOS 16.
  • Results: EfficientFormerV2-S1 matches EfficientFormer-L1 performance while being 2× smaller and 1.3× faster.The comparison is reported in Table 2.
  • Conclusion: The models are presented as vision-transformer backbones that can match MobileNetV2 size and speed while delivering better performance.The paper also positions EfficientFormerV2 as a backbone for downstream tasks.

2. Related Work

Related efficient-ViT research improves accuracy or efficiency, but mobile suitability remains limited by unsupported operations and optimization focused on only one resource metric. Existing models therefore trade speed, size, or parameter efficiency against one another.

  • Efficient ViT designs: Vision-transformer research has explored hierarchical designs, convolutional locality, local attention, and alternative token-mixing mechanisms.These approaches build on ViT and DeiT to improve vision performance and efficiency.
  • Mobile deployment constraints: Reducing attention complexity to linear can still leave reshaping and indexing operations unsupported on resource-constrained devices.The paper identifies mobile-friendly design as a separate requirement from asymptotic complexity reduction.
  • Efficiency trade-offs: Existing efficient-ViT methods often optimize one metric, producing models that are either parameter-heavy, such as EfficientFormer, or slow, such as MobileViT.LeViT and MobileFormer are also described as trading favorable FLOPs for redundant parameters.

3. Rethinking Hybrid Transformer Network

The section revisits hybrid transformer designs for mobile deployment by jointly considering performance, parameter count, and latency. It introduces architectural changes and attention strategies that improve efficiency while preserving competitive accuracy.

  • Mobile-friendly architectural choices: Replacing 3 × 3 average pooling with depth-wise convolution improves performance without latency overhead or substantial parameter growth.The replacement improves performance by 0.6% with only 0.02M extra parameters.
  • Mobile-friendly architectural choices: Deeper, narrower networks improve accuracy while reducing both parameters and latency relative to the prior baseline.The search yields a 0.2% accuracy improvement, 0.13M fewer parameters, and 0.1ms lower latency.
  • MHSA improvements: MHSA enhancements add local information to Value features and communication across attention heads, raising accuracy with similar parameters and latency.The resulting baseline reaches 80.8% accuracy.
  • Attention on higher resolution: Applying MHSA at higher resolution increases efficiency challenges because attention cost grows quadratically with spatial resolution.The baseline uses MHSA only at 1⁄32 resolution, while earlier-stage attention is evaluated at higher resolutions.
  • Attention on higher resolution: Stride Attention downsamples Queries, Keys, and Values to fixed resolution, then interpolates attention outputs back to the original resolution.This reduces latency from 3.5ms to 1.5ms while retaining 81.5% versus 81.7% accuracy.
  • Dual-Path Attention downsampling: Dual-Path Attention downsampling combines pooling, depth-wise convolution, projection, and a residual strided convolution to mix local and global information.The strategy is proposed for attention downsampling in higher-resolution stages.

4. EfficientFormerV2

EfficientFormerV2 combines mobile-oriented transformer design with fine-grained joint search over model size and speed. Its supernet and evaluation-based search support subnetworks under varied efficiency constraints.

  • Joint search objective: EfficientFormerV2 jointly optimizes model size and latency rather than optimizing only one mobile-efficiency metric.The search objective uses parameters and measured device latency to identify Pareto-optimal architectures.
  • Mobile-oriented architecture: The four-stage hierarchy uses convolutional input embedding, local FFNs in early stages, and local FFN plus global MHSA blocks in later stages.The stages use progressively reduced spatial resolutions, while MHSA is reserved for later stages.
  • Search space: The supernet searches network depth, stage width, and each FFN’s per-block expansion ratio.Per-block expansion ratios provide a finer-grained space because FFNs contain much of the network’s computation and parameters.
  • Joint search objective: MES measures accuracy together with normalized size and latency, using MobileNetV2 size and iPhone 12 latency as reference units.The formulation assigns latency importance 1.0 and size importance 0.5, and can extend to metrics such as memory or energy.
  • Search space and supernet: The supernet is pretrained with the Sandwich Rule and supports elastic depth and switchable width before searched subnetworks are trained from scratch.Each iteration trains the largest, smallest, and two randomly sampled subnetworks.
  • Search algorithm: The search evaluates frontier slimming actions and selects the action with the smallest accuracy drop per MES improvement.Actions can reduce blocks, stage width, or an individual FFN expansion ratio; latency is obtained from an iPhone 12 lookup table.

5. Experiments

Experiments evaluate EfficientFormerV2 across ImageNet-1K, downstream tasks, hardware platforms, and search baselines. The models achieve strong accuracy while jointly addressing mobile latency and parameter constraints.

  • Experimental Setup: EfficientFormerV2 is benchmarked on iPhone 12 NPU, Pixel 6 CPU, and Nvidia A100 GPU using deployment-specific compilation and batch settings.Latency uses batch size 1 on mobile devices and batch size 64 on the A100.
  • ImageNet-1K Classification: 3.5% higher top-1 accuracy than MobileNetV2 is achieved with a similar number of parameters on ImageNet-1K.EfficientFormerV2-S0 also exceeds EdgeViT-XXS by 1.3% top-1 accuracy with 0.6M fewer parameters.
  • ImageNet-1K Classification: EfficientFormerV2 improves accuracy over several comparable models at similar or lower latency, including gains of 0.8%, 0.6%, and 2.4% top-1 accuracy.The comparisons are against UniNet-B1, EdgeViT-S, and EfficientFormer-L1, respectively.
  • Jointly Optimized Size and Speed: With joint size-and-speed evaluation, EfficientFormerV2-S1 exceeds MobileViT-XS, EdgeViT-XXS, and EdgeViT-XS by 4.2%, 4.6%, and 1.5% top-1 accuracy.EfficientFormerV2-L exceeds MobileFormer-508M by 4.0% top-1 accuracy with 1.8× higher MES.
  • Downstream Tasks: EfficientFormerV2-S2 outperforms PoolFormer-S12 by 6.1 APbox and 4.9 APmask with similar model size on MS COCO detection and instance segmentation.On segmentation, EfficientFormerV2-S2 also exceeds PoolFormer-S12 and EfficientFormer-L1 by 5.2 and 3.5 mIoU, respectively.
  • Search Algorithm: The proposed fine-grained joint search finds higher-accuracy models than random search and EfficientFormer’s search under similar parameter and latency constraints.Latency for the search-algorithm comparison is measured on iPhone 12.

6. Discussion and Conclusion

The paper combines mobile-friendly architectural changes with fine-grained joint search to produce EfficientFormerV2 backbones. Across hardware and tasks, the models are lightweight, fast, and high-performing, while future work could optimize additional metrics.

  • Discussion and Conclusion: EfficientFormerV2 combines mobile-friendly architectural changes with fine-grained joint optimization of model size and inference speed.The models are benchmarked against existing studies on different hardware.
  • Discussion and Conclusion: EfficientFormerV2 is reported as lightweight, ultra-fast in inference speed, and high performance across the evaluated settings.The conclusion frames the model family as a mobile vision backbone.
  • Discussion and Conclusion: Future work could extend joint optimization to memory footprint and CO2 emission in addition to size and speed.These metrics are identified as subsequent directions rather than evaluated objectives in this work.

A. More Experimental Details and Results

Additional experiments document the training recipe, evaluate results without distillation, and analyze explicit position encoding. These details clarify how EfficientFormerV2 was trained and which design choices affect accuracy.

  • Results Without Distillation: Table 6 reports EfficientFormerV2 results without distillation.The supplied passage identifies the evaluation condition but does not provide the numerical results.
  • Position Encoding: Table 7 analyzes explicit position encoding using EfficientFormerV2-S1.The accompanying result states that EfficientFormerV2 preserves reasonable accuracy without explicit position encoding.

B. More Ablation Analysis of Search Algorithm

Ablations show that fine-grained choices and joint search improve mobile-efficiency outcomes. The analysis covers expansion ratios, search objectives, supernet evaluation, and deployment-relevant design constraints.

  • Importance of Expansion Ratios: Varying expansion ratios changes performance substantially even when models have the same number of parameters, so width-only search cannot ensure Pareto optimality.The ablation motivates searching expansion ratios alongside width.
  • Search Methods: The complete search jointly optimizes depth, width, and expansion ratios, extending EfficientFormer’s depth-and-width search and pruning-based alternatives.Table 9 compares these search procedures on ImageNet-1K under a 7M-parameter budget.
  • Search-Objective Weights: Increasing αsize produces slower models, while the search remains stable across different αlatency and αsize settings.The current setting is αlatency = 1.0 and αsize = 0.5, selected for alignment with recent comparison methods.
  • Visualization of Search Results: The proposed search method yields advantageous performance under MES compared with subnetworks found by EfficientFormer’s pipeline.Figure 3 visualizes searched subnetworks on a logarithmic MES scale from the same ImageNet-1K-trained supernet.
  • Design Choice Ablation: Without stride attention, EfficientFormerV2 encounters memory issues and cannot run on mobile in the detection and instance-segmentation ablation.The finding is reported for EfficientFormerV2-S2 trained on MS COCO from scratch.
  • Search Cost and Supernet Quality: The supernet produces subnet evaluation accuracy above 40% and correlation r2 = 0.91 with final accuracy, compared with below 10% and 0.63 for EagleEye.Random search for the L-level model is estimated at 80 GPU days, twice the supernet training time.

C. Network Configurations

This section provides architecture details for EfficientFormerV2 and describes subnet evaluation against training from scratch.

  • C. Network Configurations: EfficientFormerV2-S0, S1, S2, and L are specified by stage resolution, width, depth, and per-block expansion ratios.These detailed network architectures are provided in Table 12.
  • C. Network Configurations: Table 12 presents the architecture details of EfficientFormerV2.
  • C. Network Configurations: Figure 5 evaluates searched models against random models for subnet evaluation versus training from scratch.Searched models are shown in red, while random models are shown in blue.
Loading 2212.08059v2…