PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention

Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, Yixuan Yuan

arXiv:2305.07027v1cs.CV

TL;DR

Vision transformers deliver strong performance but their large models and computational costs hinder real-time use. EfficientViT addresses memory-bound operations and redundant attention with a sandwich-layout block and cascaded group attention, achieving improved speed–accuracy trade-offs over efficient CNN and ViT models.

  • Problem

    Vision transformers’ large model sizes and computational costs hinder real-time applications, while parameter and FLOP reductions do not necessarily improve wall-clock throughput.

  • Method

    EfficientViT combines a single MHSA layer between FFN layers with cascaded group attention that feeds heads different feature splits and reallocates parameters toward important components.

  • Results

    EfficientViT improves speed and accuracy across efficient CNN and ViT comparisons, including 1.9% higher accuracy and 40.4%/45.2% faster GPU/CPU inference than MobileNetV3-Large.

  • Takeaways & Limitations

    The model family provides a high-speed vision-transformer design with a reported efficiency–accuracy trade-off and transfer ability across downstream benchmarks.

  • Takeaways & Limitations

    EfficientViT has slightly larger model size than state-of-the-art efficient CNNs, and its architectures are designed manually rather than searched automatically.

Abstract

from arXiv · show

Vision transformers have shown great success due to their high model capabilities. However, their remarkable performance is accompanied by heavy computation costs, which makes them unsuitable for real-time applications. In this paper, we propose a family of high-speed vision transformers named EfficientViT. We find that the speed of existing transformer models is commonly bounded by memory inefficient operations, especially the tensor reshaping and element-wise functions in MHSA. Therefore, we design a new building block with a sandwich layout, i.e., using a single memory-bound MHSA between efficient FFN layers, which improves memory efficiency while enhancing channel communication. Moreover, we discover that the attention maps share high similarities across heads, leading to computational redundancy. To address this, we present a cascaded group attention module feeding attention heads with different splits of the full feature, which not only saves computation cost but also improves attention diversity. Comprehensive experiments demonstrate EfficientViT outperforms existing efficient models, striking a good trade-off between speed and accuracy. For instance, our EfficientViT-M5 surpasses MobileNetV3-Large by 1.9% in accuracy, while getting 40.4% and 45.2% higher throughput on Nvidia V100 GPU and Intel Xeon CPU, respectively. Compared to the recent efficient model MobileViT-XXS, EfficientViT-M2 achieves 1.8% superior accuracy, while running 5.8x/3.7x faster on the GPU/CPU, and 7.4x faster when converted to ONNX format. Code and models are available at https://github.com/microsoft/Cream/tree/main/EfficientViT.

1. Introduction

EfficientViT addresses the speed and computation costs that limit vision transformers in real-time applications by analyzing memory access and redundancy, then designing efficient transformer components. The resulting models improve speed–accuracy trade-offs against efficient CNN and ViT baselines.

  • Vision transformers’ increasing model sizes and computational costs make them unsuitable for real-time applications.
  • Existing efficient models often optimize parameters or FLOPs without achieving comparable wall-clock throughput.The paper notes that MobileViT-XS uses fewer FLOPs than DeiT-T but runs slower on an Nvidia V100 GPU.
  • EfficientViT analyzes memory access, computation redundancy, and parameter usage to derive principles for efficient vision-transformer design.
  • The proposed architecture combines a sandwich-layout block with one memory-bound MHSA layer and cascaded group attention using different feature splits across heads.
  • 1.9% higher top-1 accuracy and 40.4%/45.2% faster GPU/CPU inference are achieved by EfficientViT-M5 over MobileNetV3-Large.
  • EfficientViT-M2 exceeds MobileViT-XXS by 1.8% accuracy and runs 5.8×/3.7× faster on GPU/CPU, with 7.4× faster ONNX inference.

2. Going Faster with Vision Transformers

The analysis identifies memory access overhead and redundant attention as key efficiency challenges in vision transformers. It motivates reducing MHSA usage, diversifying attention inputs, and reallocating channels toward more important components.

  • Memory Efficiency: Memory access overhead, especially from reshaping, element-wise operations, and normalization, is a critical source of transformer runtime cost.
  • Memory Efficiency: Reducing the MHSA layer ratio can improve efficiency because MHSA contains more memory-inefficient operations than FFN layers.
  • Memory Efficiency: 44.26% of total runtime is spent on memory-bound operations in Swin-T-1.25× with 20% MHSA layers.
  • Computation Redundancy: Attention heads can learn similar projections, creating redundant attention maps and motivating feature diversification across heads.
  • Parameter Efficiency: Taylor structured pruning indicates that early stages preserve more channels, while Q, K, and FFN dimensions are trimmed more than V dimensions.

3. Efficient Vision Transformer

EfficientViT combines a memory-efficient sandwich layout with cascaded group attention and parameter reallocation to improve transformer efficiency across memory, computation, and parameters.

  • Sandwich Layout: EfficientViT uses a sandwich layout with one self-attention layer between FFN layers, reducing memory-bound attention overhead while improving channel communication.The block applies N FFNs before and after the single self-attention layer and adds depthwise-convolution token interaction before each FFN.
  • Cascaded Group Attention: Cascaded group attention feeds each head a different split of the full feature, explicitly decomposing attention computation across heads.For h heads, each head operates on its corresponding input split, with projection layers mapping splits into separate subspaces.
  • Cascaded Group Attention: Cascading head outputs refines later-head inputs and increases attention diversity while reducing QKV computation and parameters by h×.Each subsequent head receives its input split plus the preceding head output, allowing progressively refined representations.
  • Parameter Reallocation: Parameter reallocation expands critical modules while shrinking less important ones, including smaller Q and K projections and a reduced FFN expansion ratio.The V projection can match the input embedding dimension, while the FFN expansion ratio is reduced from 4 to 2.
  • Network Architecture: The hierarchical EfficientViT architecture uses overlapping patch embedding, three stages, progressive token reduction, and six width-depth scales.Token counts decrease by 4× at each subsampling layer, while stage widths increase by a factor no greater than 2.
  • Network Architecture: EfficientViT uses batch normalization and ReLU for runtime advantages over layer normalization and slower activations such as GELU or HardSwish.Batch normalization can be folded into preceding convolution or linear layers, and ReLU is better supported by some deployment platforms.

4. Experiments

Experiments evaluate EfficientViT on ImageNet-1K, downstream classification, object detection, and controlled ablations. Across these settings, EfficientViT generally improves the speed–accuracy trade-off, while ablations identify effective architectural and parameter choices.

  • ImageNet classification: EfficientViT achieves the best accuracy–speed trade-off in most ImageNet evaluation settings against efficient CNN and ViT models.The comparisons use ImageNet-1K classification and throughput measured on GPU, CPU, and ONNX settings.
  • ImageNet classification: 1.9% higher top-1 accuracy and 40.5%/45.2% faster V100 GPU/Intel CPU inference are achieved by EfficientViT-M5 over MobileNetV3-Large.EfficientViT-M5 is 11.5% slower as an ONNX model, while remaining 2.3×/1.9× faster than EfficientNet-B0 on GPU/CPU.
  • ImageNet classification: 4.4×/3.0× faster CPU/GPU inference and 3.7× higher ONNX speed are reported for EfficientViT-M4 at similar performance to EdgeViT-XXS.EfficientViT-M2 also achieves slightly better performance than MobileViTV2-0.5 with 3.4×/3.5× higher GPU/CPU throughput.
  • ImageNet classification: 79.8% top-1 accuracy at 3,986 images/s is reached by EfficientViT-M5↑384, while EfficientViT-M5↑512 reaches 80.8% top-1 accuracy.These results evaluate finetuning at higher image resolutions.
  • Ablation study: Replacing the sandwich block with a Swin block lowers top-1 accuracy by 3.0% at similar speed, while N=1 FFN placement gives the best efficiency.Increasing the number of FFNs further is ineffective because of limited long-range spatial relation.
  • Ablation study: Replacing CGA with MHSA decreases accuracy by 1.1% and ONNX speed by 5.9%, while parameter reallocation raises accuracy by 1.4%/1.5% and GPU throughput by 4.9%/3.8%.The parameter-reallocation comparisons remove either QKV channel reallocation or FFN-ratio reduction.
  • Ablation study: Accuracy improves from 70.3% to 71.3% as the V-dimension ratio rises from 0.4 to 1.0, whereas increasing it to 1.2 adds only 0.1%.The results support setting V channels close to the input embedding for parameter efficiency.

5. Related Work

Related work includes efficient CNNs and efficient vision transformers, with approaches targeting compact architectures or cheaper self-attention. The supplied passages also include an ablation figure on head dimensions and value-channel ratios.

  • Efficient CNNs: Efficient CNNs use depthwise separable convolutions, inverted residual expansion, or neural architecture search to build compact models.Examples include Xception, MobileNetV2, MobileNetV3, and EfficientNet.
  • Design ablations: Figure 7 ablates each head's QK dimension and the ratio of V dimension to the input embedding.These variables are evaluated as design choices for EfficientViT.
  • Efficient ViTs: Efficient ViT methods primarily reduce self-attention cost through sparse attention or low-rank approximation, or redesign the overall architecture.The passage categorizes recent efficient transformers into efficient self-attention and efficient architecture design.

6. Conclusion

The paper presents EfficientViT as a fast vision-transformer family supported by extensive experiments and downstream benchmarks. It also identifies larger model size and manual design as limitations for future work.

  • Conclusion: EfficientViT combines memory-efficient operations and cascaded group attention, with experiments showing high speed and superiority on various downstream benchmarks.The conclusion frames the method as a systematic analysis of vision-transformer inference speed and a resulting model family.
  • Limitations: EfficientViT models have slightly larger size than state-of-the-art efficient CNNs because the sandwich layout adds extra FFNs, and their designs are manual.Future work targets reducing model size and incorporating automatic search techniques.
Loading 2305.07027v1…